Can't specify multi-channel data when uploading for Ilastik batch prediction

Noah_Greenwald · August 14, 2019, 6:50pm

Hi, I’m running into an issue when trying to take a trained Ilastik Pixel Classification workflow and run it on data not seen during training. I followed the instructions here: https://www.ilastik.org/documentation/pixelclassification/pixelclassification for creating a pixel classification workflow. This worked great.

In order to run it on unseen data, I attempted to the follow the instructions here: https://www.ilastik.org/documentation/basics/batch.html for batch processing. However, I don’t see the same options. In particular, I don’t have the option to specify the format in which the data is uploaded. Instead, the TIFs are all uploaded individually, and I can’t specify the axes over which to combine them. This means that the classifier can’t run on them, because the shape up the uploaded images (3 distinct 1024x1024 images) is different from the training data (3x1024x1024). See below for screenshots of the interface I have for uploading, which lacks the option shown in the tutorial above.

Thanks in advance,
Noah

k-dominik · August 15, 2019, 6:24am

Hi @Noah_Greenwald,

supplying stacked files is indeed not possible atm with the batch processing applet. You can, however, use ilastik’s headless mode in order to process new, unseen files with your trained classifier. This means using a terminal to start ilastik. Please take the time to read the docs there.

But then you can write something like


$ /Applications/ilastik-1.3.3b2-OSX.app/Contents/ilastik-release/run_ilastik.sh  --headless \
    --project=/Users/<USERNAME>/<PATH_TO_TRAINED_PROJECT>/MyProject.ilp \
    --raw_data="/Users/<USERNAME>/<PATH_TO_TRAINED_DATA>/stack_*.tif" \
    --stack_along="c"
    .... add your output options according to the documentations

don’t hesitate to let us know if you need help constructing the command.

Noah_Greenwald · August 15, 2019, 6:48pm

Hi Dominik,
Thanks so much! That did the trick. I ran into a number of library errors that were raised during runtime, but it produced the output anyway. I’ve pasted them below for your reference in case any of these are not known issues.

Ilastik$ /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/run_ilastik.sh --headless --project MyProject.ilp --raw_data=“Point1_12_18_23_3X/Point23/raw/*.tif” --stack_along=“c”
INFO lazyflow.operators.filterOperators: Using fast filters.
WARNING init.py(11): UserWarning: init: Could not import tiktorch classifier
INFO ilastik_main: Starting ilastik from “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release”.
Starting ilastik from “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release”.
ERROR 2019-08-15 11:23:03,724 PluginManager 32853 4527752640 Unable to import plugin: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_contours_with_head_export
Traceback (most recent call last):
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/yapsy/PluginManager.py”, line 488, in loadPlugins
candidate_module = imp.load_module(plugin_module_name,plugin_file,candidate_filepath+".py",(“py”,“r”,imp.PY_SOURCE))
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 235, in load_module
return load_source(name, filename, file)
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 172, in load_source
module = _load(spec)
File “”, line 684, in _load
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_contours_with_head_export.py”, line 10, in
from skimage import measure
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/init.py”, line 177, in
from .data import data_dir
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/data/init.py”, line 15, in
from …io import imread, use_plugin
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/init.py”, line 7, in
from .manage_plugins import *
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/manage_plugins.py”, line 28, in
from .collection import imread_collection_wrapper
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/collection.py”, line 12, in
from PIL import Image
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/Image.py”, line 64, in
from . import _imaging as core
ImportError: dlopen(/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libjpeg.9.dylib
Referenced from: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so
Reason: Incompatible library version: _imaging.cpython-36m-darwin.so requires version 13.0.0 or later, but libjpeg.9.dylib provides version 12.0.0
ERROR 2019-08-15 11:23:03,750 PluginManager 32853 4527752640 Unable to import plugin: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_contour_export
Traceback (most recent call last):
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/yapsy/PluginManager.py”, line 488, in loadPlugins
candidate_module = imp.load_module(plugin_module_name,plugin_file,candidate_filepath+".py",(“py”,“r”,imp.PY_SOURCE))
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 235, in load_module
return load_source(name, filename, file)
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 172, in load_source
module = _load(spec)
File “”, line 684, in _load
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_contour_export.py”, line 10, in
from skimage import measure
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/init.py”, line 177, in
from .data import data_dir
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/data/init.py”, line 15, in
from …io import imread, use_plugin
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/init.py”, line 7, in
from .manage_plugins import *
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/manage_plugins.py”, line 28, in
from .collection import imread_collection_wrapper
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/collection.py”, line 12, in
from PIL import Image
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/Image.py”, line 64, in
from . import _imaging as core
ImportError: dlopen(/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libjpeg.9.dylib
Referenced from: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so
Reason: Incompatible library version: _imaging.cpython-36m-darwin.so requires version 13.0.0 or later, but libjpeg.9.dylib provides version 12.0.0
ERROR 2019-08-15 11:23:03,807 PluginManager 32853 4527752640 Unable to import plugin: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_mwt_export
Traceback (most recent call last):
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/yapsy/PluginManager.py”, line 488, in loadPlugins
candidate_module = imp.load_module(plugin_module_name,plugin_file,candidate_filepath+".py",(“py”,“r”,imp.PY_SOURCE))
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 235, in load_module
return load_source(name, filename, file)
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 172, in load_source
module = _load(spec)
File “”, line 684, in _load
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_mwt_export.py”, line 12, in
from skimage import measure
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/init.py”, line 177, in
from .data import data_dir
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/data/init.py”, line 15, in
from …io import imread, use_plugin
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/init.py”, line 7, in
from .manage_plugins import *
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/manage_plugins.py”, line 28, in
from .collection import imread_collection_wrapper
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/collection.py”, line 12, in
from PIL import Image
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/Image.py”, line 64, in
from . import _imaging as core
ImportError: dlopen(/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libjpeg.9.dylib
Referenced from: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so
Reason: Incompatible library version: _imaging.cpython-36m-darwin.so requires version 13.0.0 or later, but libjpeg.9.dylib provides version 12.0.0
ERROR 2019-08-15 11:23:03,818 PluginManager 32853 4527752640 Unable to import plugin: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_ctc_export
Traceback (most recent call last):
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/yapsy/PluginManager.py”, line 488, in loadPlugins
candidate_module = imp.load_module(plugin_module_name,plugin_file,candidate_filepath+".py",(“py”,“r”,imp.PY_SOURCE))
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 235, in load_module
return load_source(name, filename, file)
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/imp.py”, line 172, in load_source
module = _load(spec)
File “”, line 684, in _load
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/ilastik-meta/ilastik/ilastik/plugins_default/tracking_ctc_export.py”, line 4, in
from skimage.external import tifffile
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/init.py”, line 177, in
from .data import data_dir
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/data/init.py”, line 15, in
from …io import imread, use_plugin
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/init.py”, line 7, in
from .manage_plugins import *
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/manage_plugins.py”, line 28, in
from .collection import imread_collection_wrapper
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/skimage/io/collection.py”, line 12, in
from PIL import Image
File “/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/Image.py”, line 64, in
from . import _imaging as core
ImportError: dlopen(/Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libjpeg.9.dylib
Referenced from: /Applications/ilastik-1.3.2post1-OSX.app/Contents/ilastik-release/lib/python3.6/site-packages/PIL/_imaging.cpython-36m-darwin.so
Reason: Incompatible library version: _imaging.cpython-36m-darwin.so requires version 13.0.0 or later, but libjpeg.9.dylib provides version 12.0.0
WARNING 2019-08-15 11:23:03,988 opConservationTracking 32853 4527752640 Could not find any ILP solver
WARNING 2019-08-15 11:23:04,001 opStructuredTracking 32853 4527752640 Could not find any ILP solver
WARNING 2019-08-15 11:23:04,004 structuredTrackingWorkflow 32853 4527752640 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learning for tracking will be disabled!
INFO ilastik.shell.projectManager: Opening Project: MyProject.ilp
WARNING stype.py(181): UserWarning: ArrayLike.isCompatible: FIXME here
INFO ilastik.workflows.pixelClassification.pixelClassificationWorkflow: Beginning Batch Processing
INFO ilastik.applets.batchProcessing.batchProcessingApplet: Exporting to /Users/noahgreenwald/Documents/Grad_School/Lab/Segmentation_Project/Contours/Ilastik/Point1_12_18_23_3X/Point23/raw/_________________Probabilities.h5/exported_data
INFO lazyflow.operators.ioOperators.ioOperators.OpH5WriterBigDataset: Data shape: (3, 1024, 1024)
INFO lazyflow.utility.bigRequestStreamer: Estimated RAM usage per pixel is 684.0B * safety factor (2.0)
INFO lazyflow.utility.bigRequestStreamer: determining blockshape assuming available_ram is 20.3GiB, split between 8 threads
INFO lazyflow.utility.bigRequestStreamer: Chose blockshape: (3, 1024, 1024)
INFO lazyflow.utility.bigRequestStreamer: Estimated RAM usage per block is 4.0MiB
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype ‘UINT16’.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype ‘UINT16’.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype ‘UINT16’.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype ‘UINT16’.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype ‘UINT16’.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype ‘UINT16’.
DEBUG lazyflow.operators.classifierOperators: Features took 1.572106 seconds, Prediction took 12.105843 seconds for roi: [0, 0, 0] : [1024, 1024, 3]
INFO ilastik.workflows.pixelClassification.pixelClassificationWorkflow: Completed Batch Processing

k-dominik · August 16, 2019, 9:18am

great do hear that!
And thanks a lot for pasting the output there! That’s very good to know.

Maybe you could try the current beta version of ilastik 1.3.3b2 where we have changed a lot on the dependency side of things. Those PIL related warnings/errors should be gone… (hopefully)