Ilastik: running batch export from python, with python data structures

I’ve had a script floating round for a while for doing this, but the python API seems to change every release.

I have a pre-trained 2-stage autocontext ILP which I want to open in python, and then hand it a numpy array and get a numpy array back (i.e. not having to read from a file and then write back to a file). It would also be fine to request a region of the project’s raw data stack, instead of handing it an array.

A very cut-down version of my script looks like this (it’s basically just the example at https://github.com/ilastik/ilastik/blob/0b94ea8104e38bda5eb17b1b23df9ed54f689a8f/examples/example_python_client.py ):

from collections import OrderedDict
import ilastik_main
import numpy as np
import vigra
from ilastik.applets.dataSelection import DatasetInfo

# open the project
args = ilastik_main.parse_args([])
args.headless = True
args.readonly = True
args.project = "path/to/my/project.ilp"

shell = ilastik_main.main(args)

# get the top-level operator for the 2nd stage of the autocontext
pixel_classifier = shell.workflow.pcApplets[-1].topLevelOperator

input_data = vigra.taggedView(np.random.randint(0, 255, (100, 100, 1), dtype=np.uint8), "yxc")  # or whatever

role_data_dict = OrderedDict([
    ("Raw Data", [ DatasetInfo(preloaded_array=input_data1)])
])

predictions = shell.workflow.batchProcessingApplet.run_export(role_data_dict, export_to_array=True)

The output I get is

WARNING dataset.py(313): H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
WARNING stype.py(187): UserWarning: ArrayLike.isCompatible: FIXME here
WARNING opSimpleBlockedArrayCache.py(43): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
WARNING opSlicedBlockedArrayCache.py(141): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
WARNING opFormattedDataExport.py(149): UserWarning: The ROI you are attempting to export exceeds the extents of your dataset.  Clipping to dataset bounds.
Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 131, in run_export
    export_to_array=export_to_array)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 239, in _run_export_with_empty_batch_lane
    opDataSelectionBatchLaneView.DatasetGroup[role_index].setValue(info)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 124, in call_in_setup_context
    return func(self, *args, **kwargs)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1267, in setValue
    self._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1470, in _changed
    c._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1470, in _changed
    c._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1476, in _changed
    self._configureOperator(self)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1490, in _configureOperator
    self.operator._setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/operator.py", line 469, in _setupOutputs
    self.setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/ilastik/ilastik/applets/dataExport/opDataExport.py", line 210, in setupOutputs
    self._opFormattedExport.Input.connect( self.Inputs[selection_index] )
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 124, in call_in_setup_context
    return func(self, *args, **kwargs)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 592, in connect
    self._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1476, in _changed
    self._configureOperator(self)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1490, in _configureOperator
    self.operator._setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/operator.py", line 469, in _setupOutputs
    self.setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/operators/ioOperators/opFormattedDataExport.py", line 231, in setupOutputs
    known_keys = {"roi": list(self._opSubRegion.Roi.value)}
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1145, in value
    temp = self[:].wait()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1068, in __getitem__
    raise Slot.SlotNotReadyError(slotInfoMsg)
lazyflow.slot.Slot.SlotNotReadyError: Can't get data from slot <class 'lazyflow.operators.generic.OpSubRegion'>.Roi yet. It isn't ready.First upstream problem slot is: OpSubRegion/OpSubRegion.Roi : 	{_ready : False, shape : None, has_mask : None, _dirty : False}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/ilastik2019/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-0df88d0ccda3>", line 1, in <module>
    runfile('/home/barnesc/work/code/ilastik-tst/example_python_client.py', wdir='/home/barnesc/work/code/ilastik-tst')
  File "/home/barnesc/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/191.7479.30/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/barnesc/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/191.7479.30/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/barnesc/work/code/ilastik-tst/example_python_client.py", line 77, in <module>
    predictions = shell.workflow.batchProcessingApplet.run_export(role_data_dict, export_to_array=True)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 141, in run_export
    batch_lane_index, batch_lane_index)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/ilastik/ilastik/applets/dataSelection/opDataSelection.py", line 646, in removeLane
    super(OpMultiLaneDataSelectionGroup, self).removeLane(laneIndex, finalLength)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/ilastik/ilastik/utility/opMultiLaneWrapper.py", line 43, in removeLane
    self._removeInnerOperator(laneIndex, numLanes-1)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/operatorWrapper.py", line 285, in _removeInnerOperator
    oslot.removeSlot(index, length)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 124, in call_in_setup_context
    return func(self, *args, **kwargs)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 818, in removeSlot
    slot.disconnect()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 124, in call_in_setup_context
    return func(self, *args, **kwargs)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 674, in disconnect
    slot.disconnect()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 124, in call_in_setup_context
    return func(self, *args, **kwargs)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 699, in disconnect
    self._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1470, in _changed
    c._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1470, in _changed
    c._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1476, in _changed
    self._configureOperator(self)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1490, in _configureOperator
    self.operator._setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/operator.py", line 469, in _setupOutputs
    self.setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/ilastik/ilastik/applets/dataExport/opDataExport.py", line 250, in setupOutputs
    self._opFormattedExport.TransactionSlot.connect( self.TransactionSlot )
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 124, in call_in_setup_context
    return func(self, *args, **kwargs)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 592, in connect
    self._changed()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1476, in _changed
    self._configureOperator(self)
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1490, in _configureOperator
    self.operator._setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/operator.py", line 469, in _setupOutputs
    self.setupOutputs()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/operators/ioOperators/opFormattedDataExport.py", line 231, in setupOutputs
    known_keys = {"roi": list(self._opSubRegion.Roi.value)}
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1145, in value
    temp = self[:].wait()
  File "/home/barnesc/.pyenv/versions/ilastik2019/ilastik-meta/lazyflow/lazyflow/slot.py", line 1068, in __getitem__
    raise Slot.SlotNotReadyError(slotInfoMsg)
lazyflow.slot.Slot.SlotNotReadyError: Can't get data from slot <class 'lazyflow.operators.generic.OpSubRegion'>.Roi yet. It isn't ready.First upstream problem slot is: OpSubRegion/OpSubRegion.Roi : 	{_ready : False, shape : None, has_mask : None, _dirty : False}

I don’t know if there’s any documentation of the python API; the pages on github.io are pretty old. Not really sure where to begin.

Hi @clbarnes,

thanks for providing the example, that made it very easy to try on my own. When I run this code (with the one typo corrected input_data1 -> input_data) it simply works with the current master.

My guess is that maybe something is off with your project file? Have you tried opening it in the ilastik GUI, does that work?

On a side note: There will be a lot happening on the API side of things in the coming months. We’ll try to make it way easier calling ilastik from Python.

Sorry about that typo! My project file works fine in the GUI. What’s the correct way to install the current master into an existing conda environment? I’m not too familiar with conda deployment so the various recipes, feedstocks, and meta repositories are going over my head!

My next step is going to be trying the headless mode from the command line; not sure if that would be diagnostically valuable.

Great to hear the API will be evolving! Ilastik is such a powerful tool and great point of access for image analysis; having the flexibility of more easily using it as a backend for a python workflow would be amazing as it’s something I’ve struggled with.

Turns out I also cannot run ilastik from headless mode:

LAZYFLOW_THREADS=25 LAZYFLOW_TOTAL_RAM_MB=100000 $PYENV_VIRTUAL_ENV/run_ilastik.sh \
--readonly \
--headless \
--project full-vol-autocontext_n5.ilp \
--export_source "Probabilities Stage 2" \
--output_format "compressed n5" \
--logfile "log.txt" \
--output_filename_format "output.n5" \
excised.n5
INFO 2019-07-09 13:48:33,777 filterOperators 31461 140311977506560 Using fast filters.
WARNING 2019-07-09 13:48:33,784 warnings 31461 140311977506560 __init__.py(20): UserWarning: init: Could not import tiktorch classifier
INFO 2019-07-09 13:48:33,918 ilastik_main 31461 140311977506560 Starting ilastik from "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019".
INFO 2019-07-09 13:48:34,034 ilastik_main 31461 140311977506560 Resetting lazyflow thread pool with 25 threads.
INFO 2019-07-09 13:48:34,043 ilastik_main 31461 140311977506560 Configuring lazyflow RAM limit to 97.7GiB
INFO 2019-07-09 13:48:34,043 memory 31461 140311977506560 Available memory set to 97.7GiB
WARNING 2019-07-09 13:48:36,531 opConservationTracking 31461 140311977506560 Could not find any ILP solver
WARNING 2019-07-09 13:48:36,540 opStructuredTracking 31461 140311977506560 Could not find any ILP solver
WARNING 2019-07-09 13:48:36,541 structuredTrackingWorkflow 31461 140311977506560 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learning for tracking will be disabled!
INFO 2019-07-09 13:48:36,727 projectManager 31461 140311977506560 Opening Project: full-vol-autocontext_n5.ilp
WARNING 2019-07-09 13:48:36,729 warnings 31461 140311977506560 dataset.py(313): H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
WARNING 2019-07-09 13:48:36,761 warnings 31461 140311977506560 stype.py(187): UserWarning: ArrayLike.isCompatible: FIXME here
INFO 2019-07-09 13:48:36,977 acceleratesupport 31461 140311977506560 OpenGL_accelerate module loaded
INFO 2019-07-09 13:48:36,984 arraydatatype 31461 140311977506560 Using accelerated ArrayDatatype
WARNING 2019-07-09 13:48:37,644 warnings 31461 140311977506560 opSimpleBlockedArrayCache.py(43): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
WARNING 2019-07-09 13:48:37,656 warnings 31461 140311977506560 opSlicedBlockedArrayCache.py(141): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
INFO 2019-07-09 13:48:48,078 filterOperators 31681 140600484841216 Using fast filters.
WARNING 2019-07-09 13:48:48,083 warnings 31681 140600484841216 __init__.py(20): UserWarning: init: Could not import tiktorch classifier
INFO 2019-07-09 13:48:48,204 ilastik_main 31681 140600484841216 Starting ilastik from "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019".
INFO 2019-07-09 13:48:48,204 ilastik_main 31681 140600484841216 Resetting lazyflow thread pool with 25 threads.
INFO 2019-07-09 13:48:48,212 ilastik_main 31681 140600484841216 Configuring lazyflow RAM limit to 97.7GiB
INFO 2019-07-09 13:48:48,213 memory 31681 140600484841216 Available memory set to 97.7GiB
WARNING 2019-07-09 13:48:49,746 opConservationTracking 31681 140600484841216 Could not find any ILP solver
WARNING 2019-07-09 13:48:49,753 opStructuredTracking 31681 140600484841216 Could not find any ILP solver
WARNING 2019-07-09 13:48:49,754 structuredTrackingWorkflow 31681 140600484841216 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learning for tracking will be disabled!
INFO 2019-07-09 13:48:50,037 projectManager 31681 140600484841216 Opening Project: full-vol-autocontext_n5.ilp
WARNING 2019-07-09 13:48:50,038 warnings 31681 140600484841216 dataset.py(313): H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
WARNING 2019-07-09 13:48:50,043 warnings 31681 140600484841216 stype.py(187): UserWarning: ArrayLike.isCompatible: FIXME here
WARNING 2019-07-09 13:48:50,320 warnings 31681 140600484841216 opSimpleBlockedArrayCache.py(43): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
WARNING 2019-07-09 13:48:50,330 warnings 31681 140600484841216 opSlicedBlockedArrayCache.py(141): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
INFO 2019-07-09 13:48:52,087 newAutocontextWorkflow 31681 140600484841216 Beginning Batch Processing
WARNING 2019-07-09 13:48:52,261 warnings 31681 140600484841216 opFormattedDataExport.py(149): UserWarning: The ROI you are attempting to export exceeds the extents of your dataset.  Clipping to dataset bounds.
ERROR 2019-07-09 13:48:55,694 log_exception 31681 140600484841216 Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/shell/projectManager.py", line 441, in _loadProject
    self.workflow.onProjectLoaded( self )
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 442, in onProjectLoaded
    self.batchProcessingApplet.run_export_from_parsed_args(self._batch_input_args)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 66, in run_export_from_parsed_args
    return self.run_export(role_path_dict, parsed_args.input_axes, sequence_axis=parsed_args.stack_along)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 131, in run_export
    export_to_array=export_to_array)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 248, in _run_export_with_empty_batch_lane
    assert opDataExportBatchlaneView.ImageToExport.ready()
AssertionError

ERROR 2019-07-09 13:48:55,694 log_exception 31681 140600484841216 Project could not be loaded due to the exception shown above.
Aborting Project Open Action
ERROR 2019-07-09 13:48:55,934 excepthooks 31681 140600484841216 Unhandled exception in thread: 'MainThread'
ERROR 2019-07-09 13:48:55,935 excepthooks 31681 140600484841216 Traceback (most recent call last):
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik.py", line 139, in <module>
    main()
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik.py", line 133, in main
    hShell = ilastik_main.main(parsed_args, workflow_cmdline_args)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik_main.py", line 218, in main
    f(shell)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik_main.py", line 396, in loadProject
    shell.openProjectFile(path, parsed_args.readonly)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/shell/headless/headlessShell.py", line 117, in openProjectFile
    self.projectManager._loadProject(hdf5File, projectFilePath, readOnly)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/lazyflow/lazyflow/utility/timer.py", line 153, in wrapper
    return func(*args, **kwargs)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/shell/projectManager.py", line 441, in _loadProject
    self.workflow.onProjectLoaded( self )
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 442, in onProjectLoaded
    self.batchProcessingApplet.run_export_from_parsed_args(self._batch_input_args)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 66, in run_export_from_parsed_args
    return self.run_export(role_path_dict, parsed_args.input_axes, sequence_axis=parsed_args.stack_along)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 131, in run_export
    export_to_array=export_to_array)
  File "/home/barnesc/.pyenv/versions/anaconda3-2019.03/envs/ilastik2019/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 248, in _run_export_with_empty_batch_lane
    assert opDataExportBatchlaneView.ImageToExport.ready()
AssertionError

Okay, @clbarnes, this just shows you how badly we handle errors in command line parsing.

My guess is that you specify the input data as a positional argument. There is this bug in Python’s argparse that, in short, breaks positional arguments if you have spaces in your options… So my best guess, at the moment is that your command line might work if you replace excised.n5 with --raw_data="excised.n5"

Some combination of using the ilastik “binary” as a base virtualenv (and pip-installing extra dependencies into it), and a brand new project file (recovering the training data from the original was a bit tricky, but manageable), has made this work! From the python end as well.

Thanks for all your help, across various media!

this sounds very adventurous, glad you made it work!

as for the base env, you could have tried using just the ilastik/scripts/devenv.py script specifying your extra dependencies along -p ilastik-dependencies-no-solvers ....