Running Autocontext Workflow in Ilastik's Headless Mode?

Hi,
in the Ilastik Documentation, the two Workflows “Pixel Classification” and “Object Classification” are mentioned for usage in headless mode.
As I understood the Autocontext Workflow is basically an improvement of the Pixel Classification Workflow, so would it also be possible for it to run in headless mode?
And if so, how could this be achieved?
Even for a rather old Ilastik version like 1.2.0, or will it be necessary to use a more recent version for running autocontext in headless mode?

Many thanks in advance for your help!
Best,
Alex

Hi @alba,

sure, headless mode is also possible for the autocontext workflow (docs are being updated right now).

so in general you’d construct the command line as for pixel classification:

Example:

 $ ./run_ilastik.sh --headless
    --project="myproject.ilp"
    --raw_data="MyImg.h5"
    --export_source="EXPORT_SOURCE_NAME"

here EXPORT_SOURCE_NAME can take the following values:

  • "Probabilities Stage 2" : probability map (multi-channel) after the second stage
  • "Simple Segmentation Stage 2" : simple single channel integer segmentation image
  • "Uncertainty Stage 2": float uncertainty image
  • "Features Stage 2": multi-channel float features of the second stage
  • "Labels Stage 2": the user provided labels in the second stage
  • "Input Stage 2": the input data that the second stage receives (multichannel, stacked raw_data + predictions from the first stage).

By exchanging the 2 with a 1 you can of course also export the output fro the first stage.

1.2.0 is not really supported anymore. But your old projects should work in the recent version (1.3.2) too. If not we have an interest in fixing it :slight_smile:

Cheers!

1 Like

Hi @k-dominik!
Thank you very much for the detailed information :slightly_smiling_face:

Cheers!

@k-dominik

I am trying to set up Ilastik in headless mode for batch processing an autocontext pipeline on a Windows machine. I am struggling with the exact syntax for the input folder containing all my images. Would it look something like this:

$ cd "\Program Files\ilastik-1.3.3post2"
$ .\ilastik.bat --headless 
--project="Project.ilp" 
--batch_export_dir ~/outputDir/ ~/inputFile.h5/volume/data

Hi @Sreusch,

ilastik might not expand your tilde ~ properly. Try maybe giving the full path.
Furthermore, the parameter --batch_export_dir does not exist. You can specify the output like:

--output_filename_format=/some_dir/outputDir/{nickname}_results.h5 \

The thing in the curly braces is a placeholder that will automatically be filled for every image you have (so calling the workflow with an image that has a different name, will result in a different file).
You can read more about the headless mode in our docs.

Dear @k-dominik,

I tried to run Autocontext headless.
I use:

./run_ilastik.sh --headless \
--project="/home/trasse/Data/ilastik/vessel.ilp" \
--raw_data="/mnt/ag-microscopy_rawdata/LightSheet/user/input/200810_14_4_30RGB.tif"\
--export_source = "Probabilities Stage 2" 

and get this error:

/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/lazyflow/lazyflow/classifiers/__init__.py:20: UserWarning: init: Could not import tiktorch classifier
  warnings.warn("init: Could not import tiktorch classifier")
INFO ilastik_main: Starting ilastik from "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux".
Starting ilastik from "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux".
WARNING 2020-09-18 07:47:30,048 opConservationTracking 299131 140302718216000 Could not find any ILP solver
WARNING 2020-09-18 07:47:30,059 opStructuredTracking 299131 140302718216000 Could not find any ILP solver
WARNING 2020-09-18 07:47:30,061 structuredTrackingWorkflow 299131 140302718216000 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learning for tracking will be disabled!
INFO ilastik.shell.projectManager: Opening Project: /home/trasse/Data/ilastik/vessel.ilp
WARNING dataset.py(313): H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
ERROR 2020-09-18 07:47:30,459 excepthooks 299131 140302718216000 Unhandled exception in thread: 'MainThread'
ERROR 2020-09-18 07:47:30,460 excepthooks 299131 140302718216000 Traceback (most recent call last):
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/lazyflow/lazyflow/operator.py", line 118, in __call__
    instance = ABCMeta.__call__(cls, *args, **kwargs)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 716, in __init__
    super(AutocontextTwoStage, self).__init__(*args, n_stages=2, **kwargs)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 158, in __init__
    self._batch_input_args, unused_args = self.batchProcessingApplet.parse_known_cmdline_args(unused_args)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 56, in parse_known_cmdline_args
    parsed_args, unused_args = DataSelectionApplet.parse_known_cmdline_args(cmdline_args, self.role_names)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/applets/dataSelection/dataSelectionApplet.py", line 162, in parse_known_cmdline_args
    "" + role_args_str
Exception: Invalid command line arguments: All roles must be configured explicitly.
Use the following flags to specify which files are matched with which inputs:
--raw_data, --prediction_mask

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik.py", line 134, in <module>
    main()
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik.py", line 128, in main
    hShell = ilastik_main.main(parsed_args, workflow_cmdline_args)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik_main.py", line 170, in main
    f(shell)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik_main.py", line 347, in loadProject
    shell.openProjectFile(path, parsed_args.readonly)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/shell/headless/headlessShell.py", line 123, in openProjectFile
    project_creation_args=project_creation_args,
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/shell/projectManager.py", line 240, in __init__
    self.workflow = workflowClass(shell, headless, self._workflow_cmdline_args, self._project_creation_args)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/lazyflow/lazyflow/operator.py", line 140, in __call__
    raise RuntimeError(err)
RuntimeError: Could not create instance of '<class 'ilastik.workflows.newAutocontext.newAutocontextWorkflow.AutocontextTwoStage'>'
args   = (<ilastik.shell.headless.headlessShell.HeadlessShell object at 0x7f9ab32fa898>, True, ['--raw_data=/mnt/ag-microscopy_rawdata/LightSheet/MariaElisa/input/200810_14_4_30RGB.tif--export_source', '=', 'Probabilities Stage 2'], [])
kwargs = {}
The exception was:
Invalid command line arguments: All roles must be configured explicitly.
Use the following flags to specify which files are matched with which inputs:
--raw_data, --prediction_mask
Traceback:
Traceback (most recent call last):
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/lazyflow/lazyflow/operator.py", line 118, in __call__
    instance = ABCMeta.__call__(cls, *args, **kwargs)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 716, in __init__
    super(AutocontextTwoStage, self).__init__(*args, n_stages=2, **kwargs)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/workflows/newAutocontext/newAutocontextWorkflow.py", line 158, in __init__
    self._batch_input_args, unused_args = self.batchProcessingApplet.parse_known_cmdline_args(unused_args)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 56, in parse_known_cmdline_args
    parsed_args, unused_args = DataSelectionApplet.parse_known_cmdline_args(cmdline_args, self.role_names)
  File "/home/trasse/Software/ilastik/ilastik-1.3.3post3-Linux/ilastik-meta/ilastik/ilastik/applets/dataSelection/dataSelectionApplet.py", line 162, in parse_known_cmdline_args
    "" + role_args_str
Exception: Invalid command line arguments: All roles must be configured explicitly.
Use the following flags to specify which files are matched with which inputs:
--raw_data, --prediction_mask

Thanks a lot & Kind regards

Tobias

Hi @Tobias,

could you please post your command line again, and wrap it into a code block? Could you, furthermore please, post the full output you get on the command line, when running the command (also ideally in a code block).
You can do so by wrapping the lines with three single quotes (see maybe here).

Hope we can resolve this quickly.

Cheers
Dominik

1 Like

Dear @k-dominik,

thanks a lot and see above.
(But i found the mistake.)
The space before the backslash in line 3 was missing.

But now it seems ilastik does not like my (rather large) RGB stacks,
although autocontext was trained on the same format (small files).
I will try to figure it out and if not, i will post the detailed error.

Thank again.

Kind regards

Tobias

Dear @k-dominik,

the file is:

[544,1632,1800,3]. The RGB was created in Fiji and opens just fine there.

The error log is 3477 lines long.

It ends with:
RuntimeError:
Invalid TIFF image index
(/opt/conda/conda-bld/vigra_1567788650573/work/src/impex/tiff.cxx:356)

error_ilastik.txt (323.9 KB)

Thanks a lot &
Kind regards

Tobias

okay this seems to be a problem with the library we use to read tiffs. For a quick fix with the downside of duplicating the data would be to load the data into fiji and export it to hdf5 with our plugin.

1 Like

Thanks a lot!
I do not know how you read tif in ilastik but with tiffile i can open my file.

If you point me to the way how the Hdf5 file shall look like, i could also save it using from python using h5py, or?

(then i do not need to use ImageJ…)

sure, ideally you’d have an environment that includes vigra, too.

so when using conda, I’d probably do something like:

conda create -n tiffconvert -c conda-forge tifffile vigra h5py numpy

then writing the data would be:

import vigra
import tifffile
import h5py

# assuming the data is loaded with tifffile already to an array
image = ...

# it is cool to help ilastik and tell it the meanings of the axes, we use vigra axistags objects for that

# with your data I am assuming axisorder z y x c
axistags = vigra.defaultAxistags("zyxc")

# write out the file
with h5py.File("path/to/file.h5", "w") as f:
    ds = f.create_dataset(name="mydata", data=image, chunks=(64, 64, 64, 1))
    ds.attrs["axistags"] = axistags.toJSON()
1 Like

this should also give you some performance boost (especially if you write out to hdf5 as well)

1 Like

Thanks a lot!
Now it works!
Blockwise prediction has started (and will take a while…)

1 Like