Headless ilastik in python - batch processing

Hi. I’m new to Ilastik and I’m trying to use it in a headless mode with python. From other topics I read, I wrote a code:

subprocess.run([
    "C:\programs\ilastik-1.3.3post3\ilastik.exe",
    '--headless',
    '--project="C:/Petra/ijs/iCan/pixel_class_2.ilp"',
    '--export_source="Probabilities"',
    '--raw_data="C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0003_comb.tiff"'
])

This code produce what I want, but it gives me some warnings I do not understand:

C:\Users\petra\anaconda3\python.exe C:/Petra/ijs/iCan/software/test_ilastic.py
C:\programs\ilastik-1.3.3post3\ilastik-meta\lazyflow\lazyflow\classifiers\__init__.py:20: UserWarning: init: Could not import tiktorch classifier
  warnings.warn("init: Could not import tiktorch classifier")
INFO ilastik_main: Starting ilastik from "C:\programs\ilastik-1.3.3post3".
Starting ilastik from "C:\programs\ilastik-1.3.3post3".
WARNING 2021-04-13 17:17:00,819 opConservationTracking 8456 8988 Could not find any ILP solver
WARNING 2021-04-13 17:17:00,842 opStructuredTracking 8456 8988 Could not find any ILP solver
WARNING 2021-04-13 17:17:00,845 structuredTrackingWorkflow 8456 8988 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learning for tracking will be disabled!
INFO ilastik.shell.projectManager: Opening Project: C:/Petra/ijs/iCan/pixel_class_2.ilp
WARNING dataset.py(313): H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
WARNING opSimpleBlockedArrayCache.py(43): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
WARNING opSlicedBlockedArrayCache.py(141): FutureWarning: Conversion of the second argument of issubdtype from `dtype` to `np.generic` is deprecated. In future, it will be treated as `np.object_ == np.dtype(dtype).type`.
INFO ilastik.workflows.pixelClassification.pixelClassificationWorkflow: Beginning Batch Processing
INFO ilastik.applets.batchProcessing.batchProcessingApplet: Exporting to C:/Petra/ijs/iCan/image_recog/export/2021_01_14_PC_AS_iCAN/e02/tiff_comb/e02_Draq5_MTO__sc_xy0008_comb_Probabilities.h5/exported_data
INFO lazyflow.operators.ioOperators.ioOperators.OpH5N5WriterBigDataset: Data shape: (3, 500, 500)
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype 'UINT16'.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype 'UINT16'.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype 'UINT16'.
Warning: TIFFDecoder: no TIFFTAG_SAMPLEFORMAT or TIFFTAG_DATATYPE, guessing pixeltype 'UINT16'.
INFO lazyflow.utility.bigRequestStreamer: Estimated RAM usage per pixel is 440.0B * safety factor (2.0)
INFO lazyflow.utility.bigRequestStreamer: determining blockshape assuming available_ram is 8.2GiB, split between 4 threads
INFO lazyflow.utility.bigRequestStreamer: Chose blockshape: (3, 500, 500)
INFO lazyflow.utility.bigRequestStreamer: Estimated RAM usage per block is 1.3MiB
DEBUG lazyflow.operators.classifierOperators: Features took 0.448803 seconds. Prediction took 0.368299 seconds. Subregion: start '[0, 0, 0]' stop '[500, 500, 3]'
INFO ilastik.workflows.pixelClassification.pixelClassificationWorkflow: Completed Batch Processing

Process finished with exit code 0

I also can’t figure out how to do my analysis on multiple images at the sam time. I tried some different ways and it didn’t work:

subprocess.run([
    "C:\programs\ilastik-1.3.3post3\ilastik.exe",
    '--headless',
    '--project="C:/Petra/ijs/iCan/pixel_class_2.ilp"',
    '--export_source="Probabilities"',
    '--raw_data="C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0003_comb.tiff" "C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0010_comb.tiff"'
])

So my final wish would be to make a batch analysis, like a whole folder at the same time. In documentation I saw that you can use *, so I tried it and it didn’t work.

'--raw_data="C:/Petra/ijs/iCan/test/e02_*.tiff"'

Thank you all for any help or suggestions.

Hello @petraC,

welcome to the image.sc community! :partying_face:

Great that you already got it working! The warnings you can see there are… warnings. Mostly meant for developers.

The TIFFDecoder warnings you can probably get rid off, by using the latest ilastik beta. We have updated the tiff reading library there.

While ilastik can process multiple files “in one go” I’d probably still recommend doing it one by one. You have a tiny bit more control then at the cost of always starting ilastik + loading the project file (which can add up if you process a lot of files).

so probably something like

from pathlib import Path

# assuming you want to process all tiff files in some folder
in_folder = Path("C:/my_raw/data/folder")
out_folder = Path("C:/some_out/folder")

for in_file in in_folder.glob("*.tiff"):
    # generate some output file name
    out_file = out_folder / in_file.name.replace(".tiff", "_probabilities.tiff")
    print(f"Processing {in_file} -> {out_file}")
    subprocess.run([
        "C:\programs\ilastik-1.3.3post3\ilastik.exe",
        '--headless',
        '--project="C:/Petra/ijs/iCan/pixel_class_2.ilp"',
        '--export_source="Probabilities"',
        f'--raw_data="{in_file}"',
        f'--output_filename_format="{out_file}"' 
    ])

Disclaimer: I did not test this code.

By adding the --output_filename_format argument, you are not dependent anymore on the filename set in the export dialog (and who remembers that).

Including the * character would make ilastik to try to load you files as a single stack.

Thank you so much for your reply!
Your code works great :slight_smile:

I will probably have a lot of data and time may be an issue. Could you sill tell me how to run it on multiple files. Is there maybe a way to import al the files in a forme of an array like:

all_files = [image1.tiff, image.tiff, image.tiff]
...
f'--raw_data="{all_files}"

hmm… I guess the key here would be to omit the equals sign in --raw_data and list the files after so…

all_files = ["image1.tiff", "image.tiff", "image.tiff"]
...
files = " ".join(f'"{f}"' for f in all_files)

f'--raw_data {files}'

you’d need to specify output filenames differently I guess… something like

# note that this is _not_ an fstring, intentionally, so that {nickname}
# is passed to ilastik as is -> as a magic placeholder.
out_file = out_folder / "{nickname}_probabilities.tiff"

Thank you!!!
I know this is not really connected, but I have one more question. I’m now trying to do object classification with prediction mps. My current code is not working, and according to documentation I think I need to add something to the prediction map path. In documentation there is example for segmentation

My code

subprocess.run([
        "C:\programs\ilastik-1.3.3post3\ilastik.exe",
        '--headless',
        '--project="C:/Petra/ijs/iCan/object_class_pred_map.ilp"',
        '--export_source="Object Predictions"',
        '--raw_data="C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0010_comb.tiff"',
        '--prediction_maps="C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0010_comb_Probabilities.h5"'
    ])

hello @petraC,

hdf5 is a file format that behaves like a file system internally. You can have groups there (roughly corresponding to folders on a file system) and datasets (roughly corresponding to files). The semantics to specify an internal path to a hdf5 file would be myfile.h5/group_name/dataset_name.

Could you try adding the internal path to the dataset in the file. Per default the dataset is saved to exported_data (so not groups are added per default, the dataset is crated at the root of the h5 file). So it would be C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0010_comb_Probabilities.h5/exported_data

Cheers
Dominik

Hi.

I tried adding exported_data, but it still doesn’t work. I also tried with different Dataset name. I’m now including some errors I get

Exception: Invalid command line arguments: All roles must be configured explicitly.
Use the following flags to specify which files are matched with which inputs:
--raw_data, --prediction_maps, --atlas

RuntimeError: Could not create instance of '<class 'ilastik.workflows.objectClassification.objectClassificationWorkflow.ObjectClassificationWorkflowPrediction'>'
args   = (<ilastik.shell.headless.headlessShell.HeadlessShell object at 0x0000020FB4577AC8>, True, ['--export_source=Object', 'Predictions', '--raw_data=C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0010_comb.tiff', '--prediction_maps=C:/Petra/ijs/iCan/test/prob.h5/test'], [])
kwargs = {}
The exception was:
Invalid command line arguments: All roles must be configured explicitly.
Use the following flags to specify which files are matched with which inputs:
--raw_data, --prediction_maps, --atlas

Exception: Invalid command line arguments: All roles must be configured explicitly.
Use the following flags to specify which files are matched with which inputs:
--raw_data, --prediction_maps, --atlas


ilastik error: 
The operation completed successfully.

Hi @petraC

could you maybe post the code you executed that produced this error?

This is the code

subprocess.run([
        "C:/programs/ilastik-1.3.3post3/ilastik.exe",
        '--headless',
        '--project="C:/Petra/ijs/iCan/object_class_pred_map.ilp"',
        '--export_source="Object Predictions"',
        '--raw_data="C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0010_comb.tiff"',
        '--prediction_maps="C:/Petra/ijs/iCan/test/e02_Draq5_MTO__sc_xy0010_comb_Probabilities.h5/exported_data"'
    ])

I also find the problem, if I remove export_source it works ok. I assume the problem is the space. I know that Object Predictions is the default, but it would be still good if I could select anything else.