Ilastik batch processing of large number of electron microscopy images


I am using ilastik for crystal localization in electron microscopy. Currently I have trained a pixel classification and object classification project and I am now trying to batch process a large dataset consisting of 105 images. I found out in the documentation that I have to specify the names of the files in sequence, therefore currently my command line looks like this (I’m using .tiff as input currently as this is easy for my application):

'run-ilastik.bat --headless --project=pixelclassification_train.ilp --output_filename_format=output_ilastik_2019-10-08_14-32-24/{nickname}_{result_type}.h5 --export_source=“Probabilities” --raw_data Test_data/mrc_00000.tiff Test_data/mrc_00001.tiff Test_data/mrc_00002.tiff …

This works fine for pixel classification, but when doing automatic batch processing for object classification I have to also feed the probabilities which result in a very large string (over 9000 characters) which cannot be accepted due to command-line string limitation.

I have feeded a text file including a file string in the command but this is not recognized by ilastik. I have also tried –raw_data Test_data/*.tiff’ to point to all the files but this is seen by ilastik as a stack input which gives dimension errors.

Is there another way to provide the raw data and probabilities and limit the string length?

Kind regards, Arent

Hi @a-kievits017,

great that you want to batch process so many images! And of course this is possible via the command line.

You are right that adding --raw_data Test_data/*.tiff will (on windows) be interpreted as a stack. So you’ll have to list your images individually like you have done already:

--raw_data Test_data/mrc_00000.tiff Test_data/mrc_00001.tiff Test_data/mrc_00002.tiff …

in order to make it work with object classification, you’d also have to give the prediction maps like

--prediction_maps Test_data/mrc_00000_Prediction.tiff Test_data/mrc_00001_Prediction.tiff ...

As you have pointed out you might hit the maximum length (8191 characters) (specifying 200 files would only leave ~40 chars per file). Using a text file is a great idea, we should implement that in ilastik. Right now this is not possible.

Other users often use Python to programmatically start ilastik with one or a few datasets at a time. See e.g. this thread: Ilastik speed normal mode VS headless mode

Have you, as an alternative, considered just using the batch processing applet in ilastik?

Dear @k-dominik,

Thank you very much for your reply. Currently we are using ilastik in an automatic data acquisition pipeline in the microscope setup where everything is controlled through Python. Ideally we want to acquire data and therefore call ilastik via the command line automatically to process because measurements can be quite long.

I did not read the thread yet about calling ilastik multiple times to split the data processing (thank you for pointing that out). For now we will try to use that approach if we have to process more images than the command line can cope with. I think it would be great if an option with a text file becomes available in the future.

Thank you for you help,