How to call and run ilastik from python?

ilastik

#1

Hello,

I really like the headless mode of Ilastik to work with and I’m wondering how to call and run ilastik properly (as binary?) on python ?
I want to use the batch processing of my classifier in a python code to make some loops.
If you have some working tips i’m aware !

Cheers and Happy new year,
Emilie


#2

Hey Emilie,

you can call ilastik with Python’s subprocess module, (see maybe here for a general tutorial on using this python module.)

We have documented the headless interface here: https://www.ilastik.org/documentation/basics/headless.html

Let us know if you need any help assembling the correct command line parameters. What workflow are you using?

Cheers and a happy new year to you, too
Dominik


#3

Hey Dominik,

Thank you for your quick answer, as soon i will try i will maybe ask you some help !

Cheers !

Emilie


#4

Hello,

I can use subprocess to call ilastik but after to call my classifier in the good order i’m not sure i’m doing well.
I try that:
#import os

import subprocess

##to understand how that’s work https://docs.python.org/3/library/subprocess.html

subprocess.run("/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/run_ilastik.sh")

That’s work with my path to open Ilastik. So ok !

But how should I call the headless properly ?

When i run it in my terminal in my wd i use simply the batch mode with one label choose( here the channel 7) for one image like that (I simplify my path):

$ ./run_ilastik.sh --headless --output_format=numpy --cutout_subregion="[(None, None,7), (None, None,8)]" --project="/media/myproject.ilp" “/media/superpicture.png”
So naively
i tried like that:
subprocess.run(["/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/run_ilastik.sh","–headless --output_format=numpy --cutout_subregion="[(None, None,7), (None, None,8)]" --project="/media/myproject.ilp" “/media/superpicture.png”"])

Or also with the run_ilastik.sh before the --headless-- as the terminal but i get a syntax error of course.
How should I assemble my command for the headless mode to run properly the subprocess in python ?

Sorry my queston is probably low level i’m really beginner.

Regards,

Emilie


#5

You need to comma separate all the arguments in the list you pass to subprocess.run.
I also have an example using ilastik and subprocess. (check_call has more or less the same syntax as run.)


#6

oK NICE !!
Thank you !!
I got this error now but no clue
TypeError(“bufsize must be an integer” )but i don’t know what is the bufsize --’

Thanks for yor quick help already !
;D


#7

Maybe we can help if you paste the exact command you used and the whole error stack-trace here.


#8

Hi,
Sorry for the long path for my images acces.

import subprocess

subprocess.run("/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/run_ilastik.sh", '--headless', '--output_format=numpy',' --cutout_subregion="[(None, None,7), (None, None,8)]"','--project="/media/emfro/Photos_Emilie/For Betty/Ilastik/Betty_t4_20122019.ilp" "/media/emfro/Photos_Emilie/For Betty/imagesForTest_cv/emfro-noramdom2_11_cropped_2_labels.png""')

Sorry for the long path:
This is my error

Traceback (most recent call last):
  File "/home/emfro/Documents/Rhizoline_analysis/Emilie_scripts/Subprocess_ilastik_emfro.py", line 12, in <module>
    subprocess.run("/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/run_ilastik.sh", '--headless', '--output_format=numpy',' --cutout_subregion="[(None, None,7), (None, None,8)]"','--project="/media/emfro/Photos_Emilie/For Betty/Ilastik/Betty_t4_20122019.ilp" "/media/emfro/Photos_Emilie/For Betty/imagesForTest_cv/emfro-noramdom2_11_cropped_2_labels.png""')
  File "/home/emfro/anaconda3/envs/Virtualenv_OpenCV3/lib/python3.7/subprocess.py", line 453, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/emfro/anaconda3/envs/Virtualenv_OpenCV3/lib/python3.7/subprocess.py", line 658, in __init__
    raise TypeError("bufsize must be an integer")
TypeError: bufsize must be an integer

Bufsize is not in ilastik documentation the problem is from my subprocess module
and i fund that if that can help : if somebody can explain me the function of this bufsize
I don’t understand the meaning

bufsize, if given, has the same meaning as the corresponding argument to the built-in open() function: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size. A negative bufsize means to use the system default, which usually means fully buffered. The default value for bufsize is 0 (unbuffered).


#9

Hey hey,

you need to enclose the arguments in a list like this:

subprocess.run([
    "/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/run_ilastik.sh",
    '--headless',
    '--output_format=numpy',
    '--cutout_subregion="[(None, None,7),(None, None,8)]"',
    '--project="/media/emfro/Photos_Emilie/For Betty/Ilastik/Betty_t4_20122019.ilp"',
    '--raw_data="/media/emfro/Photos_Emilie/For Betty/imagesForTest_cv/emfro-noramdom2_11_cropped_2_labels.png"'
])

note that I also removed a space in the cutout subregion (there is a bug in python’s argparse) and also explicitly specified the --raw_data as input (at the moment this is not needed, but it might be in the future).

by not enclosing the arguments for the process in a list, your strings all got interpreted as arguments to the run function. One of those is bufsize and it expects an integer. More info on the buffer in pipes: https://en.wikipedia.org/wiki/Pipeline_(Unix)#Implementation


#10

Thank you for the documentation !
Thanks a lot for the code !
I will try and tell you … what is the next error (I hope it will be just a joke)

Kind Regards,


#11

AAAAANNND IT’S NOT ;(
there is indeed the bug for argparse.

import os

import subprocess
##to understand how that’s work https://docs.python.org/3/library/subprocess.html

subprocess.run([
… “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/run_ilastik.sh”,
… ‘–headless’,
… ‘–output_format=tiff’, #try in tiff first
… ‘–cutout_subregion="[(None, None,7),(None, None,8)]"’,
… ‘–project="/home/emfro/Documents/Ilastik/ProjectRhizoline_classifierlabel8.ilp"’,
… ‘–raw_data="/home/emfro/Documents/Rhizoline_analysis/ImagesToTests/Rootscan_croppedwithPython/emfro-noramdom2_9_1DMONO_S0_0_0_20180305071422_cropped.png"’
… ])
INFO lazyflow.operators.filterOperators: Using fast filters.
WARNING init.py(11): UserWarning: init: Could not import tiktorch classifier
Traceback (most recent call last):
File “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/ilastik-meta/ilastik/ilastik.py”, line 123, in
main()
File “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/ilastik-meta/ilastik/ilastik.py”, line 108, in main
hShell = ilastik_main.main(parsed_args, workflow_cmdline_args)
File “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/ilastik-meta/ilastik/ilastik_main.py”, line 107, in main
load_fn = _prepare_auto_open_project(parsed_args)
File “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/ilastik-meta/ilastik/ilastik_main.py”, line 365, in _prepare_auto_open_project
parsed_args.project + “’ does not exist.”)
RuntimeError: Project file ‘"/home/emfro/Documents/Ilastik/ProjectRhizoline_classifierlabel8.ilp"’ does not exist.
CompletedProcess(args=[’/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/run_ilastik.sh’, ‘–headless’, ‘–output_format=tiff’, ‘–cutout_subregion="[(None, None,7),(None, None,8)]"’, ‘–project="/home/emfro/Documents/Ilastik/ProjectRhizoline_classifierlabel8.ilp"’, ‘–raw_data="/home/emfro/Documents/Rhizoline_analysis/ImagesToTests/Rootscan_croppedwithPython/emfro-noramdom2_9_1DMONO_S0_0_0_20180305071422_cropped.png"’], returncode=1)

so File “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux/ilastik-meta/ilastik/ilastik_main.py”, line 365, in _prepare_auto_open_project
** parsed_args.project + “’ does not exist.”)**

What should I change ;/ Sorry again.


#12

what you are seeing is not related to a bug in argparse I had mentioned.

it seems that ilastik cannot find the project file. I assume you have already checked that the path "/home/emfro/Documents/Ilastik/ProjectRhizoline_classifierlabel8.ilp" is really correct (paths are case-sensitive).


#13

Yes i checked and it’s run perfectly on linux :wink:


#14

Hi,
sorry that I jump into this thread but @ehrenfeu very recently wrote a little python wrapper to run ilastik and I had the feeling it could be of interest here.
It is essentially what you already have, but written to run on Windows7.

import os
import subprocess


ilastik_location = 'C:\Program Files\ilastik-1.3.2rc2'
ilastik_project = 'd:/ilastik/myproject.ilp'

indir = 'd:/data/mydataset/'
infiles = os.listdir(indir)


os.chdir(ilastik_location)
for infile in infiles:
    if infile[-4:] != '.tif':
        print "skipping %s" % infile
        continue

    command = 'run-ilastik.bat --headless --project="%s" --export_source="Object Predictions" --raw_data="%s%s"' % (
        ilastik_project,
        indir,
        infile)
    print "\n\n%s" % command
    subprocess.call(command, shell=True)


#15

I suppress the first / to the pass and i’m stuck with the --cutout_subregion but if i change the format
it will be not correct anymore i guess. ;/

INFO lazyflow.operators.filterOperators: Using fast filters.
WARNING init.py(11): UserWarning: init: Could not import tiktorch classifier
INFO ilastik_main: Starting ilastik from “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux”.
Starting ilastik from “/home/emfro/Documents/Ilastik/ilastik-1.3.2rc2-Linux”.
WARNING cross_validation.py(41): DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
WARNING 2019-01-23 13:23:00,240 opConservationTracking 16962 139711228589888 Could not find any ILP solver
WARNING 2019-01-23 13:23:00,254 opStructuredTracking 16962 139711228589888 Could not find any ILP solver
WARNING 2019-01-23 13:23:00,259 structuredTrackingWorkflow 16962 139711228589888 Could not find any learning solver. Tracking will use flow-based solver (DPCT). Learningfor tracking will be disabled!
INFO ilastik.shell.projectManager: Opening Project: /home/emfro/Documents/Ilastik/Betty_t4_12122018.ilp
WARNING stype.py(181): UserWarning: ArrayLike.isCompatible: FIXME here
usage: ilastik.py [-h] [–cutout_subregion CUTOUT_SUBREGION]
[–pipeline_result_drange PIPELINE_RESULT_DRANGE]
[–export_drange EXPORT_DRANGE]
[–export_dtype {uint8,uint16,uint32,int8,int16,int32,float32,float64}]
[–output_axis_order OUTPUT_AXIS_ORDER]
[–output_format {bmp,gif,hdr,jpeg,jpg,pbm,pgm,png,pnm,ppm,ras,tif,tiff,xv,bmp sequence,gif sequence,hdr sequence,jpeg sequence,jpg sequence,pbm sequence,pgm sequence,png sequence,pnm sequence,ppm sequence,ras sequence,tif sequence,tiff sequence,xv sequence,multipage tiff,multipage tiff sequence,hdf5,compressed hdf5,numpy,dvid,blockwise hdf5}]
[–output_filename_format OUTPUT_FILENAME_FORMAT]
[–output_internal_path OUTPUT_INTERNAL_PATH]
[–export_source EXPORT_SOURCE] [–table_only]
ilastik.py: error: argument --cutout_subregion: Expected a list encoded as a string, e.g. ‘[0, 1]’ but got ’ “[(None, None,7),(None, None,8)]”’ -> type <class ‘str’>.


#16

Slightly confused. Could you state again what you are trying to achieve? So is it working now?

Again, not sure how you got there. With the command line I have suggested, or are you using @CellKai’s script?


#17

The issue here is that ilastik trys to parse the cutout_subregion as a json string, see
https://github.com/ilastik/ilastik/blob/cacbaa23f01af20266d9c27ae2bc7eb6db886106/ilastik/utility/commandLineProcessing.py#L58.

However, the value you are giving can not be parsed as json:

$ python - c "import json; x=json.loads('[(None, None,7),(None, None,8)]'); print(x)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cpape/Work/software/conda/miniconda3/envs/main/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/cpape/Work/software/conda/miniconda3/envs/main/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/cpape/Work/software/conda/miniconda3/envs/main/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)

The issue is that None and () are not valid json syntax. You need to use null and [] instead:

$ python - c "import json; x=json.loads('[[null, null, 7], [null, null, 8]]'); print(x)"
[[None, None, 7], [None, None, 8]]

#18

It was with your command line, @CellKai’s i didn’t try yet. SO it’s a problem with the way i encode the --cutout_subregion.
I will read the commant from@constantinpape.


#19

it will work with None allright, see here

@emfro08 could you try removing the " in your arguments: ‘–cutout_subregion=[(None, None,7),(None, None,8)]’,


#20

Oh, I didn’t see that.
Still for some reason the json decoder borks…