Ilastik speed normal mode VS headless mode

Image Analysis
, ,
#1

Hi everyone,

After giving up with Knime (Read ilastik ouput on knime), I managed to run Ilastik headless mode with Python (How to call and run ilastik from python?) thanks to @CellKai 's kind reply.
However, it appears that it takes much more time to process my datasets (each dataset contains about 100 images). With “normal” mode, it takes 1 minute per dataset, but with this script (see below), each dataset takes 10 minutes to be processed… and I have thousands of datasets per experiment.

import os
from os import listdir
import subprocess

ilastik_location = 'path/to/ilastik'
ilastik_project = 'path/to/project.ilp'


for dataset in range(first_set, last_set):

    indir='path/to/dataset/'
    infiles = os.listdir(indir)


    os.chdir(ilastik_location)


    for infile in infiles:
        if infile[-4:] != '.tif':
            print ("skipping %s" % infile)
            continue

        command = 'run-ilastik.bat --headless --project="%s" --output_filename_format="%s/{nickname}_results.h5"--export_source="Probabilities" --raw_data="%s%s"' % (
            ilastik_project,
            indir,
            indir,
            infile)
        print ("\n\n%s" % command)
        subprocess.call(command, shell=True)

Please, could you let me know if you have any clue about the way to increase speed of processing ?

Kindly yours !

Ilastik batch processing of large number of electron microscopy images
#2

Hi @chalbiophysics,

could you maybe paste the output of ilasitik for one run here? (you could use output = subprocess.check_output(command) instead of your subprocess.call and then just print the output…
Maybe that can explain something. Otherwise there should not be a reason this should be any slower than in gui mode (on the contrary…)

#3

Hi @chalbiophysics,

if I correctly understand your code, you are calling Ilastik on each individual image of each dataset. In principle you can however pass multiple images to a single call like this:

run-ilastik.bat --headless --project=my_project.ilp image1.tiff image2.tiff image3.tiff

I did a quick and dirty check for 4 images, once calling ilastik on each image independently and once passing all of them in one call and saw a difference in speed of ~20%. I don’t have a clear picture of what happens in the background so maybe this is not relevant. @k-dominik surely knows more ?

Cheers,
Guillaume

2 Likes
#4

Hi everyone,

Thanks @k-dominik @guiwitz for your great advices. After several hours of failed attempts, I managed to get this script:

ilastik_location = 'path/to/ilastik'
ilastik_project = 'path/to/project.ilp'

for dataset in range(first_set, last_set):

    indir='path/to/dataset/'
    infiles = os.listdir(indir)
    
    list_img = []
    for i in infiles:
        liste_img.append(str(indir) + str(i))

    list_img = ' '.join(list_img)

    command = 'run-ilastik.bat --headless --project="%s"  --output_filename_format="{dataset_dir}/{nickname}_results.h5" %s ' % (
        ilastik_project,
        list_img)
    print ("\n\n%s" % command)
    subprocess.call(command, shell=True)

Indeed, it runs really faster… but only if I have few images (circa 20) in my dataset. If I process all images of my dataset (113), it is not working anymore. I figured out that the problem is the limit of the command line (above 8000 characters it doesn’t work).

So I wanted to use the “stack input option” as described here (https://www.ilastik.org/documentation/basics/headless.html):


ilastik_location = 'path/to/ilastik'
ilastik_project = 'path/to/project.ilp'

for dataset in range(first_set, last_set):

    indir='path/to/dataset/'
    infiles = os.listdir(indir)
    
    list_img = []
    for i in infiles:
        liste_img.append(str(indir) + str(i))

    list_img = ' '.join(list_img)

    command = 'run-ilastik.bat --headless --project="%s"  --output_filename_format="{dataset_dir}/{nickname}_results.h5" "%simage_dataset_number_*.tif" ' % (
        ilastik_project,
        indir)
    print ("\n\n%s" % command)
    subprocess.call(command, shell=True)

But it’s not working. Am I badly calling my dataset ? In the documentation webpage, there is a “~” before the pathway of the dataset and I don’t really understand what it stands for. Please, any help would be very appreciated ! (I am really close to my goal, I feel it ^^)

Kindly !

#5

Hey @guiwitz, yes, it should definitely result in a speedup since the project has to be only loaded once for the images you have supplied.

#6

the stack option is used in order to process images that consist of more than a single file (think 3D stacks, where each z-slice is in a single file). You can not really use this option to just process more files.

I would still be really interested in the output that you are getting, since a 2x slowdown is absolutely not expected.

another thing you could try is invoke ilastik always with a couple of files. Something like this should do the trick:

for dataset in range(first_set, last_set):

    indir='path/to/dataset/'
    infiles = os.listdir(indir)
    
    list_img = []

    n_images = 10
    for i in range(0, len(infiles), n_images):
        list_img.append(" ".join(infiles[i:i+n_images]))

    for images in list_img:
        command = 'run-ilastik.bat --headless --project="%s"  --output_filename_format="{dataset_dir}/{nickname}_results.h5" %s' % (
            ilastik_project,
            images)
        print("\n\n%s" % command)
        subprocess.call(command, shell=True)

the snippet above would process 10 images (n_images) at a time.

#7

Hi everyone,
Late and short reply to let you know that the script of @k-dominik works fine.
By shortening the path and the name of your images, you can make less batchs of images and increase speed of processing.
Thanks again !

#8

Hi,
I am curious how much did you improve in time per dataset? I feed ilastik by sets of about 20 images of size 2752x2208 in png-format and (3) channel gray scale images (r=g=b). However time per image is about 10s per image for pixel classification in our workstation HP Z4 with decent CPU and running with decent CPU usage. @k-dominik is there by any means possible to get more info about the calculation performance when running in headless mode? I am very happy about the result but it would be even better if one could reach there in much shorter time :slight_smile:

Cheers,
Fredrik

#9

Hi @Fredrik,

what are you interested in exactly? What do you mean by performance. There is a log file (in your home folder, called ilastik_log.txt that will have some information about running times of certain steps. Also all logs are prefixed with time-stamps that might help indicate durations.

#10

Hi,
According to the time stamps the analysing part is divided roughly into:
“Features took 1.999829 seconds, Prediction took 5.391919 seconds for roi: [0, 0, 0] : [2208, 2752, 2]”
This is taken from a real case with 10 images with total ilastik-related analyzing time of approx 90 s. Since I feel the results are getting better with the training and prediction made by ilastik I want to make use of it more. However, the old and ordinary segmention part has an anlyzing time of only ~0.5 s per image leaving you with the occasionally difficult decision between qualtity and time :slight_smile: . I have followed the discussion around CPU/GPU but I just wonder if there still might be any modules taking part in the prediction step that maybe not yet fully have been optimized? Something left that might be switched from Python to C, which I can hope for :slight_smile: ?

Cheers
Fredrik

#11

Hi @Fredrik,

our filters are already highly optimized (although also there time in CPU tech development didn’t stand still and one might improve that further).

Have you ever looked into the “Suggest features” functionality? It helps you a bit with the tradeoff of quality and time there.

Cheers
Dominik

#12

out of curiosity - what kind of CPU usage do you see in batch processing @Fredrik?

#13

Hi,
This is what I get in our Intel Xeon E5 (3 GHz). Its easy to see the pattern for each image :slight_smile:

CPU usage
CPU usage3245×808 126 KB

#14

Thank you k-dominik!

I have conducted some tests with the suggested features. I see some minor speed improvements for less features and/or changing from Filter Method to Gini Importance while keeping an acceptable quality and that is good (but more would have been even nicer). When it comes to the Wrapper Method and the set size penalty I do not really manage to get something out from it. I tried a couple of different penalty settings in the range 0.1 - 1.0 but decided to close it after 10 minutes while still running.

Cheers,
Fredrik