Tracking project headless mode argument parsing issues

Hello, again,

Recently, I became an extensive ilastik user.

I love it.

However, I’ve got some issues with the headless mode, and I want to discuss it here:

  1. Tracking project needs the --export_source="Object-Identities" to be explicitly specified, otherwise it raises this assertion error:
ERROR 2019-04-04 16:02:46,293 log_exception 26896 139861895284544 Traceback (most recent call last):
  File "/home/tongli/Documents/ilastik-meta/ilastik/ilastik/shell/projectManager.py", line 441, in _loadProject
    self.workflow.onProjectLoaded( self )
  File "/home/tongli/Documents/ilastik-meta/ilastik/ilastik/workflows/tracking/conservation/conservationTrackingWorkflow.py", line 402, in onProjectLoaded
    self.batchProcessingApplet.run_export_from_parsed_args(self._batch_input_args)
  File "/home/tongli/Documents/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 66, in run_export_from_parsed_args
    return self.run_export(role_path_dict, parsed_args.input_axes, sequence_axis=parsed_args.stack_along)
  File "/home/tongli/Documents/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 131, in run_export
    export_to_array=export_to_array)
  File "/home/tongli/Documents/ilastik-meta/ilastik/ilastik/applets/batchProcessing/batchProcessingApplet.py", line 248, in _run_export_with_empty_batch_lane
    assert opDataExportBatchlaneView.ImageToExport.ready()
AssertionError
  1. adding --cutout-subregion="[(xx),(xx)]" gave me this
Exception: Invalid command line arguments: All roles must be configured explicitly.
Use the following flags to specify which files are matched with which inputs:
--raw_data, --prediction_maps

I believe it is linked to this issue. Is there a solution for it ? If no, where should I start to hack it ? I can try to help.



In fact, those issues are related to my final goal which is to
load one multiple ROI file(bounding box/binary mask of subregions saved in .npy .csv or whatever) for one image and batch process multiple images along with their corresponding ROI files.
Something like:

python ilastik.py --headless --project=my.ilp \
--raw_data $RAW_IMGS --prediction_maps $SEG_IMAGES \
--export_source="Object-Identities" \
--cutout-subregion_files=boudingbox1.csv boudingbox2.csv boudingbox3.csv

Each .csv contains:

|regions| axis_0_min | axis_0_max | axis_1_min | axis_1_max |
||------------|------------|------------|------------|
||            |            |            |            |

In the end, I want it return one .h5 per image per roi.
Is it feasible?

BTW, Ilastik is really awesome and it will be even more awesome with this function :wink:

Thanks for the warm words!

this is great detective work. We know about the quirks of our command line interface and are determined to improve it in the future.

The --cutout_subregion thing is so unintuitive that I always have to look into the code to understand what it is doing and how it should be used. Okay, so you need to provide the cutout subregion in the syntax --cutout_subregion="[[axis1_start,axis2_start,...],[axis2_stop,axis2_stop,...]]" as in your data. Only supply as many axes as you have in your data plus the channel axis :slight_smile: (using None, None for start and stop, respectively).

you can script it with the --cutout_subregion syntax only supplying one dataset at a time and adjusting the --cutout_subregion in your script.

Let us know if you need any specific help :slight_smile:

Thank you for your reply. Actually, it is not only --cutout_subregion that doesn’t work.

It’s the same for all the other options such as --output_filename_format or --output-format.
It seems simply I can add nothing.
The only thing I can do with a Tracking project (pixel classification is OK) is this

python $ILASTIK_PATH --headless \
--project=$PROJECT \
--raw_data $RAW_FLUO \
--prediction_maps $SEG_IMAGES

and adding = inbetween, like this --raw_data=$RAW_FLUO is also lethal…
I will try to understand the argparser part. See if I can find something.

That’s what I was planning to do. But I got around 1000 subregions per image. I believe nothing but initializing 1000 time ilastik can be painful.
That’s why I want to let it accept multiple regions for one run.

Have a nice weekend!

maybe it’s better to do that in post processing then?

I thought about that too. The thing is, the tracking results looks promising to me. However, it needs just some improvement. A typical case is when two objects get too close and an error track is produced.

Since I already got the subregion masks. I was thinking why not just track inside my subregions.
In this case, I can reduce a lot of potential errors.

But yes, if adding multiple subregion is also difficult. I think I will go for post processing.

hmm, ah. I see. Do you see better results, using just the subregions?

It can not track outside of my crop image, so yes, it’s better. :smile:

But what you said is actually easier to do and I believe ilastik is not made for the case like my dataset.

Maybe my last chance is to fine-tune the tracking parameter in the applet.
Hope it can get ride of the error tracks.

Unfortunately, I’m kind of stuck finding the CPLEX 12.8…I can only find the 12.9 version. Also, I’m trying to understand the IBM self-nomination thing…

okay, about that. I’ll rebuild our packages linking against the current (12.9) version of cplex next week…

Can’t I build it myself ? Do you have a build tutorial for that ?

well I wouldn’t advice to do that, currently. We’ve changed a bit in our building process and didn’t get to those packages yet.

OK, take your time.
I will try to understand the argparser thing with my entry-level programming skill.

Finally, I figured out what was going wrong with my previous cmd lines.
I want to share my experience here, so that later comers will not struggle like I did.

  1. Always put your options before images. The order counts
    • [options] --raw_data xxx.png => Good
    • --raw_data xxx.png [options] => Bad
  2. Always add the = symbol for the options, otherwise, ilastik will not understand what are the arguments.
    • --output_filename_format={dataset_dir}/{nickname}_tracks => Good
    • --output_filename_format {dataset_dir}/{nickname}_tracks => Bad
  3. When giving the --cutout_subregion, do not add any spaces in the coordinate string.
    • --cutout_subregion="[(0,0),(10,10)]" => Good
    • --cutout_subregion="[(0, 0), (10, 10)]" => Bad
  4. If you want to batch process more than one file, do not add the “=” symbol.
    • --raw_data file1.png file2.png => Good
    • --raw_data=file1.png file2.png => Bad

These points might be obvious for the experienced programmers, but can be a real headache for the one who is not familiar with bash cmds.

2 Likes

@BioinfoTongLI, thank you very much for going through all this trouble. And don’t worry, our headless mode is finicky to use, and we know it (and want to change it too). This is some really helpful stuff!

You’re welcome. I’m glad I can help.