Tracking error on large image stack

Hi,

I’m running an ilastik tracking project with pixel prediction maps that fails on my full 191 image stack but completes when I only run the first 114 images. Here are the last few lines from the execution and the first few of the error dump:

Flow-based tracking solver
[====================] 100%
Merger resolution
INFO ilastik.applets.tracking.conservation.opConservationTracking: Resolving mergers.
ERROR 2019-01-16 18:09:55,817 log_exception 20905 140218469783360 Traceback (most recent call last):
File “/home/users/dane/ilastik-1.3.2b3-Linux/ilastik-meta/ilastik/ilastik/shell/projectManager.py”, line 441, in _loadProject
self.workflow.onProjectLoaded( self )

I’d be happy to share the images that are causing the problem. The raw image stack is 800Mb and the probability maps are 4.6Gb.

thanks,
Mark

Hey @MarkDane,

that is really unfortunate. In order to reproduce it here, it would indeed be very helpful to get access to your raw data. If you could send an email with links to team@ilastik.org we’ll have a look.

Am I assuming right, that you are using the batch mode? Or is that in the gui?

Cheers
Dominik

Hi Dominik,

I sent an email to team@ilastik.org with a link to my data.

This is happening in headless mode. Thanks for your help.

Mark

Hi Mark,

I’ve ran your projects on your data (first pixel classification to generate the prob maps) and then the tracking. The tracking also didn’t run through for me with an error like the following:

  File ".../ilastik-1.3.2-Linux/lib/python3.6/site-packages/hytra/core/mergerresolver.py", line 94, in addNode
    assert(not division)
AssertionError

Do you get something similar? (Since it doesn’t really match what you have pasted in the original post).

1 Like

Hi Dominik,

It looks like you got the same error as me… I included a nohup.out file in the same google drive folder as the data.

File “/home/users/dane/ilastik-1.3.2b3-Linux/lib/python3.6/site-packages/hytra/core/mergerresolver.py”, line 94, in addNode
assert(not division)
AssertionError

The tracker project runs successfully on the first 48 images. It failed with the same error on 96+ images and I didin’t attempt sequence stacks between those two sizes.

thanks,
Mark

Hi @MarkDane,

so I’m still on it. It seems that the Problem that you ran into is not that uncommon. The root is that an object has been simultaneously classified as division, as well as a merger (those two classifiers are trained independently). I’m thinking about how to check for that before tracking. In the meantime the recommendation is to go back to training your two classifiers (division and object count (-> merge)) using the uncertainty layer with the goal to reduce uncertainty. Also you should make sure (and I can only say that in the current state this is difficult) that you don’t label an object a merger (2 or more objects), that you have marked as a division.

Hi @k-dominik,

Thanks for keeping this active. I added labeled objects for both classifiers and still get the same error. Is there any more guidance you can give? For instance, I’ve kept my training data fairly small at just 5 images. Should I include more images and perhaps multiple movies from other wells where cells have different phenotypes?

thanks,
Mark

Arrrr :confused: this is a persistent one.

:frowning:

yes, I would say so. I would even suggest to load the whole data and check if predictions are persistent through the time-series as conditions usually change. Is there a reason you only want to train on the first 5 time-frames?

I’ve kept the training set small just to minimize loading times. In general, I’m insure about the optimum training process. I was over my travel budget when I realized the opportunity in last year’s I2K meeting. Are there workshops or conferences coming up where ilastik will be presented?

I’ll load the whole data set, train on it and see how that effects my results.

thanks,
Mark

Hey Mark,

we haven’t planned anything overseas, but we plan to have an ilastik training in Heidelberg, Germany later this year. Once we have anything concrete we will announce it in the forum.

Cheers
Dominik

Happier days here. I used one entire movie as the training data and have applied the tracking classifier to 12 other movies without crashes. I’m moving on to assess the quality of the results.

I’ll implement any changes you develop. I have many other movies to analyze and will see if this merger/division problem occurs in any of them.

thanks for your help,
Mark

1 Like

That’s great to hear! Thanks for keeping the faith :slight_smile: