Elongated and overlapping cell segmentation challenge

Hello,

@constantinpape @fjug @akreshuk @Sebastien @iarganda @mweigert @bcimini

We would like to know if some of you are aware to solutions that would help to segment cells in images that look like the one below.

Example image

Gray: Transmission
Magenta: Nuclei
Arrow: highlights crossing of two cells

Raw data

im.zip (3.4 MB)

Challenges

The main challenge is that the cells are crossing each other with their very long protrusions.
So we would need some logic of pixels belonging to more than one cell. And also there would need to be a logic that after a crossing the cell should continue kind of straight such that after the crossing it still has the correct label.

We are very thankful for any pointers!

Christian & Veronika

1 Like

Hi @Christian_Tischer,

looks like an interesting segmentation challenge. One question:
Do you really need multiple ids per pixel or would it be ok, if objects are not spatially connected? I.e. in a crossing the pixel would (ideally) get the id of the cell on top, but the cell below would continue and only be “occluded” by the upper cell.

In terms of segmentation approaches, I can’t really think of anything “classical” that would work right now, would be very interested to hear if anyone has some non Deep Learning ideas.

In terms of Deep Learning, I can think of two approaches:

  • Predict pixel embeddings and use a clustering algorithm that does not enforce spatial connectivity.
    The basic approach comes from this paper and this paper uses it for a conceptually similar application. With this approach, you would obtain a single id per pixel and objects would be occluded at crossings.
  • Use a mask r-cnn like approach, which would allow multiple ids per pixel, because the segmentations derived from the masks can overlap.

This is not particularly helpful from an open source point of view, but it did remind me of an advertisement I had seen for MIPAR, which keeps showing up in my LinkedIn feeds.


Scroll to the bottom to see both a “traditional detection” and “deep learning detection” (which looks worse to me) for nanofibers.

If you are in a rush and want to try something like that, maybe they could give you a demo. I don’t know anything about the company, though.

Hi Christian,

It’s always a challenging problem!

A few years ago now I wrote a plugin called IdentifyLinearObjects back in CellProfiler2 (based heavily on CellProfiler’s IdentifyDeadWorms module); it handles overlapping just fine, but assumes the whole object is linear, not just part of it. You could perhaps use it to call processes, then relate the processes to the somas, but it’s a definitely a far-from-perfect solution.

1 Like

Could you comment what you mean by this? Would the object ids before and after the crossing be the correct ones?

Very good point! I fact, now that you say it: Probably not! It would be ok if the pixels at the crossing point are randomly assigned to one of the cells.

Would the object ids before and after the crossing be the correct ones?

Yup! Rather than storing the segmentations in a traditional label matrix, where each pixel is definitively assigned to 1 object, it stores each object’s segmentations in a separate layer, so that the same pixel can be assigned to many objects. Because we enforce that it should try to make it look like a straight line, an x shaped object usually successfully breaks down correctly into the two corresponding lines- see Great Overlaid Rigid Rod Segmentation Challenge.

3 Likes

Judging from the one image I’d say it should be possible. Still, I think it would come with some major effort since the only solutions I can think of to automate this problem are not simple and require some serious ‘love’ to be parametrized usefully.

DISCLAIMER: below this line I just spell out what comes to my mind. Giving the problem more thought might make me change all uttered opinions…

I would try to:

  • Perform a foreground/background segmentation in one of 1000 known ways.
  • Then use the fluorescent nuclei (or whatever it is) as seed points, constraining the space of all possible solutions only to those that have precisely as many segmented objects as one detects nuclei.
  • Use one of many existing methods developed in the context of e.g. blood-vessel segmentation or connectomics. What comes to (my) mind are the methods by Björn Andres or Pascal Fua (et al., of course). These methods can in principle incorporate constraints that express that cells, when crossing, continue on straight.

Here one such paper: Türetken, E., Benmansour, F., Andres, B., Pfister, H., & Fua, P. (2013). Reconstructing Loopy Curvilinear Structures Using Integer Programming. Presented at the CVPR '13: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society.

If anybody reads this and thinks: good, this Florian guy must be quite old… I’d be happy to hear about more modern approaches for such problems… please post your ideas! :slight_smile:

Best,
Florian

2 Likes

@bcimini @v_saharuka

Do you have an idea why that is?

When you load it, is there an error message?

What do you mean exactly? I open CellProfiler on a Mac, this gives me no error messages during the launching of CellProfiler. But then I cannot find the additional module.

Is there a direct way to load the module in CellProfiler 2.2.0 UI?

I also tried starting CellProfiler from the terminal to see whether there are error messages using
open -a "CellProfiler.app"
However this does not work. I get an error appearing in an UI window of the Mac, but no error message. What does work is for example "open -a “CellProfiler-3.1.8.app”, which is the newer version.

Is there a way to start CellProfiler 2.2.0 from command line on a Mac in a way that one can see error messages?