Nuclei selection and segmentation in FIJI

I have several images of some intestinal adenomas and I want to analyze the characteristics of the nuclei in these tissues, but I am having trouble isolating these nuclei from the rest of the image. It seems the problems are that the nuclei are similar in color to the rest of the image, and the segmentation tools I’ve been using are not separating the overlapping nuclei correctly. After isolating and separating the nuclei, I want to be able to analyze those particles for different characteristics (area, min/max gray value, mean gray value, kurtosis, etc.), so I think I also need to preserve the color of the nuclei, or at least have them in gray scale when they are analyzed.

Because the nuclei are similar in color to the rest of the image, the color threshold tool has not provided specific selection of the nuclei. I have also tried using the wand tool and the versatile wand tool, but again it is not very specific. I tried converting the color images to binary, making a mask, and using the built in watershed tool, but the watershed technique does not accurately segment the nuclei. Then I tried making a mask and then manually drawing in the lines with the freehand line tool and using the draw function to segment each nuclei, but this is not efficient. I tried using the trainable weka segmentation tool which was slightly better at selecting the nuclei, but then segmenting the nuclei with watershed was still a problem. A lot of the methods I have been looking at require the images to be in gray scale, so I also acquired images in gray scale, but these have been just as difficult to work with. I have also tried just directly converting the original image to a mask, without thresholding first, and then using the overlay tool. This method has brought me the closest to being able to analyze the nuclei, but once I try to run “analyze particles” it says that no particles have been detected.


  1. Is there a method to separate an object from the rest of an image when they share similar color
  2. Is there an alternative for watershed segmentation
  3. Is there a way to do this and analyze the nuclei in color or in gray scale

Thanks to anyone who can help!

Above is what one of the original images looks like

This is what the create mask tool produces.

This is the mask with me trying to manually separate the nuclei with the freehand draw tool and the draw function…

Dear Ning,

your images are very difficult to analyze since most of the difference between nuclei and background will be based on color and texture. So I think greyscale images and intensity thresholding will not give you good results.

Maybe you could give ilastik a try:
This program is better suited for histology images. It will be very similar to the weka segmentation but offers a bit more options for costumization. I think there should be also some object splitting options in there that are maybe better suited for your task. But for some nuclei clusters an automatic segmentation would still be problematic.

So manual curation would probably still be necessary. Maybe think of a strategy to only segment a sufficient number of your nuclei. Depending on the experimental setup it might be sufficient to measure only a few randomly selected nuclei. For example using stereological principles:
This will give you also very robust quantitative information that, depending on the application, might surpass an automatic segmentation.

As for the error with the particle analyzer you mentioned. It usually comes for the fact that the mask is inverted or your settings for foreground and background are not matching the mask. Set the foreground and background in Fiji here to match your image and make sure that foreground pixels are 255 in an 8-bit image:
Edit > Options > Colors…
Process > Binary > Options…



That’s what I feared…I’ll definitely try ilastik. I’ve been taking out the parts of the image that do not show the full nuclei, which cuts down on how many I have to segment. And I tried your fix for the particle analyzer, and it seems to have worked!

Thank you so much for your help!

Dear Ning,

You might want to also check out QuPath -
It is optimized for histology images and it’s super user friendly.

Happy segmenting,

1 Like


Thank you for the advice! I’ve downloaded QuPath, and so far it has been pretty great, as well!