Best practices for quality control of image analysis

Image Analysis
#1

Hi All,

@Julien_Colombelli @haesleinhuepf @Kota @AnneCarpenter @akreshuk @iarganda

I want to create something very basic for beginners for best practices of quality control of image analysis.
I googled a bit but could not find anything. Probably I missed many things?

Things that come to my mind are:

  • For object segmentation:
    • Save images with segmentation overlay
    • Go through the segmentation overlay images and make notes like this:
      • image_001, too many objects detected
      • image_005, objects merged
    • Go back to your image analysis pipeline and try to fix the algorithm for those images (consult an expert in your institute or www.image.sc if you need help).
    • Run the analysis again and check whether it improved! check whether the changes did no make the results worse in other images!
  • Quantification of object instance detection accuracy: TP, FP, FN, …
  • Quantification of region segmentation accuracy: https://en.wikipedia.org/wiki/Jaccard_index
  • If you have more than 100 images, going through all of them might be too much. Then you need an efficient way to only look at “outliers”, such as something like this: https://github.com/hmbotelho/shinyHTM

a.s.o., this is not meant to be comprehensive, but just to indicate the level I would like to target.

I am super thankful for any hints on this subject!

Best, Christian

5 Likes
#2

Good day Christian,

I fear the parameter space is much too large and I guess that’s the reason why you didn’t find too many approaches …

Keep on searching but better though: Do it your way!

Regards

Herbie

1 Like
#3

My humble suggestion is to save the code and comment it thoroughly!

2 Likes
#4

Hi @Christian_Tischer

thanks for bringing this up! Quality assurance in BioImage Analysis should indeed be a more discussed topic. We might even think of showing people how to do this in our courses. One could learn from the field of software engineering where the people are many steps ahead. I can recommend going through the teaching materials of the software carpentry. I especially like the first slide:

image
image1024Ă—746 84 KB

I sometimes even follow the approach ofTest Driven software Development (TDD); ClearCLIJ for example was partly made like that.

However, while we could learn from best-practices from experts, we should also keep in mind that BioImage Analysts are usually no computer scientists and software engineers. Thus, I would think boiling the best-practices down to basics makes a lot of sense and I think this is where you want to go as well. Thus, for the beginning how about focusing on unit tests for plugins, regression tests and integration tests for workflows first. Did you for example know, that there are regression tests for ImageJ?

Before I start listing more thoughts, I’m wondering shall we setup a shared document for collecting ideas?

Cheers,
Robert

1 Like
#5

Robert,

the problem starts with trying to link “software engineering” and “(bio)image analysis”.

I’m pretty sure that you will fail …

A highly renowned scientist in pattern recognition once stated that pattern recognition is an art. I second this view and extend it to many aspects of image analysis.

Regards

Herbie

#6

I’d also include the idea of, in cases where you should have a positive control, thinking ahead of time what measurement should be separate between negative and positive controls and making sure you do see that.

I’d also up your number of images in your experiment before you stop looking at segmentation- it’s pretty easy to quickly (<10 minutes) view thousands in Finder (on Mac) or Image Preview (on Windows). You don’t need to take notes so much as get the “gestalt”. I can’t tell you how many times this has personally saved my butt!

2 Likes
#7

Yes, we can! But I am currently not thinking about BioImageAnalysts but rather end-users with no programming and also no experience in image analysis.

2 Likes
#8

Great! This is in fact also what we routinely do and recommend. This is totally something to add!
This is also why image analysis workflows of assays without such biological controls are much harder to debug!

2 Likes