How to test image processing codes?

Coming from numerical algorithms, I find it difficult to test my image processing codes. The reason is that

  1. I apply many heuristics that work on the images I consider but nothing proves that they will work on images I will use later
  2. although I separate the workflow to small logical chunks, some of them are easy to be tested as they are purely numerical by nature, the core of the algorithms are heuristic (see the previous point)
  3. I cannot find an automated way of assessing an algorithm (e.g. segmentation) without checking the result by the naked eyes.
  4. tests would be indicative for concrete, real-world images, that would run for a long time. This does not fit the philosophy of tests that run quickly.

Lacking tests, I

  1. feel guilty of not being able to prove that a method works, which is crucial in research
  2. have constant anxiety that the next refactoring or feature addition will mess up the existing code

It it matters, I use Python with the scientific stack (numpy, scipy, etc.) and scikit-image. I rely on these well-established libraries to make sure I do not introduce unnecessary bugs.

Those of you who are proficient in developing complex image processing codes, what advice can you give me so that I can ensure about the validity of my approach?

2 Likes

Save input/output images as resources in your unit test project. Then have unit test verify that when a specific input is given, that specific output is produced.

Thank you, this answers how to use unit tests. However, if I use another parameter or another algorithm, the output is still acceptable when judged by the eyes, even though the produced image is not the same. The question is: how can I decide in an automated way whether the result of an algorithm (output) is acceptable or not? I would need a methodology that – once formulated – requires no human intervention/judgement.

Hi @ZCs,

I appreciate your thinking about this, and will try give some ideas. Again, way to go for thinking about this. Yes, it’s a hard problem, and there are not “easy” answers.

These few thoughts / ideas are not concrete answers, but If you’re more specific about methods / algorithms, I can try to be more specific. Since you mentioned segmentation in particular, I’ll talk about that.

Philosophical thoughts

Many segmentation methods are optimization methods, so its possible to confirm that the optimization “part” is working correctly: i.e. is a local minimum of the cost found. I would not call methods like this “heuristic” methods because there is a specific costs function that they optimize, and they can and do successfully optimize the given cost.

For example, usually a heuristic is used to choose the number of clusters when using Kmeans. But given K, Kmeans itself I would not call “heuristic”.

Sometimes those methods perform badly though. Not badly in that they failed to optimize the cost, but badly in that the cost we gave them doesn’t reflect the thing we really want: “segment” the object != minimize the functional I made up. Heuristic methods can perform better because sometimes the functionals we know how to write down using math and optimize are not good at the task we care about.

Tests

I cannot find an automated way of assessing an algorithm (e.g. segmentation) without checking the result by the naked eyes.

Related to @masamihaga 's suggestion

Generate synthetic images for which you

  1. know the correct answer
  2. know adhere to the assumptions of your algorithm

That lets you separate algorithmic errors from random noise. On point (2) for example, if you’re using /debugging an MRF-based algorithm, you can generate images that are markov.

Guilt

feel guilty of not being able to prove that a method works, which is crucial in research

Maybe it will help your guilt :slight_smile: to separate “correctness” from “working”, where:

  • “correctness” - the algorithm does what it claims to do
  • “working” - the algorithms produces (close to) correct results on real data

A “correct” algorithm may not always “work.”

2 Likes

@bogovicj Thank you for the insightful answer.

Generate synthetic images for which you
1. know the correct answer
2. know adhere to the assumptions of your algorithm

Yes, that’s exactly I did when I created a small artificial image as a 2D numpy array, constructed in such a way that it challenged my algorithm! However, that was an ad-hoc test I threw away after I had started working on a real-world image. I should have included it as a unit test! From now on, I will do so.


I mentioned segmentation because I didn’t want to go deeply to my task, rather I wanted to get some general answers. However, here is shortly what I called a heuristic method. I have a path – given as an 8-connected set of pixels – in a 2D segmented (labelled) image. This path is the result of the skeletonization of the segmented image. My goal is to determine which two labelled regions neighbours the path. As a simplification, I consider only the two end pixels of the path for determining the two neighbouring regions I seek. There are surely better strategies, but this is fast and I just use this example to shed light on the heuristics. Let us regard the attached image, which shows the neighbourhood of one of the end pixels of the path, denoted by P. The four different colours correspond to the labelled regions, surrounding the end pixel.

My first strategy, which worked on a generated test image, was to look around in the n-neighbourhood (again, 8-connectivity) of the end pixel (in the image above, I took n=1 but n is a parameter of the algorithm). The two region pixels that have the largest occurrence give the connecting regions. In this case, it is the green and the blue. However, the whole image, from which I showed the proximity of P, reveals that the green region is a tiny narrow part, and the actual regions that should neighbour the path are yellow and blue. Hence, I came up with the following heuristics: look in the direction normal to the path, but exclude the direct neighbours in that direction. This strategy makes sense as it circumvents the problem with small narrow region parts.

1 Like