Comparing photograph with generated image

What I’m after

I’m working on an embedded application running on a device with an LCD screen.
I want to run some integrations tests on this application where the tester verifies that the content on the screen (text and images) matches the expected output for various use cases.

This is running in a resource constrained environment so I don’t have the ability to take screen shots etc.

I thought I’d take the following route to run the test cases where the tester:

  1. Requests the UUT to display a particular screen
  2. Generates a reference image of what the screen should contain
  3. Takes a photograph of the display using a webcam
  4. Compares the photograph with the reference image

Prototype

I threw a prototype together to see if this idea will work. The following are three images I generated to try it out:

  • EXPECTED: the expected output - generated by writing text onto a photograph of a blank screen.
  • PASS: an image that should pass the test
  • FAIL: an image that should fail the test

EXPECTED

ABC_ref

PASS

ABC

FAIL

ABO

I used the following script to do the image comparison:

import cv2
from skimage import metrics

accept = cv2.imread("Pass.png")
reject = cv2.imread("Fail.png")
expected = cv2.imread("Expected.png")

print(metrics.structural_similarity(accept, expected, multichannel=True))
print(metrics.structural_similarity(reject, expected, multichannel=True))

The output was as follows:

0.9551051504464912
0.9545966850582358

I think the similarity factors are too close for me to clearly determine whether the actual image matches the expected one or not.

Could suggest an algorithm that I can use to achieve what I’m trying to do?

Thank you

1 Like

Hi @dushara,
I know the tutorial advises for the structural similarity rather than mean error, but still did you try it to see if the latter might perform better in this special case ? See
Structural similarity index — skimage v0.19.0.dev0 docs (scikit-image.org)

Otherwise if you know that you are expecting text you could try to run an Optical Character Recognition (OCR) on the image and compare the resulting text with the expected text.

I would try to convert the ouput image to binary, and then compare it to a binary reference image (perhaps with mean squared error, as suggested by Laurent). You can then work to make your reference image reflect the expected output closely (maybe by applying dilation, or by using the same LCD font).

Thank you for the response Laurant.

I tried the MSE approach that you suggested and the output is as follows:

accept: 77.77642018484907
reject: 82.7923768205764

Would you consider the numerical distance large enough for the comparison?

I can’t do OCR because there will be screens that also display images.

Thanks for the response Stéfan,

When you said binary image did you mean convert the png to raw binary data or something else? This is an area I don’t know much about.

I have used the same LCD font (applied using ImageMagick). I’ll read up a bit about dilation. Thanks!

I meant turning the image into False/True or 0/1 values, corresponding to black and white pixels. Since the LCD screen only renders “off” or “on”, I think this is a better match. Here’s some code to evaluate a few built-in metrics in scikit-image:

from skimage import io, color, metrics
import matplotlib.pyplot as plt
import numpy as np

T = 0.7

expected = color.rgb2gray(io.imread('desired.png')[..., :3]) < T
seen1 = color.rgb2gray(io.imread('input1.png')[..., :3]) < T
seen2 = color.rgb2gray(io.imread('input2.png')[..., :3]) < T

for (name, metric) in (
        ('normalized rmse', metrics.normalized_root_mse),
        ('hausdorff', metrics.hausdorff_distance),
        ('variation of information', metrics.variation_of_information),
        ('structural similarity', metrics.structural_similarity),
        ('normalized mutual information', metrics.normalized_mutual_information)
):
    print(f'Metric: {name}')
    print('Pass image:', metric(seen1, expected))
    print('Fail image:', metric(seen2, expected))
    print('---')

This, for me, yields:

Metric: normalized rmse
Pass image: 0.6544267936709409
Fail image: 0.6944663163659769
---
Metric: hausdorff
Pass image: 2.0
Fail image: 5.656854249492381
---
Metric: variation of information
Pass image: [0.00936584 0.02936322]
Fail image: [0.01097041 0.03379186]
---
Metric: structural similarity
Pass image: 0.9902725461569556
Fail image: 0.9892245163438118
---
Metric: normalized mutual information
Pass image: 1.4485927089240354
Fail image: 1.4004950868031385
---