Histogram1d not returning bin frequencies in order for some images

Hello,

So I’m using the net.imglib2.histogram.Histogram1d of an image to iterate over its bins and frequencies, but it seems that whenever there are a a lot of bins with zeros, their order gets mixed when iterating over them. What could be causing this?
This is the code I’m using to do so, I have tried different ways to iterate over the bins but I still get the same problem.
Histogram1d histogram = ops.image().histogram(img);
Iterator iter = histogram.iterator();
while(iter.hasNext()){
int pixelCount = Integer.parseInt(iter.next().toString());
System.out.println(pixelCount);
}

Here is a sample image that creates such problems:
lowContrast
I want to do this to estimate the contrast of an image.

Thank you

1 Like

The problem is the way that the image was encoded. You seem to have a quantised image where not all grey levels are represented (so it has less than 8 bits).

You could add a tiny amount of noise (e.g. Add Specified Noise command) with a very low standard deviation (e.g. 1 or 2). The new image would still have a very similar shaped histogram but without the gaps.

2 Likes

Thank you for your suggestion Gabriel. I did this but still had the same problem, I think it’s because the Add Specified Noise command adds noise within the same range of values that the image already has. Adding Salt and Pepper noise did help and the values were returned in order, even though it most likely skews results a bit. Thanks again.

Hmm strange. with a standard deviation of 1 I got a nice continuous histogram.
Salt and peper does not makse sens as it will add noise at the extremes of the histogram (0 and 255). You want a tiny amount of noise so some grey levels fall in the gaps:
Histogram of ttt-1

This is what I get…
Screenshot (1)
These are the first 8 values I get from the GUI:
4067, 3887, 3010, 2798, 2984, 3009, 3790

And this is a sample of what I get from my code:
4067, 0, 0, 0, 0, 3887, 0, 0 , 0, 0, 3010, 0 ,0 ,0, 0, 2798

I just downloaded the image you posted and aplied “Add Specified Noise” with a value of 1.
run("Add Specified Noise...", "standard=1");

You are probably doing something else, because your list has values larger than 255 (or the image you posted is not the one you are working with).
Sorry I do not think I can help further.

The values refer to the number of pixels in the bins.
Thanks for your help Gabriel :slight_smile:

Hello Rody -

I can’t comment on what might be going on with the imglib2
histogram function, but if your workflow permits you to use
the IJ1 implementation, that seems to work fine for me.

First, note, that many valid pixel values do not appear in your
image, so the correct histogram does have a lot of “holes,”
especially for lower (darker) pixels values. Also, your image
does not have any even pixel values, so for a 256-bin histogram
(at least) every other bin is empty.

Having said that, here is a jython (python) script that shows the
result produced by the IJ1 histogram function:

from ij  import IJ
from ij.process import ImageStatistics

print  IJ.getVersion()

imp = IJ.getImage()
print  imp.getTitle()
hist = ImageStatistics.getStatistics (imp.getProcessor()).getHistogram()
print  len (hist)
print  hist[:10]
print  hist[-10:]

When I run this script (with your sample image open and selected)
I get:

2.0.0-rc-69/1.53b
rodys_amazing_sample_image.png
256
array('l', [0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L])
array('l', [0L, 183L, 0L, 88L, 0L, 35L, 0L, 3L, 0L, 0L])

For simplicity, I haven’t printed out the entire histogram, but the
entries do appear in the correct order without getting mixed up
at all.

My results do not line up with the image of the gui-histogram you
posted. It looks like you might have an inverted LUT or something
(the low-value pixels are shown as light-colored in the calibration
bar), your range seems to be compressed by about a factor of two
compared to what I see, and in your histogram plot I don’t see the
alternating zero counts for even bins / non-zero counts for odd bins
that I get.

But my results do correspond to the actual pixel values in the .png
image you posted.

(Note, there was nothing special about jython here – it should be
pretty obvious how to do this in some other scripting language or
in java.)

Thanks, mm

1 Like

Thank you mm.
I would rather use ImageJ2 libraries for the project I’m working on. I just realized that the histogram does look different when you save the image from the forum, it must have been altered when I posted it.
Maybe it won’t get altered if it’s in a zipped folder.
troublesomeImage.zip (27.3 KB)

Many thanks

Hello Rody -

Suit yourself.

What happens if you download from the forum a (potentially altered)
copy of your sample image and apply your workflow to it? Do you still
see the issue you first reported, or do things seem to work correctly?

Thanks, mm

Hello Rody -

I can reproduce your IJ2 ops.image().histogram() issue.

First, the sample image you posted originally doesn’t have a
(non-trivial) LUT, while lowContrast.png that you posted in your
zipped folder does:

Look-Up Table for lowContrast

I didn’t look closely, but I believe that the two images are otherwise
the same. (Anyway, this explains the differing IJ1 histograms.)

As to the IJ2 issue, I can reproduce it with both images. It does
appear that the IJ2 histogram returns its bins in the wrong order,
although, superficially, there looks to be some consistent structure.
To my eye, it appears as if the IJ2 is being iterated through with
some non-zero stride, or something.

Anyway, if you sort both the IJ1 and IJ2 histograms, you get the
same counts, so the IJ2 issue has something to do with an incorrect
ordering of the histogram bins (or at least the iteration order).

Here is a script that compares the IJ1 and IJ2 histograms:

#@ Dataset img
#@ OpService ops

from ij  import IJ
from ij.process import ImageStatistics

print  IJ.getVersion()

imp = IJ.getImage()
histIJ1 = ImageStatistics.getStatistics (imp.getProcessor()).getHistogram()

print  'title =', imp.getTitle()

histogram = ops.image().histogram(img)
histIJ2 = []
for  v in histogram:
    histIJ2.append (v.get())

lbls = ['IJ1:', 'IJ2:']
hists = [histIJ1, histIJ2]

for  lbl, hist in zip (lbls, hists):
    print  lbl
    print  'n =', len (hist), ', nzero =', hist.count (0), ', cnt =', sum (hist), ', cmin =', min (hist), ', cmax =',  max (hist)
    print  'nzeven =', hist[0::2].count (0), ', nzodd =', hist[1::2].count (0), ', cnteven =', sum (hist[0::2]), ', cntodd =', sum (hist[1::2])
    print  '[ :10] =', hist[ :10]
    print  '[-10:] =', hist[-10:]

# [Edit: (histIJ2 == histIJ1) is testing a list against an array so will always fail.]
print  '(histIJ2 == histIJ1) =', (histIJ2 == histIJ1), '(sorted (histIJ2) == sorted (histIJ1)) =', (sorted (histIJ2) == sorted (histIJ1))

Here is the output of the script, first run on your original image:

2.0.0-rc-69/1.53b
title = rodys_amazing_sample_image.png
IJ1:
n = 256 , nzero = 205 , cnt = 65536 , cmin = 0 , cmax = 6785
nzeven = 128 , nzodd = 77 , cnteven = 0 , cntodd = 65536
[ :10] = array('l', [0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L])
[-10:] = array('l', [0L, 183L, 0L, 88L, 0L, 35L, 0L, 3L, 0L, 0L])
IJ2:
n = 256 , nzero = 205 , cnt = 65536 , cmin = 0 , cmax = 6785
nzeven = 101 , nzodd = 104 , cnteven = 40782 , cntodd = 24754
[ :10] = [1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L]
[-10:] = [88L, 0L, 0L, 0L, 0L, 35L, 0L, 0L, 0L, 3L]
(histIJ2 == histIJ1) = False (sorted (histIJ2) == sorted (histIJ1)) = True

And then run on your zipped-folder image:

2.0.0-rc-69/1.53b
title = lowContrast.png
IJ1:
n = 256 , nzero = 205 , cnt = 65536 , cmin = 0 , cmax = 6785
nzeven = 102 , nzodd = 103 , cnteven = 34265 , cntodd = 31271
[ :10] = array('l', [5830L, 2507L, 2687L, 2398L, 3477L, 2696L, 2444L, 6785L, 5262L, 3610L])
[-10:] = array('l', [0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L])
IJ2:
n = 256 , nzero = 205 , cnt = 65536 , cmin = 0 , cmax = 6785
nzeven = 102 , nzodd = 103 , cnteven = 35581 , cntodd = 29955
[ :10] = [5830L, 0L, 0L, 0L, 0L, 2507L, 0L, 0L, 0L, 0L]
[-10:] = [0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L, 1L]
(histIJ2 == histIJ1) = False (sorted (histIJ2) == sorted (histIJ1)) = True

On the face of it, this would appear to be an IJ2 issue, although
perhaps we’re using the histogram op incorrectly.

Thanks, mm

[Edit: There is an error in the histogram-comparison script. It
performs an equality test for an array vs. a list, so the test will
always fail. See comment in the script.]

2 Likes

This is very strange indeed, from what I’ve noticed is that it tends to happen with images that have histograms with bins with a lot of 0 values. I may have no option but to use IJ1 histograms.
Thanks again mm,
Rody

Hello Rody -

To explore a little further, I took your no-LUT, original image, and ran
Process > Enhance Contrast...
and
Process > Noise > Add Specified Noise...
to get an image with no empty histogram bins whatsoever:

enhance_contrast_add_noise

Here is its Analyze > Histogram histogram:

Histogram of enhance_contrast_add_noise

The IJ2 histogram is still wrong, but it looks like my “stride” theory
is all wet.

[Edit: In this case, the IJ2 histogram is actually correct – the
histogram-comparison script always reports a mismatch.]

Here is the result of my IJ1-IJ2 comparison script:

2.0.0-rc-69/1.53b
title = enhance_contrast_add_noise.png
IJ1:
n = 256 , nzero = 0 , cnt = 65536 , cmin = 11 , cmax = 888
nzeven = 0 , nzodd = 0 , cnteven = 32706 , cntodd = 32830
[ :10] = array('l', [17L, 11L, 21L, 25L, 42L, 31L, 46L, 47L, 46L, 46L])
[-10:] = array('l', [194L, 183L, 152L, 156L, 138L, 115L, 91L, 60L, 53L, 34L])
IJ2:
n = 256 , nzero = 0 , cnt = 65536 , cmin = 11 , cmax = 888
nzeven = 0 , nzodd = 0 , cnteven = 32706 , cntodd = 32830
[ :10] = [17L, 11L, 21L, 25L, 42L, 31L, 46L, 47L, 46L, 46L]
[-10:] = [194L, 183L, 152L, 156L, 138L, 115L, 91L, 60L, 53L, 34L]
(histIJ2 == histIJ1) = False (sorted (histIJ2) == sorted (histIJ1)) = True

[Edit: The last line should be (histIJ2 == histIJ1) = True,
that is, that the IJ1 and IJ2 histograms in fact agree. The
comparison script – see my previous post – incorrectly tests
a list against an array, so (histIJ2 == histIJ1) always
evaluates to False.]

My simplistic probes agree for the IJ1 and IJ2 histograms, but the
complete histograms still compare as unequal, so the IJ2 issue is
still there (but maybe more modest). I haven’t drilled down to find
specifically where the two histograms differ, but it doesn’t appear
to be anything as simple as a data-independent reordering of the
bins.

Thanks, mm

[Edit: The script (from my previous post) used for this result has
error. See the edits in this post and my earlier post.]

1 Like

I just found something interesting, by adding one black pixel, the IJ2 code returns the values in order.
These are the first 10 values for the image in the zip folder:
5830,2507,2687,2398,3477,2696,2444,6785,5261,3610
They seem in accordance with the first 10 values from the GUI histogram.

This is the java code I used to set the value of one pixel to black and another to white (since the image in the zip folder has an inverted LUT).

    final RandomAccess<UnsignedByteType> r = img.randomAccess();
    final Random random = new Random();
    int color = 0;
    int pos = 5;
    for ( int i = 0; i < 2; ++i ) {
        if(i == 1){
            color = 255;
            pos = 7;
        }
        r.setPosition( pos, 0 );
        r.setPosition( pos, 1 );
        final UnsignedByteType t = r.get();
        t.set( color );
    }

As a side note, what a difference good contrast makes on an image!

1 Like

I just downloaded the zipped version of the image, and it seems to be a “palette” image that happens to have a greyscal palette/LUT.
There is very little chance of being able to process that image as is.
To convert that into a greyscale image that you can process, convert first to RGB and then to 8bit.
Then you can add the noise as suggested before.

The imglib2 histogram stills returns the values out of order even after this, any idea of why this is only a problem for IJ2 but not for IJ1 though?

What do you mean “after this”? Converting to RGB and then to 8bit?
If so, then yes there is something wrong, but I have no idea what it might be.

Yes sorry, after trying your suggestion.

Hi @Rody and everyone,

It’s not that the values are “out of order”, its that the range of values that are being displayed are “too dense” and not what you expect. Edit: I’ve created an issue

i.e. the histogram you’re seeing is not for the intensity range [0,255], but rather for some smaller range [143, 253] - the min and max values of your image.

Note, this is why your experiment to add a black pixel gives you the results you expect.

I’ll post a script that shows this shortly, but here’s the code that computes the min/max of the image and only uses histogram bins between the two.

More detail to come,
John

2 Likes

There is a groovy script that’s very similar to your initial script below, along with part of its output.
Conclusion, I think the ops implementation of the histogram needs some work. I’ve created an issue about it.

Edits to your code to get what you expect

Don’t use ops, since it’s defaults do nasty things in this case.

Instead, create your own Real1dBinMapper as below, which says -
make bins with a min at zero, max at 255, and with 256 bins total.

If you replace the false argument with true, then you should get two additional bins which include
values below or above the specified range.

import net.imglib2.histogram.Histogram1d;
import net.imglib2.histogram.Real1dBinMapper;

histogram = new Histogram1d( new Real1dBinMapper( 0, 255, 256, false));
histogram.countData( img );
iter = histogram.iterator();

The issue

The main difference is I’m printing out the range of intensities that each bin of the histogram corresponds to. Notice these things:

  1. They don’t start at zero
  2. They don’t end at 255
  3. Some bins are nonsense - with the lower and upper range of the bin at the same value

(1) and (2) are for the reasons I said above, the ops code computes the min and max values of the image, and chooses the lowest and highest bins of the histogram. What about (3)? Well,
the histogram keeps track of the bin min/max/center with the same type as the image: UnsignedByteType in this case. If you have a bin width smaller than one (as is the case here), weird things happen.

A modified script that shows the issue

#@ Dataset img
#@ OpService ops

import net.imglib2.type.numeric.integer.UnsignedByteType;

histogram = ops.image().histogram(img);
iter = histogram.iterator();

binLo = new UnsignedByteType()
binCenter = new UnsignedByteType()
binHi = new UnsignedByteType()

histogramIndex = 0;
while(iter.hasNext())
{
	int pixelCount = iter.next().getInteger();

	histogram.getLowerBound( histogramIndex, binLo );
	histogram.getCenterValue( histogramIndex, binCenter );
	histogram.getUpperBound( histogramIndex, binHi );

	println( String.format("histogram bin range: [%f - %f] has count: %d", 
						binLo.getRealDouble(), binHi.getRealDouble(), pixelCount ));

	histogramIndex++;
}

Abridged output

histogram bin range: [237.000000 - 237.000000] has count: 2119
histogram bin range: [237.000000 - 238.000000] has count: 0
histogram bin range: [238.000000 - 238.000000] has count: 0
histogram bin range: [238.000000 - 238.000000] has count: 0
histogram bin range: [238.000000 - 239.000000] has count: 0
histogram bin range: [239.000000 - 239.000000] has count: 2507
histogram bin range: [239.000000 - 240.000000] has count: 0
histogram bin range: [240.000000 - 240.000000] has count: 0
histogram bin range: [240.000000 - 241.000000] has count: 0
histogram bin range: [241.000000 - 241.000000] has count: 0
histogram bin range: [241.000000 - 241.000000] has count: 2398
histogram bin range: [241.000000 - 242.000000] has count: 0
histogram bin range: [242.000000 - 242.000000] has count: 0
histogram bin range: [242.000000 - 243.000000] has count: 0
histogram bin range: [243.000000 - 243.000000] has count: 2294
histogram bin range: [243.000000 - 244.000000] has count: 0
histogram bin range: [244.000000 - 244.000000] has count: 0
histogram bin range: [244.000000 - 244.000000] has count: 0
histogram bin range: [244.000000 - 245.000000] has count: 0
histogram bin range: [245.000000 - 245.000000] has count: 1229
histogram bin range: [245.000000 - 246.000000] has count: 0
histogram bin range: [246.000000 - 246.000000] has count: 0
histogram bin range: [246.000000 - 247.000000] has count: 0
histogram bin range: [247.000000 - 247.000000] has count: 0
histogram bin range: [247.000000 - 247.000000] has count: 183
histogram bin range: [247.000000 - 248.000000] has count: 0
histogram bin range: [248.000000 - 248.000000] has count: 0
histogram bin range: [248.000000 - 249.000000] has count: 0
histogram bin range: [249.000000 - 249.000000] has count: 88
histogram bin range: [249.000000 - 250.000000] has count: 0
histogram bin range: [250.000000 - 250.000000] has count: 0
histogram bin range: [250.000000 - 250.000000] has count: 0
histogram bin range: [250.000000 - 251.000000] has count: 0
histogram bin range: [251.000000 - 251.000000] has count: 35
histogram bin range: [251.000000 - 252.000000] has count: 0
histogram bin range: [252.000000 - 252.000000] has count: 0
histogram bin range: [252.000000 - 253.000000] has count: 0
histogram bin range: [253.000000 - 253.000000] has count: 3
1 Like