ImgLib2 Neighborhoods behaving unexpectedly on border of image

Hi all,

I’m iterating over neighborhoods of pixels using the net.imglib2.algorithm.neighborhood shapes and struggle to understand their behavior at the border of an image. Here is an example:

Img<IntType> img = new ArrayImgFactory<>(new IntType()).create(2, 2);
img.forEach(pixel -> pixel.set(1));

DiamondShape shape = new DiamondShape(1);

IterableInterval<Neighborhood<IntType>> neighborhoods = shape.neighborhoods(img);

Cursor<Neighborhood<IntType>> cursor = neighborhoods.cursor();

while(cursor.hasNext()) {

	Neighborhood<IntType> val = cursor.next();

	float[] pos = new float[cursor.numDimensions()];
	cursor.localize(pos);

	System.out.print("pos: " + Arrays.toString(pos) + " neighborhood: ");
	val.forEach(neighbor -> {
		try {
			System.out.print(neighbor.get() + " ");
		} catch(ArrayIndexOutOfBoundsException e) {
			System.out.print("X ");}
	});
	System.out.println();
}

I create a 2x2 image, set it all to 1, and print the neighborhood value for each pixel for a diamond shape of radius 1. If an ArrayIndexOutOfBoundsException occurs, I print an X. I expect this to return the 4-connected neighborhood for each pixel, including the center. Here is the result:

pos: [0.0, 0.0] neighborhood: X X 1 1 1 
pos: [1.0, 0.0] neighborhood: X 1 1 1 1 
pos: [0.0, 1.0] neighborhood: 1 1 1 1 X 
pos: [1.0, 1.0] neighborhood: 1 1 1 X X 

I would have expected two X for all positions since all pixels have two neighbors outside of the image border… What am I missing?

Ping @tpietzsch @hanslovsky @axtimwalde @StephanPreibisch @ctrueden

2 Likes

Hello @frauzufall

I am the author of the diamond shape. I will look into this.

3 Likes

Thanks for looking into it! I have the same /similar issue though with the RectangleShape. Changing the one line in the example where the shape is created to this (skipping the center pixel):

RectangleShape shape = new RectangleShape(1, true);

results in this:

pos: [0.0, 0.0] neighborhood: X X X X 1 1 1 1 
pos: [1.0, 0.0] neighborhood: X X 1 1 1 1 1 X 
pos: [0.0, 1.0] neighborhood: X 1 1 1 1 1 X X 
pos: [1.0, 1.0] neighborhood: 1 1 1 1 X X X X 

Which I understand even less. Now this should be an 8-connected neighborhood (right?), where I would expect 5 ArrayIndexOutOfBoundsExceptions for each pixel, but there are 4 for the top left and bottom right pixels and 3 for the others.

I don’t want to preempt what a true ImgLibier might say, but I think that the out-of-bounds behavior is simply not defined for sources that are not extended.

So I think that getting an AOOB or not getting one is meaningless here.

2 Likes

I was not aware of the neighborhoods signature that takes a RandomAccessibleInterval. As @tinevez says, it is the caller’s responsibility that all neighborhood voxels are defined because the neighborhoods do not check for boundaries themselves. Maybe a better interface would be to pass a RandomAccessible and an Interval. I personally use Shape.neighborhoodsRandomAccessible in conjunction with Views.extend (before) and Views.interval (after).

Regarding out of bound errors: If you think about how the ArrayImg maps into a linearly indexed Java array, it may become obvious why you see fewer X pixels than expected: The pixel to the left of the top left corner maps to -1 in linear index space and that is clearly OOB. The pixel to the right of the last column, second row, however maps to the same pixel as first column, third row, i.e. it “wraps” around. This behaviour is specific to each implementation of RandomAccessibleInterval and may differ, e.g. for CellImg.

Summary: Instead of using Shape.neighborhoods, use this:

val rai: RandomAccessibleInterval<T> = ...
val extended = Views.extendValue(rai, t)
val shape: Shape = ...
val neighborhoods = shape.neighborhoodsRandomAccessible(extended)
val neighborhoodsInterval = Views.interval(neighborhoods, rai)
5 Likes

Thank you for the detailed explanation! So if I want to loop over each neighborhood, but only pixels in bound, I have to come up with a value which does not exist in the image itself, extend the image with it and filter the neighborhood for this value?

2 Likes

That would not be super super efficient.

What do you want to do as filter? People tend to expand the source image with mirror conditions to deal with border effect.

Or you can test at any position in the neighborhood if you are in a position outside the source before accessing its value (with Intervals.contains) but I am not sure it is efficient.

5 Likes

This became a theoretical question since my problem was indeed solvable with a mirror extension. Maybe finding a suitable extension is always the most efficient solution. I’ll mark this as solved. Thank you @tinevez and @hanslovsky!

4 Likes

Yes, in my experience, there is no good “one fits all” solution and RAIs need to be extended by the caller for their specific use cases.

2 Likes

You could think of extending by something like NaN that are then ignored by the filter, depending on the algorithm. But to my knowledge, there’s no such solution in ImgLib2, is there? I.e. you always have to extend with some value.

Several ImageJ1 filters act as if the image was extended with NaNs, i.e. effectively ignoring OOB pixels.

1 Like

I was going to point that out as well… for things like median, it is actually quite nice to really get the median of the actual neighborhood, even when that neighborhood is smaller around the edges.

@tpietzsch @axtimwalde @StephanPreibisch Do you have any ideas for elegant ways to support this sort of edge behavior of neighborhoods? The only way that occurs to me immediately is what @frauzufall and @imagejan suggested: extend by NaN and then filter out the NaNs—slower (as @tinevez points out), but it works.

3 Likes