Implementation plan for Imglib2-rois 2D

#1

Hello,

I am working on adding support for 2D ROIs to ImageJ-OMERO. In order to do this, I will need to expand the ImgLib2-Roi API. My basic plan for doing this is outlined below. @ctrueden, @tpietzsch, @dietzc, @axtimwalde, @joshmoore please let me know your thoughts on this plan, and if you have any questions/concerns regarding it.

Overview

  1. Create the below classes:
    a. BinaryMask
    b. Composite (OMERO allows for ROIs which are a union of shapes, this would be the equiavalent to that)
    c. Rectangle2D
    d. Ellipse2D
    e. Polyline2D
    f. Line2D
    g. Point2D
  2. Rename Polygon to Polygon2D
  3. Add methods to Regions to create:
    a. BinaryMask
    b. Composite
    c. LabelRegions
  4. Create GeomRegions in net.imglib2.roi.geometric and add methods to create the below ROIs to it:
    a. Polygon2D
    b. Rectangle2D
    c. Ellipse2D
    d. Polyline2D
    e. Line2D
    f. Point2D

Details

  • All of the proposed new classes would implement the Contains interface, and the RealRandomAccessibleRealInterval interface
  • Create all classes expect BinaryMask & Composite in `net.imglib2.roi.geometric’
  • Create BinaryMask & Composite in net.imglib2.roi
  • BinaryMask would essentially be an Img<BoolType>
  • Composite
    • Contain a list of ROIs in the composite
    • Allow for operations (union, difference, etc.) between ROI structures via JTS Topology Suite
  • Point2D would be a RealPoint
  • Line2D would be defined by two RealLocalizables
  • Polyline2D would be defined by a List<RealLocalizable>
    • Could also specify which points in the list are the start/end points. If not specified assume first point in list is start and last point in list is end
  • Rectangle2D defined by width, height, center point, and AffineTransform2D
    • AffineTransform2D would allow for rotations if someone didn’t want a rectangle which was parallel to the xy-axes
  • Ellipse2D defined by xy radii, center, AffineTransform2D

Questions

  1. Do we want to include a method for creating LabelRegions in Regions? I can see keeping it separate or adding it …
  2. Would we prefer to define a rectangle by the four corner points? Or some other way? Or both?
  3. What should be the equivalent data structure in ImgLib2 for OMERO Label ROIs?
    a. We could try to equate OMERO Labels to ImgLib2 LabelRegions
    b. Or we could create a new type of label (either in ImgLib2 or ImageJ-OMERO), which is more of a direct equivalent to OMERO Labels

This is just a very high level plan. There’s still more details to work out, but I wanted to make sure we were ok with this basic structure before I proceeded. Please feel free to make suggestions or bring up any issues which I haven’t addressed.

Thanks,

Alison

4 Likes
Proposed fix for `ShapeRoi` issue
#2

Hi Allison,

please find some comments/questions below.

Why do you explicitly need this class? How is this different from RAI<? extends BooleanType>.?

Why only 2d? We should think careful here and think about synergies with existing Neighborhood implementations?! Why should we have an extra class for Point2D and Line2D?

Why?

I have the feeling that these are methods which are highly specific to your implementation. Therefore i suggest to put them rather in another util class. Composite sounds really interesting, but this could be covered by Views. because basically these are operations on RAI<BoolType>?

In general I’m happy that you start working on Imglib2-roi again! We just should be really, really careful what we do in the imglib2 world. Why can’t we have a package Imagej-roi based on Imglib2-roi and carefully merge stuff to imglib2-roi before we release imagej2-roi? Like this we can figure out what to put where.

Thanks again for your efforts!

Christian

#3

Hello Christian,

Thanks for your comments/questions!

You are correct that this is essentially a RAI<? extends BooleanType>. The only difference being that this class would implement the Contains interface, which I think would work well with the existing Regions framework.

I’m focusing only on 2D implementations in order to be consistent with OMERO. And it was my understanding that we wanted a distinction between 2D and 3D ROIs based on @ctrueden’s notes from his discussion with @tpietzsch at the last hackathon. However, if this is inaccurate please let me know.

Implementing Point2D and Line2D, is for similar reasons to having a BinaryMask class. These classes would implement the Contains interface.

These wouldn’t be operations on RAI<BoolType>, but are instead operations on RealRandomAccessibleRealInterval<BoolType>. The goal of the Composite class would be to implement operations like those in the diagram below:

Composite-operations.png666Ă—707 69.6 KB

As such, I am not sure if Views would be able to do this. However, please let me know if I am mistaken.

Thanks again for your feedback, and please let me know if you have any additional comments/questions!

Alison

#4

Thanks for your comments, @dietzc. :smile:

A couple other points in addition to @awalter17’s remarks:

This rename was specifically requested by @tpietzsch. Looking at the code, it clearly only handles the 2D case. And as we have discussed a couple of times in the past, generalizing 2D polygon to N-D polytope is not a trivial thing to do.

If imglib2-roi were stable (i.e., at 1.x or later), then I would agree with you. But the imglib2-roi project is still in incubation (i.e., it is a 0.x version). The purpose of such projects is rapid development. Why do we need to put experimental code into a second experimental project rather than simply develop it within imglib2-roi?

#5

You don’t need it. Have a look at https://github.com/imglib/imglib2-roi/blob/master/src/main/java/net/imglib2/roi/Regions.java. This already is what you want.

Actually, for both of these classes the contains is not really defined, as e.g. a RealPoint can’t really contain another point. Also a line shouldn’t implement contains. @tpietzsch can maybe comment on this. However, if we add those classes without contains I’d rater call them RealPoint (which we already have) and Line. Both are defined in n-d. Then we could add a Point and a RasterizedLine. This would fit to what we already have. Same is applicable for the other classes, too.

At the moment neither Views nor RealViews can do it, but I think you could implement it in the View style, for both Real and Rasterized. Maybe in the Regions. class (@tpietzsch)? Finally, we could wrap these operations as ops then. I think @haesleinhuepf has tackled similar problems?

I agree, we just should be careful, because imglib2-roi is, even not out of beta, already consumed by quite a few projects and people. So we shouldn’t (even not being out of beta) change everything all the time. But again, I agree, let’s do it in ops and carefully review PRs.

#6

I agree with what @dietzc said, except for the Polygon2D. The current implementation is really 2D. It defines the inside of a polygon. If it would define only the lines that make up the polygon, then the same criticism as for Point and Line would apply.

Regarding BinaryMask, I agree that it is superfluous. The idea of Contains is to have an easy way of making something act as a RealRandomAccessible< ? extends BooleanType >. So making BinaryMask implement it would be beside the point.

Point and Line are not regions because they do not contain anything. These would be more properly defined as annotations I think. I think in general the ROI concept in ImageJ confuses (at least) two concepts that we should separate (annotations and regions).

#7

Regarding Composite and Views. There is Views.pair which can give you a RAI<Pair<BoolType,BoolType>> from RAI. Then you can use a converter to implement the (pixelwise) BoolType x BoolType -> BoolType operation.

In general this should go through Regions such that more efficient special cases can be implemented etc.

#8

This has been a common “meme” over the past few years. “A line cannot contain anything!” Well, that’s true for 2D images. A 1D line “manifold” embedded in a 2D space is infinitely thin, yes. But a 1D subline of a 1D image most certainly contains points.

By the same reasoning, a 2D polygon cannot contain anything in a 3D image volume, because the polygon is infinitely thin. Does that mean the 2D polygon should not implement Contains?

It seems to me that what’s most important is for the dimensionalities to match here. No?

(Actually: these subdimensional manifolds can contain samples—it is just highly unlikely in practice. But there certainly can be an intersection…)

I would like to note that 0.x does not mean “beta”. The term “beta” usually means “feature complete” whereas “0.x” means “still developing rapidly, and unstable.” If so many things depend on imglib2-roi, then according to SemVer, we should put out 1.0.0 and start maintaining backwards compatibility properly. We are probably years overdue on doing that for several components, actually…

3 Likes
#9

Yes, true. I meant Line2D. This I would call an annotation. It should not implement RRAI (or Contains). But it can be used to define a 1D hyperslice of a 2D image and bounds such that we get the 1D subline. My point was that Line2D is not a “region” in the sense that I would like to use for imglib2-roi.

I would be ok with splitting the deprecated / unstable stuff into imglib-roi-experimental or similar and releasing imglib2-roi 1.0.0 (containing the new labeling stuff and very few other things). This would be actually nice because it would discourage people from using old stuff for which there is no replacement yet.

1 Like
#10

Hmm, actually I’m not so sure anymore… I see your argument now. If the 1D subline is a region, then why not the Line2D. Hmmm… @axtimwalde, do you have an opinion?

#11

Just pointing to an earlier implemented potential alternative:

It allows nicely readable code like RealRandomAccessibleRealInterval<BoolType> result = RealBinaryUnion(a, b);. Resulting booleans are calculated on the fly.

#12

Feel free to reuse code from here; these are elipsoids and rectangles implementing RRARI and Contains, but n-dimensional, instead of just 2D:


1 Like
#13

Thanks for all the feedback, everyone. For now, @awalter17 is going to work in the imagej-omero repository, copying in needed classes. Thanks very much @haesleinhuepf for your hyper-ROI classes; we will indeed use many of these as starting points. Once the dust settles, we can decide whether to put all the code.

#14

Great, thanks @ctrueden and @awalter17. @awalter17 if you want intermediate feedback on your code, let us know. It’s really great that you work on this not too easy project.

I vote for this :wink:

#15

I have to confess that it is rather frustrating trying to work on the ImgLib2 ROI data model. It feels like we are consistently told what not to do, without sufficient guidance on how things should be structured. Everything is deprecated with no replacement, and there is no writeup anywhere (to my knowledge) detailing any future plans for correcting this.

As things stand, it really will be much easier to just “do our own thing” and leave imglib2-roi completely alone. I fear we will create even more diverging ROI structures in the medium term… I really hope the pain of unifying things later will not be too much.

#16

That’s why I vote for deleting the deprecated stuff and and release the imglib2-roi components which are stable (Regions, ImgLabelings etc). Then we can iteratively add stuff. I fear that if we do everything with the current state of the imglib2-roi package it will become a mess.

About telling you what not to do: I think the main problem is that nobody really knows what to do. So it’s important that someone comes up with ideas and suggestions which we can discuss.

2 Likes
#17

About telling you what not to do: I think the main problem is that
nobody really knows what to do. So it’s important that someone comes up
with ideas and suggestions which we can discuss.

I’m already working quite a lot with the imglib2-roi stuff and agree with @dietzc : (Label)Regions and (Img)Labelings may be released at some point as it is implemented now. If this part of the library becomes re-invented, it could be a waste of time. Not just for the people doing it, but also for guys like me who would have to change their code to keep compatibility with whatever is coming up.

#18

OK. Why not move the stable stuff into imglib2 core, instead of inventing another new component? The reason we invented imglib2-roi in the first place was because those classes were not stable enough to include in core.

#19

Hello everyone!

There’s a few questions I have and some things I’ve been waffling on that I’m hoping you all can help with.

Thanks to @haesleinhuepf’s work on ROIs, the only remaining things that need to be implemented (with regards to ImageJ-OMERO) are line, polyline, point, and floating text. Once those are implemented there will be a roughly equivalent ImgLib2 ROI for each OMERO ROI.

It was brought up previously that there is a difference between a region and an annotation. As such, would it be beneficial to include an Annotations class? This class would function similarly to Regions and GeomRegions, and would hopefully emphasize the separation between these two concepts. Something like:

Regions

  • .sample(…)
  • .iterable(…)
  • .intersect(…)*
  • .not(…)*
  • .subtract(…)*
  • .union(…)*

GeomRegions

  • .polygon2D(…)
  • .hyperrectangle(…)
  • .hyperellipsoid(…)

Annotations

  • .point(…)
  • .text(…)

*I was thinking it made sense to add calls to @haesleinhuepf’s RealBinary operators in Regions, but if it doesn’t make sense to add them to Regions please let me know.

And if I create an annotations class, what should be included as an annotation? I 100% agree that points and floating text are annotations, but I go back and forth on lines and polylines.

Are lines/polylines regions or annotations?

In previous ROI implementations it appears as though lines and polylines were regions. And in order to be considered within the region a point just needed to be within a certain distance from the line. So if we wanted to be consistent with previous versions, it would make sense to have lines and polylines be regions.

Additionally, it appears as though PointCollection and RasterizedPolygonalChain are regions. To my understanding, RasterizedPolygonalChain is a region which contains the discrete points along a closed polygonal chain and PointCollection represents a region containing a set of discrete points. A closed polygonal chain is just several connected line segments, so in order for a polygonal chain to contain anything the individual line segments need to be able to contain points. And if that’s the case then lines and polylines can contain points and should be classified as regions. It just seems like if closed polygonal chains can be regions, then lines and polylines are closely enough related that they too should be considered regions. However, if RasterizedPolygonalChain and PointCollection are not intended to be regions or are some special case please let me know.

The counter to all of this being, that lines are infinitely thin and thus highly unlikely to contain points of interest in anything except 1D. And I go back and forth on lines/polylines as regions/annotations. What does everyone else think?

Are PointCollections and RasterizedPolygonalChains Regions?

As mentioned above, it seems to me that they are intended to be regions but this seems contradictory to the idea that lines and points are not regions. If someone could please explain the intention of these classes to me, I would really appreciate it. Also should GeomRegions contain methods for creating these (i.e. GeomRegions.pointCollection(...))?

Naming conventions?

It seems as though there is a naming convention in which each shape has a rasterized counterpart. So am I correct in assuming that I should add a RasterizedHyperRectangle and a RasterizedHyperEllipsoid class? And if this is the case why isn’t there a PolygonalChain class?

Sorry if some of those questions seem redundant, a few of them kind of piggy-back off one another. @ctrueden @dietzc @tpietzsch @haesleinhuepf please let me know your thoughts on the above. Additionally please feel free to let me know if you have any questions, concerns, or suggestions regarding this project.

Thank you very much,

Alison

1 Like
#20

related: https://github.com/imglib/imglib2-roi/issues/28