Mesh data in ome-zarr

For an unrelated project, I started looking into storing meshes in zarr.

Following yesterday meeting, I thought it might be interesting for the community here. Turns out the maintainer of the meshio python library has been looking into zarr, so I opened an issue there:

Could be a way to have a standard mesh specification inside and outside the microscopy world.

I’ll update any progress here.

3 Likes

Thanks, @glyg :heart: Very looking forward to it. ~J

1 Like

This is quite interesting. We have been storing mesh data inside N5s, so I would be interested in this discussion. Which sort of attributes are you storing? It would be nice if the format was more clean than STL/OBJs have ended up.

1 Like

My ‘outside of microscopy’ use case is to store epithelium models - so lots of attributes with bool, int, float and string data types, plus metadata.

The structure I’m using is inspired by CGAL linear cell complexes so a 3D generalization of 2D half-edges.

In short it’s a fairly generic mesh structure: edges are oriented, faces are not triangular, and cells are not tetrahedra…

For now I use hdf5 to store four tables (pandas DataFrames) - for the vertices, edges, faces and cells. The connectivity is stored in the edge table, the metadata is serialized in json.

2 Likes

Cool, we’re storing chemical kinetic and biophysical attributes from simulations that are associated with vertices and edges; however, our mesh data structures from image processing and the simulation data structures haven’t converged yet because we’re merging existing tools. Great to see your Tyssue effort!

1 Like

Well it’s not outside of microscopy that much, it could (should?) also be the output of a segmentation (and @glyg this comment is not entirely random as I am currently using tyssue in a project :)). And as @kephale said, if formats for segmentation and simulation could converge…

As a follow up of yesterday call where this was discussed, a few semi-random points:

  • In anycase, I think we need to be able to store not just triangulation, but generic polyhedron which all format does not allow (WKT does though AFAIK).
  • we may want to store the meshes at multiple resolution, (which the neuroglancer spec seem to allow?), along with the multiresolution image
  • formats that also store visualization info could also be useful (like clean normals?)
1 Like

Hey good to hear you’re using tyssue @Anatole_Chessel - feel free to contact me if you have questions!

So the author of meshio gave this answer (tl;dr; don’t create a new format) but I’m not sure we can comply in this case.

we may want to store the meshes at multiple resolution, (which the neuroglancer spec seem to allow?), along with the multiresolution image

Even if the mesh format is not supporting multi-resolution, would it be enough to have one mesh per resolution along the images, and make the link solely based on the common spatial reference?

In anycase, I think we need to be able to store not just triangulation, but generic polyhedron which all format does not allow (WKT does though AFAIK).

As far as I can tell, WKT does not allow more than the purely spatial data to be stored on the mesh, I think is also important to store arbitrary attributes (normals as you pointed out, but also e.g. labels and things like intensity in the region).

next paragraph is speculative, and might not be practical

Maybe both for ROIs and mesh, we can store only the minimal geometric info with WKT and have a mechanism to link more generic data, contained in tables, back to these objects and the images? GIS software does that all the time doesn’t it?

3 Likes

Hi @glyg. Your last speculative paragraph makes sense to me. Is there anything could help move us forward here? ~Josh

Ha ha thanks, I started working not on meshes but SMLM because collaborators are doing STORM:

It is extremely early, but helps me to get a feeling of zarr, and both problems are similar, feed backs welcome!

One of the mechanisms I feel could be interesting in SMLM (I was discussing this with Roxane Fabre at CIML) would be the ability to select a detected point in the rendered image and link it back to the raw file (to assess quality of the input, see if it’s a false positive, etc). This is not possible today because the tiffs are too big, and it’s a hassle to manually open them.

So (sorry for the detour), as for mesh data, and ROIs, we want to be able to link an object to voxel(s) in the raw data. Fortunately, everybody leaves in the same 5D space, neatly discretized :slight_smile:

For that, we can have 'table' groups in the zarr (I still need to look how xarray does that þ), and we can enforce that every such table has at least [x, y, z, c, t] columns.
That way we can access data associated with a region of the image through a simple query, irrespective of the current scale.

For meshes, we can have a mesh group, formatted as WKT (or PLY), and associated data can be stored in adjacent table groups, one for each element: vertex, edge, face.

The thing I don’t see how to setup with WKT is the link between the tables and the geometry elements. For ROIS, I assume each ROI would be an entry in the WKT file, so we can refer to the position of the entry, but for meshes it’s less clear.

In PLY it is easy to get a flat index for the elements as their number is declared in an element line in the header. The problem with PLY is the specification is super vague, so if you do more that 2D it seems difficult to decide how to call your elements (cell, volume, and thus parsers aren’t happy).

That’s it for my current thoughts, any insight welcome (especialy by the persons who work on ROIs, I’m afraid I don’t remember who that was.)

Guillaume

1 Like

:smile: Very cool. Glancing quickly at the code, am I right that the most important bit (for my purposes) would be:

? Also, worth splitting this to its own thread? ~J

Yes I think you’re right, it would be the important part, and yes the thread can be split for SMLM :slight_smile:

One question I have is how to write zarr images (from scratch). I was planning on copying the code in ome-zarr.data:

https://github.com/ome/ome-zarr-py/blob/ea299f3fa6963ad8700317df431696b1bfc37e57/ome_zarr/data.py#L106

If you see a way forward, feel free to open a PR against ome-zarr-py to make it reusable for your purposes, and we’ll tag it ASAP. Alternatively, say what you’re looking for, and I’ll update. A general :+1: for minimizing the number of implementations that we need to correct when the spec changes. ~J.

1 Like

just adding that we have a surface layer in napari that can accept arbitrary meshes and i’d love us to extend the ome-zarr napari plugin so that i could read mesh data into napari :slight_smile:

4 Likes

Adding a link to Code for multi-resolution mesh generation? · Issue #266 · google/neuroglancer · GitHub here* after it came up during yesterday’s discussions

~Josh


* at least until there’s an issue under Issues · ome/ngff · GitHub

Here it is:

Here is an implementation of ply in zarr:

Also relevant:

Best,
Guillaume

1 Like