5D volume viewing in Python

guiwitz · August 28, 2019, 2:47pm

Hi everyone,

I’m currently putting together a short presentation on doing 3D image processing with Python and Jupyter. Reviewing existing visualisation tools, I noticed the absence of a general tool handling most classical cases of interactive volume visualisation. Here’s a short summary of existing tools and their limitations (they all are great tools that can also do other things than volume viewing and I don’t mean to be negative here, just trying to find out where the field is heading to):

https://github.com/maartenbreddels/ipyvolume : no colormaps choice, no time-lapse visualization
https://github.com/InsightSoftwareConsortium/itkwidgets: no multi-channel visualization, no time-lapse-visualization
https://github.com/maweigert/spimagine : no multi-channel visualization

So it seems that there’s no single tool that handles at the same time multi-channel + time-lapse + colormaps.

Therefore my questions here are:

Does anybody know about another tool that would be more complete?
Does anybody know if someone is working on a new tool?
Maybe someone has a solution I don’t know to tweak one of the above tools?

Thanks for any infos !

Cheers,

Guillaume

hanslovsky · August 28, 2019, 3:00pm

There is napari but I do not know how mature it is or if it is more complete than any of the above. Pinging @jni for some information

Another option would be to use the Java-based BigDataViewer (BDV) through imglyb or PyImageJ. BDV supports 5D data but AFAIK does not support arbitrary color maps out of the box. You would have to implement your own Converter into ARGB color space. Maybe a color map converter is readily available in the BDV world (cc @tpietzsch) or somewhere in ImageJ (cc @ctrueden).

kephale · August 28, 2019, 3:18pm

We are building a python interface for SciView based on @hanslovsky’s imglyb. It is in testing, but we’re waiting on one feature (poke @skalarproduktraum). This will provide BigDataViewer style volumes for 5D data. You can test it in ImageJ with BigDataViewer files, but we’re generalizing that to numpy arrays for this upcoming release.

guiwitz · August 28, 2019, 5:52pm

Thanks for the tips!

I had a look previously at Napari but it seemed still under heavy construction. I just tried it again now and the volume rendering seems to work nicely and has all the features I was looking for!

I of course already tried BigDataViewer through your imglyb package and it worked great. It’s nice to see that these two worlds are converging and I’ll definitely mention this solution in my presentation. Any hope to make it work on Mac OSX however ?

guiwitz · August 28, 2019, 4:43pm

That sounds great, I’ll be happy to test it as soon as it is officially released!

bnorthan · August 28, 2019, 5:42pm

@thewtex what is the latest with VTK python interfaces?? I know it is actually really easy to visualize volumes inn Python using the VTK interface, and I believe you’ve presented example notebooks in the past, however my understanding is there is not support for more than 4 channels, which is a potential road block for microscopy applications.

hanslovsky · August 28, 2019, 5:54pm

You would have to run it through the wrapper that is provided by imglyb.

Note that this wrapper is (probably) not necessary for JavaFX applications like, for example, Paintera. However, Paintera currently does not support time series and there is no straight forward way to start it from Python (unless you really know about both Python and JavaFX). The experimental Python wrapper payntera is outdated but may be a starting point.

hadim · August 28, 2019, 6:02pm

A Python interface for BDV/SciView would be awesome. Glad to hear you guys are working on it.

In the meantime, you can have a look at https://github.com/pyvista/pyvista, a Python interface to vtk. Development is very active.

ctrueden · August 28, 2019, 6:05pm

Are you talking about this issue?

There is a workaround. I (or anyone!) just need to find time to package it up in a nice way so things “just work” out of the box.

guiwitz · August 28, 2019, 6:12pm

It might be the same issue. One of the imglyb example notebooks has a note saying that one can only run it as a file through the wrapper:

github.com

imglib/imglyb-learnathon/blob/master/notebooks/bdv/bdv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BigDataViewer in imglyb\n",
    "\n",
    "Note that this notebook does not run on OSX! You can copy the contents of the relevant cells into a file and run the file through thw `OSXAWTwrapper`:\n",
    "```bash\n",
    "WRAPPER=\"$(python -c 'import os; import site; print( os.path.join(site.getsitepackages()[0], \"imglyb\", \"OSXAWTwrapper.py\") )')\"\n",
    "python $WRAPPER /path/to/file\n",
    "```\n",
    "This example uses, in addition to imglyb, the *h5py* and *scikit-image* packages. You can install them through conda:\n",
    "```bash\n",
    "conda install h5py scikit-image\n",
    "```"
   ]
  },
  {

This file has been truncated. show original

jni · August 28, 2019, 10:34pm

Thanks @guiwitz! It’s great to hear that napari fits your use case, which indeed is right in our ballpark. We are indeed still under heavy development but basically everyone on the team uses it in day-to-day work and it’s working great! (All credit to @sofroniewn for the state it’s in.) Please chime in with an issue if you have any feedback at all. We love to hear from new users!

thewtex · August 29, 2019, 7:09pm

https://github.com/InsightSoftwareConsortium/itkwidgets: no multi-channel visualization, no time-lapse-visualization

Multi-channel visualization is actively being implemented – feedback as drafts come out would be much appreciated ! Time-lapse-visualization is on the roadmap, but longer out.

@bnorthan Yes, at first there will support for up to 4 independently color-mapped channels. The limit is 4 because of technical and performance considerations working with GPU’s and Jupyter. But, we could create an additional rendering path that does some pre-processing CPU-side to support more channels.

@jni @sofroniewn @kephale @@skalarproduktraum @tpietzsch @ctrueden et al, what sequences of colormaps are used in napari, BDV, SciView, ImageJ?

Is there a best practice and standard we can provide throughout tools?

With many channels, what colormap sequence provides differentiation of channels without being “messy”?

What sequence of colormaps are perceptually meaningful when multiple channels are combined?

VolkerH · September 2, 2019, 1:43am

With more than 3 channels I find that no colormaps are perceptually meaningful, in particular for individual voxels that have significant contributions from all channels.
What does become important is the ability to quickly toggle the visibility and contrast/brightnes of each channel using a checkbox etc. (similar to Photoshop layers).

John_Xu · November 28, 2019, 10:42am

Hi @guiwitz,

I am wondering if you have better idea now? I am working on the same problem.

Do you mind to share your presentation or solutions?

Best regards,
John

guiwitz · November 29, 2019, 8:28am

Hi @John_Xu,

I guess my short answer is : go for napari ! This tool is very recent but already provides a very complete viewer for anyone dealing with multi-dimensional imaging. In particular it does exactly what I was asking for in this thread.

It has a very intuitive api that allows one to very quickly plot quite complex cases. If you want to see an example you can have a look at this demo that I wrote and in particular the napari_3d_image_processing.ipynb notebook. Clicking on the binder badge in that repository will also allow you to play live with the demo (just follow instructions in the repository and in the notebook).

napari is also in general written in a way that makes it super-easy to build on top of it even if you are not a professional coder. You can for an example look at the other example napari_ilastik.ipynb that I provided in the repository.

Let me also add that sometimes I’m reluctant to start using a very new open-source tools because it’s unclear e.g. how good support is and for how long it’s going to be supported. In this case, napari is developed by some of the best specialists in the domain, and they are supported by CZI, an institution that has shown clear willingness to support os software. Also the team and in particular @sofroniewn, is very responsive, so don’t hesitate to tag him with questions on this forum or directly on Github.
Cheers,
Guillaume

John_Xu · November 29, 2019, 12:44pm

Thanks a lot ! @guiwitz. I will give it a try!

Best regards,
John