Imglyb + BigDataViewer

Hi everyone,

I have recently switched our light-sheet microscope control code to Python. Everything is working well, and I am now hoping to use imglyb to visualize the data as the scan progresses. To quickly summarize the workflow of the current code, data is streamed from the camera to a numpy array -> the numpy array is saved to an H5 file with a resolution pyramid -> an XML file is written for the H5 file -> this process is repeated for N imaging tiles and M color channels for N*M total tiles. The tiles are written into the H5/XML files formatted for downstream processing for BigStitcher.

My question are:

  1. Whether it is possible to use imglyb to open and display the H5/XML file during the acquisition as the imaging progresses. And update it after a new image tile is added. This would be ideal, since the XML file already contains all of the affine transformations to deskew and roughly align the tiles.

  2. If not, if it is possible to add multiple image volumes to the same bdv instance with imglyb. And if it is possible to apply a separate affine transformations to each volume? I have found this method util.BdvOptions.options().sourceTransform() but it is not clear to me what the inputs to sourceTransform() are. And it seems to apply the same sourceTransform to all volumes I add to the same bdv instance. Example code snippet below:

bdv = util.BdvFunctions.show(vol1, 'vol1',util.BdvOptions.options().sourceTransform(1.0, 2.0,1.0))
util.BdvFunctions.show(vol2, 'vol2', util.BdvOptions.addTo(bdv))
  1. Whether imglyb is the correct way to go about this, or if an easier method might be to use pyimagej to open the H5/XML file using the BigDataViewer or BigStitcher plugins as the imaging progresses.

Thanks!
Adam

  1. I am not familiar with the BDV h5/xml format and I do not know what the best way is to open a BDV h5/xml file and if it’s possible through BdvFunctions.

  2. It is possible to add multiple image voumes to the same bdv instance by passing util.BdvOptions.options().addTo(bdv). I would probably generate a RandomAccessibleIntervalMipmapSource like in this Java example code (it should translate to Python fairly easily through the by importing Java classes through jnius.autoclass):

import bdv.util.BdvFunctions;
import bdv.util.BdvOptions;
import bdv.util.BdvStackSource;
import bdv.util.RandomAccessibleIntervalMipmapSource;
import net.imglib2.RandomAccessibleInterval;
import net.imglib2.img.array.ArrayImgs;
import net.imglib2.realtransform.AffineTransform3D;
import net.imglib2.type.numeric.integer.UnsignedByteType;
import net.imglib2.util.Util;

public class BdvVistoolsTest {

	public static void main(String[] args) {
		final RandomAccessibleInterval<UnsignedByteType>[] imgs = new RandomAccessibleInterval[] {ArrayImgs.unsignedBytes(10, 20, 30)};
		final double[][] mipmapScales = {{1.0, 1.0, 1.0}};
		final AffineTransform3D transform = new AffineTransform3D();

		final RandomAccessibleIntervalMipmapSource<UnsignedByteType> source = new RandomAccessibleIntervalMipmapSource<>(
				imgs,
				Util.getTypeFromInterval(imgs[0]),
				mipmapScales,
				null,
				transform,
				"name");

		final BdvStackSource<UnsignedByteType> bdv = BdvFunctions.show(source);
		BdvFunctions.show(source, BdvOptions.options().addTo(bdv));

	}

}

You would need to populate imgs with your numpy arrays through imglyb.to_imglib and set the mipmapScales and transform appropriately.

  1. Again, not familiar with the bdv h5/xml format but I suggest that depends on your intended use case: If you would like to modify the images in place both with Java and Numpy with shared memory, then you should definitely use imglyb. For any other case it probably doesn’t matter that much and you should use whichever approach is the most practical.

As I am not very familiar with the BDV h5/xml format (and how to open it), I’ll ping @ctrueden for hints on how to open an h5/xml file through pyimagej and @tpietzsch for the best ways to open h5/xml files through BigDataViewer or BdvFunctions and appending to an existing instance.

2 Likes

The API will be analogous to how it’s done in Java. So we just need to hear from @tpietzsch or other person more familiar with the BDV API to recommend the best path forward there.

1 Like

This is how to open h5/xml file and display using BdvFunctions:

For appending, it will probably not work, as the attempts to read missing h5 stacks are cached. But I’m not completely sure about this, and the peculiarities of your setup. It’s worth trying, maybe it just works.

If not: You can leave the BDV window open, remove the old source and replace with newly loaded h5/xml.

The above BdvFunctions.show(spimData) gives a List<BdvStackSource<?>>, just call BdvStackSource.removeFromBdv() on each of them. I would first add the new spimData, then remove the old sources, to make sure window is not closed or viewer transformations reinitialized (again, I don’t remember whether that would happen if you do it the other way around, but just to be sure…)

1 Like

Thanks @hanslovsky @ctrueden @tpietzsch for the input. Very helpful and informative! It sounds like the last option from @tpietzsch would be best since at the moment I don’t need to manipulate the data. I simply want to visualize it as the scan progresses.

I tried implementing it in Python. It is not throwing any errors… but at the same time it isn’t opening and displaying a BDV viewer. Code below, also the terminal output of the code. I must be going wrong somewhere… Also, if I add any other inputs to the show() command, such as util.BdvFunctions.show(spimData, ‘test’) I get the error below. Any tips or thoughts on where I might be going wrong? Thank you!

Code

xmlFilename = 'X:\\S006_1\\data.xml'
spimData = autoclass('bdv.spimdata.SpimDataMinimal')
print(spimData)
spimXML = autoclass('bdv.spimdata.XmlIoSpimDataMinimal')
print(spimXML)
spimData = spimXML().load(xmlFilename)
print(spimData)
bdv = util.BdvFunctions.show(spimData)
print(bdv)
bdv_exit_check(bdv.getBdvHandle().getViewerPanel())

Output

<class 'jnius.reflect.bdv.spimdata.SpimDataMinimal'>
<class 'jnius.reflect.bdv.spimdata.XmlIoSpimDataMinimal'>
<mpicbg.spim.data.generic.AbstractSpimData at 0x1eaee2f8fc0 jclass=mpicbg/spim/data/generic/AbstractSpimData jself=<LocalRef obj=0x-154a2eb0 at 0x1eaee23e290>>

Error

Traceback (most recent call last):
  File "test2.py", line 35, in <module>
    bdv = util.BdvFunctions.show(spimData, 'test')
  File "jnius\jnius_export_class.pxi", line 1044, in jnius.JavaMultipleMethod.__call__
  File "jnius\jnius_export_class.pxi", line 760, in jnius.JavaMethod.__call__
  File "jnius\jnius_conversion.pxi", line 78, in jnius.populate_args
  File "jnius\jnius_utils.pxi", line 205, in jnius.check_assignable_from
jnius.JavaException: Invalid instance of 'mpicbg/spim/data/generic/AbstractSpimData' passed for a 'net/imglib2/RandomAccessibleInterval'

I have implemented the same style code for BigVolumeViewer using the examples at: https://github.com/tpietzsch/jogl-minimal and to my surprise this is working… Not sure what the problem with BDV is? I have some questions about BVV, but will move those to a separate thread. Thanks again for the continued help! It is excited to use these tools in Python and integrate directly with microscope control for real-time viewing of data coming off the system.

This looks to me like it’s trying to call the wrong BdvFunctions.show() signature is called.
This is strange, because there is no exact BdvFunctions.show(RandomAccessibleInterval) signature. I have no idea what is throwing pyjnius off or why it works with BigVolumeViewer.

If you have any java component in what you are doing, maybe you can just make a small wrapper where it is less ambiguous. And then call MyShow.show(spimData) from python? Does that work?

public class MyShow
{
	public static List< BdvStackSource< ? > > show( final AbstractSpimData< ? > spimData )
	{
		return BdvFunctions.show( spimData );
	}
}
1 Like

PyJNIus sometimes has problems resolving the correct signature if a method is overloaded. You can try casting to AbstractSpimData explicitly with the jnius.cast method when passing the source to BdvFunctions.show to exactly match the Java method signature, something like this:

from jnius import cast
spimData = ...
bdv = util.BdvFunctions.show(cast('mpicbg.spim.data.generic.AbstractSpimData', spimData))

I am not sure that this will work but it may be worth a try.
Possibly related issue: https://github.com/kivy/pyjnius/issues/427

@adamkglaser this forum has markdown support and you can enclose code or error messages in triple backticks (```) on the lines before and after for better readability.

1 Like

Great idea! You don’t actually need a Java component for that but you can compile the class on the fly as long as you have a complete JDK and not just a JRE, e.g, something like this (not a working example):

import jnius_config
import pathlib
import subprocess
import tempfile

tmp_dir = tempfile.mkdtemp()
jnius_config.add_classpath(tmp_dir)

import imglyb
from jnius import autoclass

java_code = '''
import ...List
import ...BdvStackSource
import ...AbstractSpimData
import ...BdvFunctions

public class MyShow
{
	public static List< BdvStackSource< ? > > show( final AbstractSpimData< ? > spimData )
	{
		return BdvFunctions.show( spimData );
	}
}
'''
fp = pathlib.Path(tmp_dir) / 'MyShow.java'
with open(fp, 'w') as f:
    f.write(java_code)

# this may be a different location on Windows or macOS
javac = pathlib.Path(os.environ[ 'JAVA_HOME' ]) / 'bin' / 'javac'
proc = subprocess.run( 
    [ javac, '-cp', jnius_config.split_char.join(jnius_config.get_classpath()), fp ],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE)
if proc.returncode != 0:
    print (proc.stderr)
MyShow = autoclass('MyShow')

See also: https://github.com/imglib/imglyb-learnathon/blob/master/notebooks/bdv/bdv.ipynb

Thanks everyone. @hanslovsky I tried running your code but keep returning the error below. Not sure what the error could be? It looks like the proc is being executed properly. Also on Windows it seems like the inputs to proc have to be converted from WindowsPath to String, otherwise an error is thrown. Code and output below.

Code

import jnius_config
import pathlib
import subprocess
import tempfile
import os

tmp_dir = tempfile.mkdtemp()
jnius_config.add_classpath(tmp_dir)

import imglyb
from jnius import autoclass

java_code = '''
package bdv.util;
import java.util.List;
import bdv.util.BdvStackSource;
import mpicbg.spim.data.generic.AbstractSpimData;
import bdv.util.BdvFunctions;

public class MyShow
{
	public static List< BdvStackSource< ? > > show( final AbstractSpimData< ? > spimData )
	{
		return BdvFunctions.show( spimData );
	}
}
'''
fp = pathlib.Path(tmp_dir) / 'MyShow.java'
with open(fp, 'w') as f:
    f.write(java_code)

# this may be a different location on Windows or macOS
javac = pathlib.Path(os.environ[ 'JAVA_HOME' ]) / 'bin' / 'javac'

proc = subprocess.run( 
    [ str(javac), '-cp', str(jnius_config.split_char.join( jnius_config.get_classpath()) ), str(fp) ],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE)

print('proc output')
print(proc)

if proc.returncode != 0:
	print (proc.stderr)

MyShow = autoclass('MyShow')

Output

proc output
CompletedProcess(args=['C:\\Users\\AERB\\Anaconda3\\envs\\imagej\\Library\\bin\\javac', '-cp', 'C:\\Users\\AERB\\AppData\\Local\\Temp\\tmpymdid6wd;C:\\Users\\AERB\\Anaconda3\\envs\\imagej\\share\\pyjnius\\pyjnius.jar;C:\\Users\\AERB\\.jgo\\net\\imglib\\imglib2-imglyb\\0.3.0\\*;C:\\Users\\AERB\\Anaconda3\\envs\\imagej\\lib\\site-packages\\jnius\\src', 'C:\\Users\\AERB\\AppData\\Local\\Temp\\tmpymdid6wd\\MyShow.java'], returncode=0, stdout=b'', stderr=b'')
Traceback (most recent call last):
  File "test3.py", line 46, in <module>
    MyShow = autoclass('MyShow')
  File "C:\Users\AERB\Anaconda3\envs\imagej\lib\site-packages\jnius\reflect.py", line 159, in autoclass
    c = find_javaclass(clsname)
  File "jnius\jnius_export_func.pxi", line 26, in jnius.find_javaclass
jnius.JavaException: Class not found b'MyShow'

In your java_code string you define a package bdv.util but you import the class MyShow without the package. I suggest you try removing the line package bdv.util from your java_code string (preferred) or importing MyShow = autoclass('bdv.util.MyShow'). Let me know if that works.

Update: Based on proc.returncode == 0 I assume that compilation worked successfully.

Thanks @hanslovsky. Removing bdv.util fixed the issue. The code now works for both BigDataViewer by doing MyShow.show(...).

However, it only seems to work with BDV H5 files containing a single imaging tile. For example, the fused output of BigStitcher. However, it does not work with BDV H5 files containing multiple imaging tiles, such as the input BDV H5 files for BigStitcher. This is ultimately the goal, to be able to open and visualize the BDV H5 files on the fly as each image tile is added in.

The strange thing is that opening these same BDV H5 files with multiple imaging tiles works through Fiji -> BigDataViewer -> Open XML/HDF5. So there must be some additional lines or classes necessary in the Python code to properly loop over or parse through files containing multiple imaging tiles?

@adamkglaser maybe you can get some insight from the BDV plugin source code?

Thanks @hanslovsky. It looks like all the plugin is doing is parsing the filename and then running:

BigDataViewer.open( file.getAbsolutePath(), file.getName(), new ProgressWriterIJ(), ViewerOptions.options() );

Do you think I could autocast this method and then try calling it through Python? Or would it be better to compile on to go, as you helped me get working above for the MyShow() example. The Java code for BigDataViewer.open() is:

public static BigDataViewer open( final String xmlFilename, final String windowTitle, final ProgressWriter progressWriter, final ViewerOptions options ) throws SpimDataException
	{
		final SpimDataMinimal spimData = new XmlIoSpimDataMinimal().load( xmlFilename );
		final BigDataViewer bdv = open( spimData, windowTitle, progressWriter, options );
		if ( !bdv.tryLoadSettings( xmlFilename ) )
			InitializeViewerState.initBrightness( 0.001, 0.999, bdv.viewer, bdv.setupAssignments );
		return bdv;
	}

Actually, that should work.
Can you send an example XML/HDF5 so that I can have a look?

1 Like

My recommendation is to always try to call it through Python first. I consider on-the-fly compilation only a workaround if the former does not work.

Thanks everyone. I ran some more tests this morning and determined what the issue may be.

I tested many things, but what seems to be the issue is the type of compression used in the H5 file. The python code we’ve discussed above works with BigStitcher files if they are uncompressed, or compressed using GZIP as an example.

However, we are currently using B3D (https://git.embl.de/balazs/B3D) to compress our datasets. This is what must be causing the issue. For our B3D compressed H5 files, whether it is a single tile or multiple tiles, the Python code does not work. No errors are thrown, but no BDV window opens during execution.

What is strange though, is that opening these B3D compressed H5 files works through ImageJ -> BigDataViewer -> Open XML/HDF5. Not sure what the difference could be? @tpietzsch I can send along a small example B3D compresed dataset link through a private message. Although I’m not sure if you are equipped on your end to work with it.

Thanks everyone,
Adam

1 Like

As a follow up, I’m guessing the issue may be that the imglyb bdv is calling from an invalid HDF5 distribution? For using B3D with Fiji, you have to explicitly use their version of a jdhdf5 JAR that is compiled with a compatible Visual Studio version. This may explain why I can open the files through Fiji (where I have inserted their JAR) but not through python with imglyb?

If this is the source of the issue, is there a way to explicitly have imglyb, or compile java code on the fly, which would call the same jhdf5 JAR file I have locally for my Fiji installation?

Text below pasted from the B3D github page.

Reading files in Fiji

Fiji comes with its own distribution of HDF5 in the form of jhdf5-14.12.5.jar in the Fiji.app/jars folder. On Windows machines this is not compatible with the filter because a different version of Visual Studio was used to compile the native libraries. To circumvent this issue, we provide a recompiled version of the same file, included in the win64 folder. Copy this file to the Fiji.app/jars folder, overwriting the original if necessary.

To open HDF5 files, you need to enable the HDF5 update site (Help > Update, then click on “Manage update sites” and select “HDF5” from the list). During the update process you might get a warning message: “There are locally modified files”. As this is a consequence of the previous step, it’s important not to let Fiji overwrite the locally modified files.

Thanks!
Adam

You could try adding the B3D jhdf5 jar to the class path with the scyjava_config module by either one of scyjava_config.add_endpoints or :

# run these lines before importing imglyb or import jnius 
import scyjava_config
scyjava_config.add_endpoints(f'{groupId}:{artifactId}:{version}')
scyjava_config.add_classpath('/path/to/jhdf5.jar')

I am not sure that this will solve the issue but it is worth a try.

Without a doubt, the difference in classpaths is the culprit.

You can wrap a local Fiji installation using imagej.init('/path/to/Fiji.app'), which would include all the JARs in that installation. But if you use imagej.init() or imagej.init('sc.fiji:fiji') then of course you won’t have any classpath modifications.