ND2 in Python redux

jni · June 28, 2019, 8:11am

Hi all,

I have a set of nd2 files that I can’t read with either pims_nd2 or nd2reader. I have shared an example file here (warning: it’s 36GB). If I read a single frame in random-access, it works fine. But if I try to read a bunch of frames in sequence, it fails. With pims_nd2, I get a LIM_ERR_FAIL error. With nd2reader, I get a “ValueError: the file seems to be corrupted” error.

I’ve made a test Python script in this gist, shared also with the tracebacks I get from both libraries. Run python test_nd2.py --help for usage info. The program creates a dask array (lazy numpy array) from the file based on this code from dask_image, but any sequential access to the frames fails.

I presume there will not be a short term fix to this, but I wanted to provide the file as an example for the aforementioned libraries.

I’ll also note that I could find the metadata for “channel name” and “channel colour” in pims_nd2, but not in nd2reader…

jni · June 28, 2019, 8:22am

I feel like I should explain that the title is in reference to this thread:

But I felt like this issue warranted a new thread.

bioimage-analysis · November 24, 2020, 5:03am

Hi @jni, I wonder if you were ever able to resolve that issue?

jni · November 24, 2020, 12:06pm

Yes and no!

No, nd2reader still fails after a few frames of random access. But yes, we managed a workaround, which is to open and close the file to load each frame. This is slower than it should be but it works! @DragaDoncila has this still-hacky nd2->dask array reader here:

github.com

DragaDoncila/nd2-dask/blob/42a91e64b2031e3e19c4d25fcaa5d31b13548f67/nd2_dask/nd2_reader.py

"""
This module is an example of a barebones numpy reader plugin for napari.

It implements the ``napari_get_reader`` hook specification, (to create
a reader plugin) but your plugin may choose to implement any of the hook
specifications offered by napari.
see: https://napari.org/docs/plugins/hook_specifications.html

Replace code below accordingly.  For complete documentation see:
https://napari.org/docs/plugins/for_plugin_developers.html
"""
import numpy as np
from nd2reader import ND2Reader
from dask import delayed
import dask.array as da
import toolz as tz
from napari_plugin_engine import napari_hook_implementation
import dask
from vispy.color import Colormap

This file has been truncated. show original

It includes a napari plugin so you can view nd2 files in napari with that, which is pretty nice! BUT, currently a bunch of stuff, like channel colours, is hardcoded to the data we are working with. We’d definitely welcome pull requests to improve things!

bioimage-analysis · November 24, 2020, 6:10pm

Thank you, that is exactly was I was looking for!