Rescue a >4GB label image saved from napari?

I am afraid I have done something kind of foolish. I am just starting to learn to use napari and I got ahead of myself. I was painting labels in a label layer on a fairly large 3D stack (RGB, 7506x3558x37 slices), and it was going well. But then I wanted to save the label layer to come back to later. I had the label layer layer selected and then I clicked ‘File>Save Selected Layers’ in the GUI. And then foolishly I closed the napari session before I confirmed that the saved image was valid.

The resulting label image was ~7.91GB. I have a feeling it was not saved with a BigTiff flag. When I try to open this image with tifffile.imread('filename.tif') or anything built off of it (opening directly with napari filename.tif at the command line, or skimage.io.imread('filename.tif') (I think that is basically a fancy wrapper around tifffile.imread ?) I get the following error:

OSError                                   Traceback (most recent call last)
<ipython-input-20-3623d7d33559> in <module>
----> 1 x = tifffile.imread('interstitial_label.tif')

~/.virtualenvs/napari/lib/python3.6/site-packages/tifffile/tifffile.py in imread(files, **kwargs)
    719         if isinstance(files, (str, os.PathLike)) or hasattr(files, 'seek'):
    720             with TiffFile(files, **kwargs_file) as tif:
--> 721                 return tif.asarray(**kwargs)
    722 
    723     with TiffSequence(files, **kwargs_seq) as imseq:

~/.virtualenvs/napari/lib/python3.6/site-packages/tifffile/tifffile.py in asarray(self, key, series, level, out, maxworkers)
   2803                 self.filehandle.seek(series.offset)
   2804                 result = self.filehandle.read_array(
-> 2805                     typecode, product(series.shape), out=out)
   2806         elif len(pages) == 1:
   2807             result = pages[0].asarray(out=out, maxworkers=maxworkers)

~/.virtualenvs/napari/lib/python3.6/site-packages/tifffile/tifffile.py in read_array(self, dtype, count, out)
   7494             raise ValueError('size mismatch')
   7495 
-> 7496         n = fh.readinto(result)
   7497         if n != size:
   7498             raise ValueError(f'failed to read {size} bytes')

OSError: [Errno 22] Invalid argument

I am concerned that this is because the tiff file saved by napari was >4GB.

I can open this image in ImageJ with BioFormats, but the image is 32-bit of all zeros, no label information at at all.

In the future I will try to export it via the actual napari API rather than the GUI, but having sunk a few hours into this annotation, is there any way to get my labels out of this tif file? Or is this just a live and learn scenario?

Thanks so much!

Update: By opening in BioFormats and then exporting in Bioformats in ImageJ I am at least able to load the image now in python, but it appears to all be zeros. So perhaps the actual answer to this question is that somehow none of the label data was saved in the first place? I had the “Label” layer selected so that it was blue…A reminder to check things more carefully next time!

@akennard please don’t blame yourself for this! It’s definitely a situation where napari should have worked!

On our end we use (or should have used!) tifffile.imsave to save (code here), and I would expect that tifffile.imsave and tifffile.imread would do a round trip, or raise an error in the first place. On my machine, tifffile does work, saving np.random.randint(0, 5, size=(7506, 3558, 37)) and reloading it no problem. Could you test whether you can roundtrip a similar array with tifffile on the same environment as your previous attempt?

One possibility is that tifffile did raise an error but napari masked it — our error catching is a bit subtle currently, if launched from the command line! (See this comment, for example.) But as mentioned above, tifffile should work — please report your tifffile version just in case. Another possibility is that napari saves in the background (I haven’t tried this), and you interrupted an incomplete save in closing it?

Again, I’m not sure. I would recommend trying a few things out on your end with sample volumes like the random one I mentioned above to see if you can (a) get the save/load to work, and (b) figure out what happened with your current tiff.

Incidentally for this size of volume I would recommend saving to a .zarr. In fact, you can open a .zarr for read/write and add that to napari as a labels layer, and painting goes directly to disk, no need to save!

Based on the forensics you’ve done so far I suspect that tiff is lost, but it would be great if you could help us figure out exactly what went wrong so we can prevent other users from having the same experience!

1 Like

Without specifying any other options, tifffile.imsave uses the BigTIFF format for array sizes > ~4 GB and stores the array data contiguously.

I don’t understand how a 32-bit 7506x3558x37 image can result in a ~7.91GB file. Probably it’s a 64-bit integer labeled image.

This indicates a limitation/bug in the Python interpreter, which is apparently unable to read > 4 GB at once. There are reports that even Python 3.6.2 is still affected by this bug on some platforms. Try a newer Python version to read the file.

2 Likes

This fixed it! I was using Python 3.6.4. I upgraded to Python 3.8.6 and I was able to load the image with imread and recover all the labels to napari. Thanks a lot!

Yes I think that is exactly right. I got to ‘32-bit’ just from looking at how it was loaded in Fiji/Bioformats, but 64-bit is not a common datatype there, so I suspect it tried to force the 64-bit image into a 32-bit datatype, and in doing so all the data was lost upon opening with Bioformats.

2 Likes

Awesome! That is so great to hear. Thank you @cgohlke as always!

On the napari side this is great news since we are dropping py3.6 support in the next version, so that should prevent repeats. :blush:

2 Likes