Reading hyperstack tiff saved by micromanger with python

Hi all,

I have some light sheet data acquired using Micro-manager, all time-channel-z-y-x hyperstacks. The size of the dataset is between few hundreds of GB to a few TB. And I am trying to read out the data using tifffile in Python for further processing.

Micro-manager saves the hyperstack into many 4GB-sized tiff files. It can handle the dataset well using virtual stack and I can explore through the dataset easily.
However, when I try to read the data using tifffile in Python, I got errors:
“TiffPage 0: TiffFileError: invalid tag value offset”.

I tried a few solutions:

  1. Open the 4GB-sized tiff files with Fiji and save them to tiff files which are then readable by tifffile.
    However reading out the tiff files gets very slow with Fiji after the first few tiff files.
  2. Saving the hyperstack with Micro-manager to single-page tiff files. This strategy worked for a dataset which had ~ 100k single-page tiff files. But for another dataset where I have over 1 million single-page tiff files, saving them into a single directory apparently made things very slow again.

Anyone here in the forum having any suggestion or experiences with similar problems, I would really appreciate your inputs.

Thanks for your attention,
Bin

1 Like

Please share a file and a short script to reproduce the error.

Hi Bing,

Have you monitored the computer’s RAM to make sure that Fiji releases memory after closing a file? In my experience, there is a memory leak in Fiji that becomes a big deal with large files and that is still an open issue [1]. What works for me is to call the garbage collector by double-clicking on the status bar of Fiji or through a macro [2].

[1] Memory not clearing over time
[2] https://imagejdocu.tudor.lu/faq/technical/how_do_i_run_the_garbage_collector_in_a_macro_or_plugin

Hi Bin,

One (possibly not very helpful) thing that we have used in the past to quickly get around the issue of large numbers of individual TIFF files being heavy on the OS file handling vs stacks is to make the stacks into linear montages and undo the operation in the processing software at the other end as appropriate. Handling a single (very large) file has less overhead than many small ones.

Regards,
Sunil