A recipe to open fast a very large stack (~200 GB) without loading it all

Recently I found myself with ~200 GB stacks and larger. Opening these all at once requires a computer with a lot of RAM, or, using some of the tools provided by ImageJ and Fiji.

For the general case, it is true that LOCI BioFormats offers options to virtualize large stacks (even when the file is a single file, rather than a series of files, one per slice).

In my case, the file was a custom format that has a 512-byte header and then images are encoded in 16-bit unsigned. For this, we’ll use ImageJ’s “File - Import - Raw…” command, which is a true Swiss-army knife if only you have a bit of information on your data.

Step 1: "File - Import - Raw…"
Step 2: Select the file.
Step 3: Enter the data describing your volume:

  • width (e.g 6500) in pixels.
  • height (e.g. 6500) in pixels.
  • header size (e.g. 512) in number of bytes between the start of the file and the start of the first slice.
  • the number of slices. If unknown, type a large number, larger than you expect.
  • pixel type (e.g. unsigned short for 16-bit range of values from 0 to 65535).
  • whether the encoding was in little endian or big endian. Generally it’s little endian. Try both when unsure.
  • open as virtual stack: True (tick the box!)

The last step is essential: the single 200-GB file will not be loaded in full, but rather, only the relevant chunks to show one slice at a time will be loaded.

Now, moving the slice slider at the bottom of the image window will be painful: it takes too long to locate each slice and open it, and therefore you shouldn’t use it: moving the slider will result in dozens of slices being read, which can cost you many seconds or even minutes (if the stack lives on a network-mounted drive).

Instead, use a script to jump straight to specific slices you want to see. For example, open the Script Editor, choose the “Python” language and type:

from ij import IJ

imp = IJ.getImage()
imp.setSlice(1567)

Now, slice 1567 will be shown. This slice is fully loaded in RAM, so drawing an ROI on it, and duplicating the image (or copy and paste into a new image, or creating a new image from the internal paste buffer) will be instantaneous. On this copy, run any script or plugin or adjust the contrast as desired.

Note that if the slice is beyond the existing range of slices, it will be ignored silently.

Hope this helps with managing the increasingly large image stacks available to biologists nowadays.

12 Likes

Thanks for sharing, @albertcardona!

With script parameters, you can have this even shorter:

#@ ImagePlus imp
imp.setSlice(1567)

And for those people who don’t like scripts at all, the functionality is available from the menu as Image > Stacks > Set Slice…

3 Likes

Hi,
Can you explain how you know the header length and on how many bits you image is (and if it is signed or unsigned)?
I usually deal with .nd2 files lately…
Thanks!

If the image is uncompressed, then the header length is easy: fileSize - width * height * pixelDepth.

The image, if it originated in a scientific instrument, will almost always (should always be) unsigned.