Open part of tif stack

Hello,

@Wayne

I would like to open only a part of a tif stack using java specified by [xmin,xmax,ymin,ymax,zmin,zmax]
Currently I am using this class for opening:
https://imagej.nih.gov/ij/developer/source/ij/io/FileOpener.java.html
And my current code, which I basically copied from related classes, looks like below; for instance to only open one slice of a tif stack:

n = 10; // example z-slice that I want to load from the stack
FileInfo[] info = new Opener().getTiffFileInfo(directory+filename);
fi = (FileInfo) info[0];
long size = fi.width*fi.height*fi.getBytesPerPixel();
fi.longOffset = fi.getOffset() + (n-1)*(size+fi.gapBetweenImages);
fi.offset = 0;
fi.nImages = 1;
FileOpener fo = new FileOpener(fi);
return fo.open(false); // returns an imp

However, I do not know how I could restrict the width and height to a xy-ROI of one tif slice.

The issue is that we have many 2000x2000 pixel images and it takes (too much) time to load all of it into RAM. Is there a way to only open part of a tif slice in a fast way? Or would we have to go for other file formats such as h5?

P.S.:
Related inquires:


You can do that using Bio-Formats, see the example script by @ctrueden:

specifically this part:

from loci.common import Region
from loci.plugins.in import ImporterOptions

options = ImporterOptions()
options.setColorMode(ImporterOptions.COLOR_MODE_GRAYSCALE)
options.setCrop(True)
options.setCropRegion(0, Region(15, 25, 50, 70))
options.setId(file)

imps = BF.openImagePlus(options)
for imp in imps:
    imp.show()
3 Likes

Hi, Thanks!

I also found these options:

setZBegin(int s, int value) 
setZEnd(int s, int value) 

I guess combining this with

setCropRegion(0, Region(15, 25, 50, 70))

allows me to read a 3D cube, which is very nice.

However, I had one issue with this. If I do this repeatedly on huge tiff files BioFormats always reads all the tiff IFDs (each plane in a tiff stack can have some kind of header) which takes forever. In principle this is of course what one should do, but since in my case all the tiff files have the same structure my solution was to only read the IFDs (FileInfo[] info) from the first tiff file and then used them to open all the others.
This was a massive speed-up for us.
I don’t know how to do this with Bio-formats and I was thus looking for some solution in the frame-work of the FileOpener.

That’s a good point. Maybe reading all the IFDs of multi-page tiff files should be made optional? I’m not sure how SCIFIO’s TiffFormat currently handles this. At least you can also define a crop region when using SCIFIO via File > Import > Image… in Fiji.

@dgault, @ctrueden any comments?

1 Like

For any normal user interface I think you would want to read the IFDs just to be sure that you open the file correctly. What I am looking for is some kind of hack, where I can say

IFDs = getIDFs(PathToFirstFile)
ImagePlus imp = open(IDF[5], PathToSomeOtherFile)

In order to open, e.g. the 5th plane, of the OtherFile.

Wayne’s FileInfo class allows me to do this trick, because I can modify some entries such as

fi.filename
fi.directory

manually, while leaving others as is.

In fact, I wasn’t even aware of this.
Can one open a sequence of images (or even sequence of stacks) with this?
Looks like, but I did not get it too work…Is there an example somewhere?

If you have a small number of huge TIFF files, you can use the lower level Bio-Formats or SCIFIO API to get a reader, open plane by plane, convert the bytes to the correct primitive array type using DataTools, and wrap into an ImagePlus. Unfortunately, it is redoing a lot of the work that goes on in the BF (for Bio-Formats) and IO (for SCIFIO) API layers. You might be able to use the Bio-Formats ImageProcessorReader layers of Bio-Formats to successfully reuse most of the logic, while still also reusing the same reader stack (and hence not reparsing IFDs over and over).

If you have a large number of TIFF files, then I totally agree about being able to reuse the IFDs. This is not currently easy to do using Bio-Formats or SCIFIO, but is definitely the direction I want to go for performance reasons in the future. I believe that there are heuristics we can use which will make SCIFIO be able to detect most cases where reusing the same IFDs is likely to be safe. (And it can be made a flag in the options somewhere too.)

Another thing we will do more in the future (and which I thought the BF team explored and implemented, but I could be wrong) is more aggressive caching of IFDs, so they don’t need to be reparsed. But this would obviously only help if reopening the same TIFF file multiple times. In general, in SCIFIO, our plan is to fully cache all parsed metadata of all data sources for the lifetime of the JVM (barring memory limits or manually requested cleanups). And maybe even cache it to disk via serialization, for performance—historically I have seen huge speedups (20x) when doing this.

Very sorry I don’t have better news for you in the immediate term.

OK. Thanks for the clarification!

I feel my immediate options are:

  1. get the FileOpener hack to work with loading cropped regions
  2. save the data in h5, which doesn’t have the IFD issue, I hope?!

Here is some JavaScript code that uses FileOpener to open part of a TIFF stack. For each image, it reads [0,width,ymin,ymax], which is then cropped to [xmin,xmax]. In this example, where 2048x2048 images are cropped to 512x512, the FileOpener is reading 1/4 of each image.

path = "/Users/wayne/Downloads/Untitled.tif";
nSlices = 200;
xmin = 768;
ymin = 768;
xmax = xmin + 512;
ymax = ymin + 512;
zmin = 75;
zmax = zmin + 50;
//imp = IJ.createImage("Untitled","16-bit ramp",2048,2048,nSlices);
//IJ.saveAs(imp, "Tiff", path);
info = Opener.getTiffFileInfo(path);
fi = info[0];
offset = fi.offset;
width = fi.width;
height = fi.height;
size = fi.width*fi.height*fi.getBytesPerPixel();
stack = new ImageStack(xmax-xmin, ymax-ymin);
for (i=zmin; i<zmax; i++) {
     IJ.showProgress(i-zmin, zmax-zmin-1);
     fi.longOffset = offset + i*(size+fi.gapBetweenImages);
     fi.nImages = 1;
     fi.height = ymax - ymin;
     fi.longOffset += ymin*width*fi.getBytesPerPixel();
     fo = new FileOpener(fi);
     imp = fo.open(false);
     imp.setRoi(xmin, 0, xmax-xmin, imp.getHeight());
     imp = imp.crop();
     stack.addSlice(imp.getProcessor());
}
new ImagePlus("Cropped Stack", stack).show();
1 Like

Hi Wayne,

Thank you very much!

In fact, in the meantime I also figured out how to use stripOffset and stripLength to immediately only read out a part of the image without cropping later. I however believe that there is a slight issue with this (see below).

Right now I am using your openTiffStack method with a modified FileInfo.
The main part of the code is this:

// read crop: z,nz,x,nx,y,ny
for (int iz=z; iz<(z+nz); iz++){
        infoModified[iz-z] = (FileInfo) info[iz].clone();
        infoModified[iz-z].longOffset = infoModified[iz-z].getOffset();
        infoModified[iz-z].longOffset += (y*fi.width+x)*fi.getBytesPerPixel();
        infoModified[iz-z].offset = 0;
        infoModified[iz-z].stripLengths = new int[ny];
        infoModified[iz-z].stripOffsets = new int[ny];
        for (int i=0; i<ny; i++) {
            infoModified[iz-z].stripLengths[i] = nx * fi.getBytesPerPixel();
            infoModified[iz-z].stripOffsets[i] = (int) infoModified[iz-z].getOffset() + i * fi.width * fi.getBytesPerPixel();
        }
        infoModified[iz-z].height = ny;
        infoModified[iz-z].width = nx;
    }

ImagePlus imp = openTiffStack(infoModified);

The issue is that for me it only works in this case if I add infoModified[iz-z].getOffset() to each stripOffsets[i]. I also checked the source code and it seems that they are interpreted as absolute positions in the tiff-stack-file when using the openTiffStack because it uses the seek command of RandomAccessStream:


When it does not use the RandomAccessStream (the ‘else’ case) it seems to treat the stripOffsets relative to the current fi.longOffset. I believe it would be better if also in the RandomAccessStream case the stripOffset positions where interpreted relative to the fi.getOffset() positions, because the fi.getOffset() can be ‘long’ while the stripOffsets are only ‘int’ and will thus not suffice for large files.
What do you think? Or am I doing something wrong?

I suspect ImageJ would fail to open some TIFF stacks if I made your suggested change. What software creates the TIFF stacks you are trying to open? It would help if you could post an example stack.

For instance an ome.tif file, saved from ImageJ using Bio-formats exporter:
T0003.ome.tif.zip (598.3 KB)
Some holds true for Tiff stacks saved with MATLAB (I can sent you later if needed).

I found this page:
http://www.awaresystems.be/imaging/tiff/tifftags/stripoffsets.html
It seems that it is in fact the standard that the stripoffsets are specified with respect to the beginning of the file. However then it would be really good (in fact, necessary for SPIM data sets) if you could allow for ‘long’ values there. Is that possible?

Now, I am also getting the logic: the ‘else’ case is for the contiguous IJ tiff files where everything is deduced from the first IFD, in this case the stripOffsets of course need to be interpreted relative to where the input-stream pointer in order to work also for images deeper in the stack.

However, I think right now for the ‘else’ case at k=0 it will read starting from fi.getOffset() and not from fi.stripOffsets[0], which seems to be the “normal definition” according to above website. I think if(k==0) one would need to add also skip from fi.getOffset to fi.stripOffsets[0] in order to be consistent with the standard definition. What do you think?

The following updated version of my JavaScript example opens this file cropped. The fact that the offsets are 32-bit ints is not a problem since Bio-formats cannot create standard TIFF files larger than 4GB. Try to save a stack larger than 4GB using File>Save As>OME-TIFF and you get this exception:

loci.formats.FormatException: File is too large; call setBigTiff(true)

      path = "/Users/wayne/Downloads/T0003.ome.tif";
      nSlices = 200;
      xmin = 30;
      ymin = 30;
      xmax = xmin + 70;
      ymax = ymin + 70;
      zmin = 7;
      zmax = 68;
      info = Opener.getTiffFileInfo(path);
      fi = info[0];
      offset = fi.offset;
      width = fi.width;
      height = fi.height;
      stack = new ImageStack(xmax-xmin, ymax-ymin);
      for (i=zmin; i<zmax; i++) {
         IJ.showProgress(i-zmin, zmax-zmin-1);
         fi.longOffset = info[i].offset;
         fi.nImages = 1;
         fi.height = ymax - ymin;
         fi.longOffset += ymin*width*fi.getBytesPerPixel();
         fo = new FileOpener(fi);
         imp = fo.open(false);
         imp.setRoi(xmin, 0, xmax-xmin, imp.getHeight());
         imp = imp.crop();
         stack.addSlice(imp.getProcessor());
      }
  new ImagePlus("Cropped Stack", stack).show();
1 Like

More generally: there is no such thing as a “standard TIFF file” larger than 4GB. That’s what the BigTIFF standard is for.

Hi @ctrueden,

In fact, with my own code writing I now also arrived at a point where I want to store FileInfos to disk. However it turns out that ij.io.FileInfo currently is not serializable.
Do you know a work-around?

One idea I had was to make my own FileInfoSer class that only holds the serializable fields of FileInfo, like so:

public class FileInfoSer implements Cloneable, Serializable {

  public String fileName;
  public String directory;
  ...

    public FileInfoSer(FileInfo fi) {
      this.fileName = fi.fileName;
      this.directory = fi.directory;
      ...
    }

}

Are you sure you really need to do that?

If so, why not simply write two methods, writeFileInfo(FileInfo fi, File file) and FileInfo readFileInfo(File file), which persist the objects ad hoc?

You could also try hacking the IJ1 source code and make it Serializable, and if all appears to work well, ask @Wayne to add that interface.

That’s a good point. I could only use the "serializable FileInfo’ temporarily during writing and reading not to affect the rest of the code. In terms of hacking IJ1: as far as I understood it from reading some info on the web, the FileInfo class in IJ cannot be serialized as it contains non serializable fields like VirtualStack.