Bioformats saving with compression is too slow

I’m also posting here as the bio-formats forum seems down.

Any format I want to use using BioFormats Export takes ages

Example: The 33MB Mitosis stack takes oaver 15 minutes to resave as a compressed tiff (JPEG2000). I’ve tried other formats and it’s the same, about one slice per second…

I’m experiencing this under Win10 and Java 8…

Anyone got any thoughts on this?

Best

Oli

1 Like

Nothing constructive to contribute beyond the fact that I have been using BioFormats from Apple OS-X an Java 8 and also experienced saving files with compression can significantly increase the time take to save.

The time seems to be consumed when saving each slice, the writeBytes() call, so at least you put in a progress bar.

– Michael Ellis

1 Like

Same experience here, saving Tiff with zlib or lzw compression is super slow.

Is there any interest from BioFormats to increase performance? @joshmoore

Does someone know other Java Tiff stack writers supporting compression? @Wayne

One would have to do some benchmarking to find out what is the slow part.

@oburri
Would you be interested to benchmark using BioFormats without compression (and with different compression modes) and compare this to just using plain ImageJ to save a Tiff?

@joshmoore @dgault
I did dig a bit into the TiffSaver code to look for places of improvement.

Compression of one plane

This seems to be the place where the compression for one plane happens:

for (int strip=0; strip<nStrips; strip++) {
      strips[strip] = stripBuf[strip].toByteArray();
      TiffCompression.difference(strips[strip], ifd);
      CodecOptions codecOptions = compression.getCompressionCodecOptions(
          ifd, options);
      codecOptions.height = tileHeight;
      codecOptions.width = tileWidth;
      codecOptions.channels = interleaved ? nChannels : 1;

      strips[strip] = compression.compress(strips[strip], codecOptions);
      if (LOGGER.isDebugEnabled()) {
        LOGGER.debug(String.format("Compressed strip %d/%d length %d",
            strip + 1, nStrips, strips[strip].length));
      }
    }

Suggestion: maybe one could parallelise this code in terms of generating the compressed strips?
Looks like a classic embarrassingly parallel code already to me, but maybe I am overlooking something?

Compression of a Tiff stack

/**
   */
  public void writeImage(byte[][] buf, IFDList ifds, int pixelType)
    throws FormatException, IOException
  {
    ...
    for (int i=0; i<ifds.size(); i++) {
      if (i < buf.length) {
        writeImage(buf[i], ifds.get(i), i, pixelType, i == ifds.size() - 1);
      }
    }
  }

This is where you seem to loop over the different image planes.

If one wants to be fancy one could parallelise the compression of some planes with the writing of others. However, from my experience just writing a Tiff stack can be very fast (at least with ImageJ).

Thus, suggestion: Maybe one could simply give the option to compress everything (the whole stack) in RAM (e.g., parallelising over planes) and then in one go write the whole stack. I am not sure, but I think this could help because (on top of the parallelisation) one does not have this “compute a little bit, write a little bit, compute a little bit, write a little bit, …” cycle for which my feeling is that this is less efficient than “compute everything, write everything”. The reason I think so is that I feel that for the underlying file system it is nicer to “get things over with” quickly rather than keeping the OutputStream open for minutes with only intermittently something actually being written…

One issue, at least when using bfconvert, is that OME-TIFF files are saved with RowsPerStrip=1 by default. Each scanline is compressed separately. Even for moderately sized images, the number of strips is in the tens/hundreds of thousands. Compressed strips are often only a few bytes long. There’s a lot of overhead when writing and reading such files that can be avoided by using larger RowsPerStrip. The TIFF specification from 1992, when RAM was measured in MB, recommends strip sizes of 8K uncompressed. Tifffile uses 64K by default. Zarr recommends at least 1M uncompressed for chunks.

1 Like

This makes tons of sense! I would also for sure look into changing the rowsPerStrip default value in BioFormats along those lines.

@dgault @joshmoore
I would in fact be motivated to give it a go in terms of improving the performance of the current code. Should I just go ahead and let you know about how it goes and then you can think of merging it at some point?

1 Like

Try saving using FIle>Save As>ZIP. It should be faster.

Thanks, yes I thought about it, but can one open the resulting files with anything else but ImageJ?
They currently end on .zip such that other software would not recognise them as Tiff files, isn’t it?

Of course! And especially to have help doing so. But I’d note that this post was before @ome joined image.sc, so for everyone: if there’s a thread from before Dec '18 (I2K Heidelberg) that we’ve missed, sorry! Please ping.

3 Likes

I started with some benchmarking.

Data: 

BitDepth: 16
nX: 2048
nY: 2048
nZ: 401

Saving to disk as Tiff:
 
ImageJ (fileSaver.saveAsTiffStack):  22s
BioFormats (looping nZ times over tiffWriter.saveBytes, UNCOMPRESSED): 71s
BioFormats (looping nZ times over tiffWriter.saveBytes, LZW COMPRESSION plane-wise): 80s
BioFormats (looping nZ times over tiffWriter.saveBytes, LZW COMPRESSION row-wise): 93s

plane-wise: rowsPerStrip = 2048
row-wise: rowsPerStrip = 1

As already the uncompressed saving is much slower than with ImageJ, it does not look like it is worth, at this point, to optimise the compression per se, but more looks like an issue with the TiffWriter in general.

Other opinions and/or do you maybe already have an idea what the slow part in the BioFormats code could be? @dgault @joshmoore (cc @ctrueden)

@oburri did you make similar observations when you did your testing?

For completeness, I am also pasting my code where I do the for-loop over the tiffWriter, as it could also be that I am doing something suboptimal there?!

            ImageWriter writer = getImageWriter( imp, path );

            if ( settings.compression.equals( SavingSettings.COMPRESSION_ZLIB ) )
                writer.setCompression( TiffWriter.COMPRESSION_ZLIB );
            else if ( settings.compression.equals( SavingSettings.COMPRESSION_LZW ) )
                writer.setCompression( TiffWriter.COMPRESSION_LZW );

            TiffWriter tiffWriter = (TiffWriter) writer.getWriter();

            if ( rowsPerStrip == -1 )
                rowsPerStrip = imp.getHeight(); // use all rows

            //rowsPerStrip = 1;
            long[] rowsPerStripArray = new long[]{ rowsPerStrip };

            for (int z = 0; z < imp.getNSlices(); z++)
            {
                // save, configuring myself how many strips to use for compression
                IFD ifd = new IFD();
                ifd.put( IFD.ROWS_PER_STRIP, rowsPerStripArray );
                if (imp.getBytesPerPixel() == 2)
                {
                    byte[] bytes = ShortToByteBigEndian( ( short[] ) imp.getStack().getProcessor( z + 1 ).getPixels() );
                    tiffWriter.saveBytes(z, bytes, ifd);
                }
                else if (imp.getBytesPerPixel() == 1)
                {
                    byte[] bytes = ( byte[] ) ( imp.getStack().getProcessor( z + 1 ).getPixels() );
                    tiffWriter.saveBytes(z, bytes, ifd);
                }
            }

            writer.close();

Note: For each z-plane ShortToByteBigEndian takes around 20 ms for above use-case and tiffWriter.saveBytes takes around 200 ms (for LZW plane-wise compression).

3 Likes

I am not aware of any other software that directly opens the .tif.zip files created by ImageJ, so a script would be needed.

1 Like

The Tiff writing performance is something we keep coming back to and trying to improve. Below is a quick profile of an an export (100 frames, 2560 x 2160). The majority of the time appears to be seeking in the buffer as opposed to writing. I remember I had a more detailed investigation into this at some point, I will try to find those old notes.

It would probably be worth profiling using tiling options as well to compare. Tiled writing isn’t yet available through the exporter in the ImageJ plugin but that is something which could be added.

5 Likes

Thanks for looking into this. There is also an issue, I found: https://github.com/ome/bioformats/issues/3480
Do you think it is realistic to improve this? It would be great!!

Naive idea that comes to my mind is: Maybe you are not writing each IFD in one go but piece by piece and that’s slow? Disclaimer: I have not even looked at the code yet…

The IFD itself is written in in one go, that I don’t believe is itself the main problem. Having done some further testing on this today and the performance bottlenecks vary quite a bit depending on the type of data and how its being written, very large planes behave differently that a large number of small planes, writing everything sequentially should be much faster etc.

There are likely still improvements that can be made to the TiffWriter and TiffSaver and we would be happy to make any improvements identified. The issue is that the Tiff writing code covers a huge number of different scenarios and is so fundamental that even simple looking changes are going to take a lot of time and effort to properly test and verify. Realistically in the short term we don’t have a significant amount of time or resources to dedicate to profiling. If you have specific ideas for improvements we can help investigate those, the GitHub Issue you linked would be a good place for keeping that discussion and investigation ongoing or we would be happy to review and test any PR’s.

If you haven’t yet taken a look at the new tools and workflow from Glencoe (Converting Whole Slide Images to OME-TIFF: A New Workflow) then I would also recommend it for improved performance of conversions.

1 Like