Slow TIFF writing performance on network drives from Windows

We observe poor writing performance when saving relatively large image stacks to smb-mounted network drives on Windows (File > Save As > Tiff…). I created a small test script that loads an image stack and saves it in two different locations, local and remote:

/*
 * Benchmark script for saving TIFF images
 */

#@ LogService log

import ij.IJ
import groovy.time.TimeCategory

filepath = "X:/path/to/large-image-stack.tif"
savedir_remote = "X:/output/benchmark"
savedir_local = "C:/temp/benchmark"

timer = { description, closure ->
	timeStart = new Date()
	closure.call()
	timeStop = new Date()
	duration = TimeCategory.minus(timeStop, timeStart)
	log.info("Time spent $description: " + duration)
}

timer ("opening", { imp = IJ.openImage(filepath) })

log.info("Opened ${imp.getTitle()}")
imp.show()

timer ("writing local", { IJ.saveAs(imp, "Tiff", savedir_local + "/" + imp.getTitle()) } )
timer ("writing remote", { IJ.saveAs(imp, "Tiff", savedir_remote + "/" + imp.getTitle()) } )
timer ("writing local", { IJ.saveAs(imp, "Tiff", savedir_local + "/v2" + imp.getTitle()) } )
timer ("writing remote", { IJ.saveAs(imp, "Tiff", savedir_remote + "/v2" + imp.getTitle()) } )

imp.close()

With an up-to-date Fiji installation (ij-1.52p.jar), we get:

[INFO] Time spent opening: 5.696 seconds
[INFO] Opened CombinedStacks-Raw_DeconvMeas_DeconvTheo.tif
[INFO] Time spent writing local: 1.960 seconds
[INFO] Time spent writing remote: 19.796 seconds
[INFO] Time spent writing local: 1.414 seconds
[INFO] Time spent writing remote: 20.003 seconds

I noticed there’s a count value hard-coded to 8192 in ImageJ1’s ImageWriter class. When I change this number to e.g. 65536, like such:

- int count = 8192;
+ int count = 65536;

… the performance gets much better:

[INFO] Time spent opening: 5.752 seconds
[INFO] Opened CombinedStacks-Raw_DeconvMeas_DeconvTheo.tif
[INFO] Time spent writing local: 1.090 seconds
[INFO] Time spent writing remote: 4.579 seconds
[INFO] Time spent writing local: 0.786 seconds
[INFO] Time spent writing remote: 4.164 seconds

@Wayne, is this something that could be changed in ImageWriter.java? Maybe it can even be made configurable via an option?

EDIT:
The chunk size of 64k is still rather conservative. Increasing to 8M (count = 8388608) gives another improvement by factor 2 on our network, and a slight speedup for saving locally. I didn’t test exceedingly large numbers yet…

2 Likes

It can be changed. What are the image dimensions? The count should probably be proportional to the image size.

We’ve faced similar issues with reading and writing from and to network drives in the past. Actually a really great solution was to write to temporary (memory or disk cache) and then use a system move command to transfer the full block to the network share. This achieves by far the best performance.
Just my two cents.

The stack I used for the above tests was an 8-bit 2598 x 986 x 325 (xyz) stack.
As far as I understand from reading the ImageWriter code, a new call to OutputStream.write is performed for each count bytes, but at least for every slice of the image. So I assume setting count to the number of bytes per plane (2561628 in my case) would avoid unnecessary chunk creation.

Things might even be sped up by writing several planes sequentially in the same call to OutputStream.write, but maybe there’s anyhow room for improvement by switching to BufferedOutputStream instead of the non-buffered variant, or by using Java NIO altogether?

@Wayne I’ll be happy to test any changes you’d like to include in the ImageJ1 source code.


Yes, we’ve been educating our users in the facility for years to save locally and then transfer over network. But first, some people will still and always try to save directly to network drives, and second – and more importantly – we have a very good network storage since a few years, and a 10Gbit connection to our acquisition and analysis machines, and writing of tif files works very fast with other software applications, so why not take advantage of #open-source and try to improve things, instead of continuing to recommend workarounds??

1 Like

In the latest ImageJ daily build (1.52r30), with 8-bit stacks, the count is set to the number of bytes per plane.

4 Likes

@imagejan This is exactly my solution, I’m sorry if this was not becoming clear. I did not mean to change the behavior of your users but to change the behavior of the software. When writing to a file like tiff, there are many many individual small write operations involved. This can never achieve the same performance on a network drive than on a local or RAM drive. Therefore my recommendation is to add an option to the image writer to transparently write to a temporary file and later move that file to the final storage. We’ve been doing this with CellProfiler and it works great, users don’t notice any difference and the writing is almost twice as fast (and somethimes much faster, depending on network load).

Just my two cents.

1 Like

For those interested, the performance improvement is included in the ImageJ 1.52r release (although the release notes don’t mention it), and the relevant commit is here (although the commit message is misleading):

Thanks again, @Wayne, for making these changes. In our network infrastructure, this has a huge impact, with writing performance increasing by a factor of 10 for large images.


Here are the results of a real-world test (i.e. on a normal workstation in our network, with potentially other processes running on the same machine and on the network), comparing ij-1.52p.jar (the one shipping with Fiji up to now) with ij-1.52r.jar (released last week) and saving 10-slice stacks of different plane sizes:

The (samba-mounted) network drive is shown in the panels on the right-hand side.

1 Like

Hi all,

thanks for figuring this out and releasing a fix!

I noticed that we have the same problem in our institute and would benefit a lot from this update.

@ctrueden do you have a schedule for when ij-1.52r.jar or higher is going to be added in Fiji? Currently Fiji ships with ij-1.52p.jar, i.e. before the fix.

Thanks and cheers,
Kai

1 Like

As far as I know, Curtis is currently working on fixing a number of compatibility issues between various components in the SciJava/ImgLib2/ImageJ/Fiji software stack, and a release of pom-scijava-29.0.0 (presumably followed by a big bunch of uploads to the update sites) is imminent.

Specific to ImageJ 1.x, there is this open pull request in ij1-patcher (the component that allows ImageJ 1.x to be run e.g. in headless mode) that shows some of the current backwards-breaking changes that need to be accounted for:


@CellKai for the time being, it will certainly help if you put the newest ij.jar version 1.53a into your local Fiji installation and report any issues you encounter. (I’d expect issues to occur mostly when trying to run from the command line plugins that make use of the GenericDialog class, not in “normal” UI usage…)

1 Like

Hi Jan, thanks for the details!

In this case I think we will wait for the updates :slight_smile:
Thanks for your help!