Slow TIFF writing performance on network drives from Windows

We observe poor writing performance when saving relatively large image stacks to smb-mounted network drives on Windows (File > Save As > Tiff…). I created a small test script that loads an image stack and saves it in two different locations, local and remote:

/*
 * Benchmark script for saving TIFF images
 */

#@ LogService log

import ij.IJ
import groovy.time.TimeCategory

filepath = "X:/path/to/large-image-stack.tif"
savedir_remote = "X:/output/benchmark"
savedir_local = "C:/temp/benchmark"

timer = { description, closure ->
	timeStart = new Date()
	closure.call()
	timeStop = new Date()
	duration = TimeCategory.minus(timeStop, timeStart)
	log.info("Time spent $description: " + duration)
}

timer ("opening", { imp = IJ.openImage(filepath) })

log.info("Opened ${imp.getTitle()}")
imp.show()

timer ("writing local", { IJ.saveAs(imp, "Tiff", savedir_local + "/" + imp.getTitle()) } )
timer ("writing remote", { IJ.saveAs(imp, "Tiff", savedir_remote + "/" + imp.getTitle()) } )
timer ("writing local", { IJ.saveAs(imp, "Tiff", savedir_local + "/v2" + imp.getTitle()) } )
timer ("writing remote", { IJ.saveAs(imp, "Tiff", savedir_remote + "/v2" + imp.getTitle()) } )

imp.close()

With an up-to-date Fiji installation (ij-1.52p.jar), we get:

[INFO] Time spent opening: 5.696 seconds
[INFO] Opened CombinedStacks-Raw_DeconvMeas_DeconvTheo.tif
[INFO] Time spent writing local: 1.960 seconds
[INFO] Time spent writing remote: 19.796 seconds
[INFO] Time spent writing local: 1.414 seconds
[INFO] Time spent writing remote: 20.003 seconds

I noticed there’s a count value hard-coded to 8192 in ImageJ1’s ImageWriter class. When I change this number to e.g. 65536, like such:

- int count = 8192;
+ int count = 65536;

… the performance gets much better:

[INFO] Time spent opening: 5.752 seconds
[INFO] Opened CombinedStacks-Raw_DeconvMeas_DeconvTheo.tif
[INFO] Time spent writing local: 1.090 seconds
[INFO] Time spent writing remote: 4.579 seconds
[INFO] Time spent writing local: 0.786 seconds
[INFO] Time spent writing remote: 4.164 seconds

@Wayne, is this something that could be changed in ImageWriter.java? Maybe it can even be made configurable via an option?

EDIT:
The chunk size of 64k is still rather conservative. Increasing to 8M (count = 8388608) gives another improvement by factor 2 on our network, and a slight speedup for saving locally. I didn’t test exceedingly large numbers yet…

2 Likes

It can be changed. What are the image dimensions? The count should probably be proportional to the image size.

We’ve faced similar issues with reading and writing from and to network drives in the past. Actually a really great solution was to write to temporary (memory or disk cache) and then use a system move command to transfer the full block to the network share. This achieves by far the best performance.
Just my two cents.

The stack I used for the above tests was an 8-bit 2598 x 986 x 325 (xyz) stack.
As far as I understand from reading the ImageWriter code, a new call to OutputStream.write is performed for each count bytes, but at least for every slice of the image. So I assume setting count to the number of bytes per plane (2561628 in my case) would avoid unnecessary chunk creation.

Things might even be sped up by writing several planes sequentially in the same call to OutputStream.write, but maybe there’s anyhow room for improvement by switching to BufferedOutputStream instead of the non-buffered variant, or by using Java NIO altogether?

@Wayne I’ll be happy to test any changes you’d like to include in the ImageJ1 source code.


Yes, we’ve been educating our users in the facility for years to save locally and then transfer over network. But first, some people will still and always try to save directly to network drives, and second – and more importantly – we have a very good network storage since a few years, and a 10Gbit connection to our acquisition and analysis machines, and writing of tif files works very fast with other software applications, so why not take advantage of #open-source and try to improve things, instead of continuing to recommend workarounds??

1 Like

In the latest ImageJ daily build (1.52r30), with 8-bit stacks, the count is set to the number of bytes per plane.

3 Likes

@imagejan This is exactly my solution, I’m sorry if this was not becoming clear. I did not mean to change the behavior of your users but to change the behavior of the software. When writing to a file like tiff, there are many many individual small write operations involved. This can never achieve the same performance on a network drive than on a local or RAM drive. Therefore my recommendation is to add an option to the image writer to transparently write to a temporary file and later move that file to the final storage. We’ve been doing this with CellProfiler and it works great, users don’t notice any difference and the writing is almost twice as fast (and somethimes much faster, depending on network load).

Just my two cents.

1 Like