Bio-Formats ome-tiff write time issue / bug?

fiji
bio-formats
icy
imagej
ome-tiff
apeer
#1

ome tiff export is super slow:
Example with an (empty) simple empty hyperstack, 5 channels 16 bits, 1000x1000 , 300 time points (about 3GB in tif).

Tiff writing with Fiji: 10 sec
Tiff writing from Fiji Bio-Formats exporter or Icy (save as Tiff, using Bio-Formats by default): more than 2 minutes (I was not patient enough to let it finish).

Do you have any idea how to fix that in Bio-Formats (super useful otherwise, thanks for developing and maintaining btw)? could all software propose an alternate saving otherwise (in icy it is the only export format in tif available for now)?
Kind regards,
Perrine

1 Like
Make tags obligatory, and introduce tag hierarchy?
#2

I also noticed that that writing OME-TIFF recently got super slow, so that I was running into a timeout. I do not remember the exact numbers but it was several minutes for MB-sized image.
I noticed that when writing images from inside a Fiji Docker container while testing a new module for the APEER platform.
Any hints what could be the reason would be really helpful.

0 Likes
#3

These times do appear to be longer than expected so thank you for raising this as a concern. The Bio-Formats team has been carrying out some tests on this today and we have been able to reproduce these sort of times using similar dimensions. My initial view is that this is a performance issue rather than a bug and it is not specific to the FIJI plugin. We have also begun to look into profiling the tiff writing process in order to better understand what the bottlenecks are and how they can be improved going forward. This is something that we will have to further investigate going forward.

David Gault

3 Likes
#4

Hi David,

thanks for the update. Let me know if I can help with more info etc.

Sebi

0 Likes
#5

Hi David,

are there any updates on this because I emcountered that issue today again. Basically I did run into a timeout when using BioFormats Export to write a file.

If it is supposed to be fixed, then which version of BioFormats should I use?

Sebi

0 Likes
#6

We are still trying to profile this better and we are hoping to prioritise work to make improvements to this shortly. There is some ongoing discussion on https://trello.com/c/OimiHAQY/39-ome-common-profile-tiff-writing which you can follow. This will be further updated with the latest findings over the next few weeks.

0 Likes
#7

Thx for the update.

Sebi

0 Likes
#8

Thanks for the link.

0 Likes
#9

I too have been using BioFormats from my own Java application to read/write OME-TIF files and the performance can be very slow. Whilst on my Mac with internal SSD the performance is acceptable, the moment I switch to using a slower target (network disk for example) the performance really plummets.

I have some more tests to try, but I suspect that during the write process calls to ImageWritter::saveBytes might be opening/writing/closing the file each time which perhaps may defeat some file system caching?

One thing I will look to test is whether writing to a temporary file on the local filesystem and then block moving this into the final destination improves matters for a network drive.

I’d sure be interested to hear if there has been progress with this issue.

– Michael Ellis

0 Likes
#10

Dear all,

sorry for the long delay in answering and thanks everyone for all the hindsights. The link given by @dgault in Bio-Formats ome-tiff write time issue / bug? contains references to some writing performance improvements change that have been recently proposed by ourselves and contributors.

Although these changes have given us substantial writing improvements in our recent testing use cases, largely using the Bio-Formats command line tools, it does not address the speed of writing OME-TIFF using the ImageJ plugin in the conditions described above.

We believe the issue discussed in this thread is specific to the usage of the writing API in the ImageJ plugin. As we have now completed a large portion of our work on pyramidal OME-TIFF (watch https://blog.openmicroscopy.org/ for more information), tackling this performance issue is now part of our outstanding and immediate priorities.

For developers, you can track the latest progress on the dedicated reference card - https://trello.com/c/Qk6NBnPs/92-imagej-ome-tiff-writing-performance. Once we have a fix ready to test by the community, we would propose to enable development milestones of Bio-Formats 6 via the development Bio-Formats update site.

Best,
SĂ©bastien, David, Melissa & Josh

0 Likes
#11

Are there any real news because from checking the progress an Trello I do not really know, what is going on.

We use the BioFormats OME-TIFF writer on APEER (www.apeer.com) inside docker containers and run sometime into the issue that or Kubernetes system (where the workflow engine runs) sometimes “kills” a running container because it thinks “it is dead”.

But if fact writing the OME-TIFF to the outputs is still running (slowly …). Do you have a rough idea, when this issue might be solved so that we can update our modules?

0 Likes
#12

Hi @sebi06,

we are currently on the process to freeze our specification and API frozen for Bio-Formats 6. Current expectation is to have Bio-Formats 6.0.0 released in February including some improvements for writing OME-TIFFs.

Following up on my Bio-Formats ome-tiff write time issue / bug?, we have encountered issues which prevented us from pushing the Bio-Formats 6 development milestones to the Bio-Formats development update site. Addressing these blockers if currently high on our priority list and we hope to have the first release candidate of Bio-Formats 6 available via this update site.

If you are downloading Bio-Formats independently of Fiji within a Docker container, you could also try to the latest development milestone.

2 Likes
#13

Thanks a lot for the update.

Sebi

0 Likes