PixelData / Pyramid Creation does not Multithread correctly

Hey,

we are testing an OMERO instance for larger scale deployment down the line at the moment. Some of our images take a very long time to process by the PixelData-Process. To speed up the process, I increased the omero.pixeldata.threads option to 6 threads. From the PixelData-0.log I can see, that it indeed starts to use all 6 threads to generate PixelData. However, no new threads are started until all the original threads complete operation. This of course is a problem, if the batch of 6 images take very different times to process. Is this behaviour normal or did I configure something wrong?

I’ve attached the admin diagnostics, omero.config and a tail of PixelData-0.log from the last startup.
I should add, that we are running our instance inside a docker environment if that can matter.

Thanks for your help
//
Julian
admin_diag.txt (5.0 KB) omero-conf.txt (905 Bytes)PixelData-log.txt (17.8 KB)

Hi @julianHn,

Unfortunately this is an issue that appeared just before the pandemic and I have to admit that I completely lost track of it. Very sorry about that.

Can you take a look at my post OMERO pyramid generation time - #7 by joshmoore and see if setting omero.threads.min_threads helps?

All the best,
~Josh

Hi @joshmoore,

thanks for the quick reply and no worries, Omero works great so far other than this slight hickup :slight_smile:
I have already read the linked thread and had the min.threads bumped in my config. My problem is a bit different I think:
The PixelData Process starts to multithread as expected (all 6 threads are working). However the images in one “batch” of 6 threads in our case might take very different times to process (30s to 45 mins per image). This causes all threads to wait for the slowest thread until a new “batch” of 6 threads is started.
This of course means that the waiting time for the “easy” images is unnecessarily long and the system is unproductive for most of the time if there is a batch of 5 “easy” and 1 “difficult” image.

Best
Julian

1 Like

Thanks, Julian. You had shown as much in your omero-conf.txt (:+1:) but I failed to check after getting side-tracked with my forgotten thread. That’s then sounding very much as if this is a limitation of the current queue: each batch that gets allocated must be completed before another is taken.

The only immediate option I can think of is to use bioformats2raw and raw2ometiff to convert your input data outside of OMERO. If this is an option you’d like to pursue, let us know.

~Josh

1 Like

Thanks for the answer Josh.
We’ll probably go ahead and put the burden of pyramiding onto the users as suggested by you. Just wanted to be sure that I did not misconfigure something before deciding on a workflow.
//
Julian

Understood. I filed All pixel data threads hang per batch until all complete · Issue #122 · ome/omero-server · GitHub.

~Josh

Thanks for the heads-up. I’ll track the issue over there as well, in case you need any debug info etc.

// Julian