OMERO pyramid generation time

Dear @OMETeam,

One of our users recently contacted us saying that she can’t see her images on our OMERO server since the pyramid are still not generated. Those images were uploaded the day before and are OME-TIFF.

I took the liberty of checking other of her images and there are some which were uploaded more than 6 months ago and for which the pyramids are still missing. I could understand about one day old images not being processed yet but 6 months old should have been processed I think.

Our current server is running OMERO 5.4.10 which has an older Bio-Formats version. We’ll soon update to 5.5.1 (5.6 will unfortunately require more work for which we don’t have time at the moment). Do you think that the new OMERO version with its new Bio-Formats should fix this ?

I checked with the person in charge of the server and this is his message :

I would say that the Pyramid processing job in general is running fine. I can “see” it working as progress of pyramid data generation is shown in the PixelData log file. Since I restarted OMERO processes yesterday for a different reason, the pyramid processing job had been re-initialized as well.

My current speculation is, that the pyramid processing job is “simply” too slow, meaning seriously lagging behind.

We had a similar issue a couple of years ago where some pyramid data got corrupted messing up new ones, which is not the case today.

Do you know any way to make this smoother ?

Thanks a lot !

1 Like

Hi @lguerard,

If any newer pyramids are being generated at all, then I’d agree with you that something went wronger with the older ones. Can you get the ID for one of those and possibly for one of the newer ones? It would be good to know both if there were any errors in the log files for those images and whether or not a _pyramid file was generated for them. If yes, then it can be deleted and the generation re-triggered. If that doesn’t show an error in the log, then I may need a stack trace from the server to figure out what’s going wrong.

Hard to know. Where are the OME-TIFFs coming from? How large are they? And do you know if they are intended to have pyramids already generated internally?

What platform is your server deployed on? It’d be interesting to hear if there’s anything we can do to help you migrate directly to 5.6 so as to not invest time in an out of date version, though that should likely be another thread.

I’d need to know more about what “too slow” means. How many imports? How large? How much memory the process has? How many threads it’s configured to use? etc. Though see also Good value for omero.pixeldata.threads as well.

All the best,
~Josh

Hi @joshmoore,

Thanks a lot for the fast answer ! :slight_smile:

I’m waiting for the person in charge of the server in order to be able to answer to most of your question and will come back to you for that.

The data seems to come from a Ni-E microscope and are somehow in OME-TIFF, I’ll contact the user to know if she did some conversion/processing.
As for the data itself, I opened them in Fiji, they don’t seem to have pyramids and are not super heavy data, less than 700MB but they’re 14kx15x pixels which I guess triggers generation of pyramids in OMERO ?

Thanks again !

Best,
Laurent

Hi @joshmoore,

“person in charge” for the server speaking :rofl:

The server is running on CentOS Linux release 7.5.1804.
Generally speaking, I obviously fully agree. Unfortunately, “just” upgrading OMERO itself to 5.6 is not the full story!. The person being in charge for certain customization & adaptation of features is currently on paternity leave. That’s the main reason for us in hesitating to do so immediately :wink:

How many imports + How large @lguerard already tried to answer.
Regarding the more system related parameters, those are as follows:

JVM Settings:
============
blitz=-Xmx96000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '40', 'strategy': 'percent'})
indexer=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})
pixeldata=-Xmx72000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '30', 'strategy': 'percent'})
repository=-Xmx24000m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions # Settings({'max_system_memory': '240000', 'system_memory': '240000', 'percent': '10', 'strategy': 'percent'})

The server has 256 GB memory in total.

omero.pixeldata.cron=* * * * * ?
omero.pixeldata.repetitions=2
omero.pixeldata.threads=12

If I got your point right: having more than 4 threads will not really contribute (won’t increase our parallelism)? :smirk:
So we could safely decrease this value again …

Thanks a lot for your support, best,
-Rainer

Definitely, though newer OME-TIFFs could be created directly with the pyramids and then no generation would be needed by OMERO.

Hi Rainer :wink:

Understood (and congratulations!) That’s life. As long as everyone on board is aware of the security patch warning from Release of OMERO 5.6.0 with Python 3

Inviable! :smile: So that’s unlikely the issue. Does anyone have an estimate for how long a successful pyramid generation is currently taking?

Correct, not until the bug is fixed.

Gladly.
~Josh

Something like that (as an example)?

2020-01-30 17:08:30,825 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 1/4488 (0%).
2020-01-30 17:09:06,776 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 449/4488 (9%).
2020-01-30 17:09:42,109 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 897/4488 (19%).
2020-01-30 17:10:16,919 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 1345/4488 (29%).
2020-01-30 17:10:53,016 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 1793/4488 (39%).
2020-01-30 17:11:28,512 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 2241/4488 (49%).
2020-01-30 17:12:05,359 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 2689/4488 (59%).
2020-01-30 17:12:41,183 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 3137/4488 (69%).
2020-01-30 17:13:17,758 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 3585/4488 (79%).
2020-01-30 17:13:53,195 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 4033/4488 (89%).
2020-01-30 17:14:28,736 INFO  [                ome.io.nio.PixelsService] (2-thread-5) Pyramid creation for Pixels:1222379 4481/4488 (99%).
2020-01-30 17:14:29,374 INFO  [                ome.io.nio.PixelsService] (2-thread-5) SUCCESS -- Pyramid created for pixels id:1222379
2020-01-30 17:14:34,119 INFO  [ ome.services.pixeldata.PixelDataHandler] (2-thread-5) HANDLED EventLog:159448366(entityId=1222379) [363448 ms.]

The corresponding image is 16G in size:

du -sh NNNNNNN_4365/2020-01/27/11-26-54.136/NSG_TUMOR19\ -01.czi
16G	NNNNNNNN_4365/2020-01/27/11-26-54.136/NSG_TUMOR19 -01.czi

Does this make any sense?

Thanks,
-Rainer

Awesome, thanks. So ~6min for CZI files. If you also find a result for one of the OME-TIFFs, I’d be interested to see that as well.

An update on https://github.com/ome/omero-model/issues/46 : there might just need to be another property bumped (omero.threads.min_threads) in order to increase parallelism. I’ll keep you posted.

~Josh

Hi @joshmoore,

So, is this more :+1: or :-1: according to your expertise?

Regarding those *.ome.tif I cannot find a single entry about those in the PixelData log file?!?!???
@lguerard downloaded several of those that still showed this “not-yet-ready” clock like symbol in OMERO and could not see any pyramid data existing. Also, when using the “old” OMERO viewer it clearly complains about “A Pyramid of zoom level is currently being calculated for this image. Please try viewing again later.”

On the other hand, I am not able to detect any information about pyramid data generation in the log files for those files although they are generated (very, very, slowly …)!

Puzzled and confused …

-Rainer

It’s not a size that’s going to scale to a large number of CZI files, but seems tolerable.

I assume not in the rotated files either? If so, that’s quite odd. Are there any WARNINGs or ERRORs which might be related?

Ditto.
~Josh

I agree, 6 minutes should be ok.

Nothing in the logs about any errors or warning unfortunately :confused:

Is that working ? We should be able to update this settings if it could speed up the process.

By the way, is there a simple command to see all threads and what they’re doing in OMERO ? I couldn’t find it when googling…

Thanks a lot !

Edit : Would there be a way to dedicate a thread to pyramid generation ? It looks like there hasn’t been any new pyramid generated since the 31st of January now while new data has been imported, every hour we have this message in the PixelData.log file…

2020-02-14 12:13:29,814 INFO  [                      ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=144, min=0.456405, max=3.2189968221244E7, mean=5128103.351815632, stddev=7971386.290137979, median=1295336.263915, p75=4824353.15372525, p95=2.0897378470465314E7, p98=3.2189968221244E7, p99=3.2189968221244E7, p999=3.2189968221244E7, mean_rate=9.77995762474444E-5, m1=2.964393875E-314, m5=1.4821969375E-313, m15=4.44659081257E-313, rate_unit=events/second, duration_unit=milliseconds

Yes, in my testing, setting both properties increased the parallelism.

This is provided directly by Java. If you have a JDK installed, then use jstack. If not, send the QUIT signal to the Java process to have the stack printed to var/log/master.out.

The entire PixelData process is dedicated to pyramid generation. Something seems to be quite wrong in detecting pyramids in your case.

~Josh

Sorry I’m kinda new to omero admin thing, so I might be asking stupid question, let me know if that’s the case !

If I understood correctly, each image where pyramids need to be generated is put in a queue and picked up by the PixelData process, is that correct ? If so, is there a way to check that queue ?

2020-02-14 12:13:29,814 INFO [ ome.system.metrics] (r-thread-1) type=TIMER, name=ome.services.pixeldata.PixelDataThread.batch, count=144, min=0.456405, max=3.2189968221244E7, mean=5128103.351815632, stddev=7971386.290137979, median=1295336.263915, p75=4824353.15372525, p95=2.0897378470465314E7, p98=3.2189968221244E7, p99=3.2189968221244E7, p999=3.2189968221244E7, mean_rate=9.77995762474444E-5, m1=2.964393875E-314, m5=1.4821969375E-313, m15=4.44659081257E-313, rate_unit=events/second, duration_unit=milliseconds

Is there a reason why we have a similar line every hour in the PixelData.log file ? And what does the r mean in the thread number ?

Thank you very much !

1 Like

Hi Iguerard,

This is a very good question actually.

I am suffering from this as well. Sometimes, it may crashes out of “no reason?” like this.

2020-02-13 15:14:48,026 ERROR [  ome.services.pixeldata.PixelDataThread] (1-thread-5) ExceptionException!
ome.conditions.SessionTimeoutException: Session (started=2020-02-13 14:50:01.123, hits=17, last access=2020-02-13 14:50:31.168) exceeded timeToIdle (600000) by 856852 ms
	at ome.services.sessions.state.SessionCache.getDataNullOrThrowOnTimeout(SessionCache.java:470) ~[omero-server.jar:5.5.5]
	at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:368) ~[omero-server.jar:5.5.5]
	at ome.services.sessions.state.SessionCache.getSessionContext(SessionCache.java:353) ~[omero-server.jar:5.5.5]
	at ome.security.basic.CurrentDetails.login(CurrentDetails.java:164) ~[omero-server.jar:5.5.5]
	at ome.services.util.Executor$Impl.execute(Executor.java:439) ~[omero-server.jar:5.5.5]
	at ome.services.util.Executor$Impl.execute(Executor.java:392) ~[omero-server.jar:5.5.5]
	at ome.services.pixeldata.PixelDataThread.go(PixelDataThread.java:302) ~[omero-server.jar:5.5.5]
	at ome.services.pixeldata.PixelDataThread.access$000(PixelDataThread.java:51) ~[omero-server.jar:5.5.5]
	at ome.services.pixeldata.PixelDataThread$1.call(PixelDataThread.java:250) ~[omero-server.jar:5.5.5]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_242]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_242]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_242]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_242]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_242]
2020-02-13 15:14:48,027 ERROR [  ome.services.pixeldata.PixelDataThread] (1-thread-5) ExceptionException!

@joshmoore Actually, my another question is : if it is possible for us to run this Pixeldata conversion process manually and in parallel. In that case we know what we are expecting.

Best regards,
John

Dear @joshmoore, @lguerard, @John_Xu, all,

first of all thanks a lot for all your valuable input on this apparently rather complex topic.

However, I am still feeling puzzled about what’s actually going on (or in this case not …).

  • The observed issue (missing pyramid data of >6 months old datasets) is still the same. I just re-checked those again: still no pyramid data. How could we repair those?
  • Is there any way to detect the scope of this problem on our instance? Only few user affected, maybe substantial no. of users? So far we rely on the feedback of our user reporting us back somehow broken images. But if they don’t tell us, we do not know!
  • Any tool/command/SQL query/whatever existing that might allow to search for those cases? Ideally, generating a list of images with missing conversion data?
  • And, finally, to also support the idea that one could maybe manually start another process helping to catch-up? Or even better starting to work on a list (see above)?

Any ideas/suggestions/… would be really appreciated.

Best regards,
-Rainer

P.S.: I am fully aware that we do not run the very latest OMERO version yet, which might improve a lot. Nevertheless, I doubt that it will start working on missing pyramid data from old datasets, will it? So, an upgrade might improve future handling of new data but not necessarily “repair” existing data?

1 Like

Definitely interested to hear if there is any server-side log activity in master.* or Blitz-0.log or PixelData-0.log when you try to access an image with missing pyramids.

The pixels ID for image ID 12345 is revealed by,
omero hql "SELECT id FROM Pixels WHERE image.id = 12345"
It could be interesting to see if there is a file somewhere within your server’s binary repository’s Pixels directory with a *_pyramid file for that ID, even to try deleting that file and trying again.

When an image needs pyramids creating then the event log gets a PIXELDATA action logged for its pixels ID, e.g.,
omero hql "SELECT id FROM EventLog WHERE action = 'PIXELDATA' and entitytype = 'ome.model.core.Pixels' AND entityid = 2468". From the database (e.g., with psql) one can see the latest event log ID worked on:
SELECT value FROM configuration WHERE name = 'pixelDataEventLogLoader.v1.current_id';

Also note OMERO.cli commands like omero admin fixpyramids and the discussion at https://docs.openmicroscopy.org/omero/5.5/sysadmins/server-upgrade.html#corrupted-pyramids and https://docs.openmicroscopy.org/omero/5.5/sysadmins/limitations.html#large-images-with-floating-point-pixel-data. Also as at
https://docs.openmicroscopy.org/omero/5.5/sysadmins/server-performance.html#memory-configuration the pixel data thread may simply need more memory to be able to cope with users’ images.

Also for new images note Glencoe Software’s work at https://www.glencoesoftware.com/blog/2019/12/09/converting-whole-slide-images-to-OME-TIFF.html - converting problem images to pyramidal OME-TIFF before importing them may work around issues with server-side pyramids.

I hope that among the above there are clues that get you usefully further along.

Hi Mark,

I tried that with one of the old broken files and this request returned nothing so something strange is going on there…

I might actually be doing something wrong so I’ll wait for @rpoehlmann to know if he has more chance than me !

Is there any way in Fiji to convert to pyramidal OME-TIFF ? We looked for it but couldn’t find anything to specify that it should also generate the pyramid.

Thank you for your help !

Maybe also try with the --all flag after the hql if the image is not in the user’s current group. Could also try SELECT name FROM Image WHERE id = 12345 just to check it even finds the image! If it does but not the pixels then, yes, very odd.

I know bfconvert can write pyramidal OME-TIFF but I don’t have a clue about Fiji but with luck someone else will soon answer on that.

Great, the “–all” did the trick!!!
The image in question having no pyramid data:

$BINOMERO hql  -q "select id from Pixels where image.id=1232301"
 # 
---
(0 rows)

2nd try with “–all”:

$BINOMERO hql --all -q "select id from Pixels where image.id=1232301"
 # | Col1    
---+---------
 0 | 1232051 
(1 row)

However, for the “name” instead of id it does not really returns anything useful:

$BINOMERO hql --all -q "select name from Pixels where image.id=1232301"
Bad query: Illegal query:select name from Pixels where image.id=1232301
No data type for node: org.hibernate.hql.ast.tree.IdentNode 
 \-[IDENT] IdentNode: 'name' {originalText=name}

But regarding the name that seems to be similar for other images where id do see pyramid data …?!?
:+1: Thanks a lot!

Unfortunately, the 2nd step aiming to find some incomplete pyramid data, failed:

(omeroweb)[omeronas@omero02 Pixels]$ $BINOMERO hql --all -q "select id from Pixels where image.id=1232301"
 # | Col1    
---+---------
 0 | 1232051 
(1 row)
(omeroweb)[omeronas@omero02 Pixels]$ pwd
/export/omero/OMERO/Pixels
(omeroweb)[omeronas@omero02 Pixels]$ find . -type f -name "*1232051*"
(omeroweb)[omeronas@omero02 Pixels]$ find . -type f -name "1232051_pyramid"

Okay, next question is if there are requests to generate the pyramid data, like when you try to open the full viewer for the image. Then does,

$BINOMERO hql "SELECT DISTINCT entityId, event.time FROM EventLog WHERE action = 'PIXELDATA' AND entityType = 'ome.model.core.Pixels' ORDER BY event.time DESC"

show something near the top for 1232051 for when you opened the viewer?

Also, what are you getting on the server machine from,

$BINOMERO admin diagnostics | grep PixelData