OMERO.importer 5.4.10 throws Ice.ObjectNotExistException when importing hundreds of files

Hallo,

my users have problems importing hundreds or thousends of files every now and then (OMERO version: 5.4.10).
There is an Ice.ObjectNotExistException of ome.formats.OMEROMetadataStoreClient for some files and they are not imported. The lightweight importer UI freezes so there is not hint for the user if the import is going on or not. If they restart the importer and retry the import it works. But for thousends of files is hard to detect, witch files are not imported and it takes a long time to reimport all files…

I can’t figure out what the problem is by looking into the error logs. In Blitz.log there are some errors of omero.util.ServantHolder: servantsPerSession reached for 9f71b50f-0aef-4769-ac2f-c65e88650d79: 10000 . It is related?

I will upload the omeroimporter.log and the Blitz.log to qa and maybe you have time to look in it.

Thanks a lot
Susanne

This sounds like a single session is doing a lot of work, either simultaneously (unlikely) or is not cleaning up services when completed.
~Josh

It seems so. I’ve started the import of 3601 files again against a server without local changes and with the original importer client 5.4.10 ;-). The importer GUI is frozen at uploading 2315 files but the import still going on. The Blitz.log shows an increasing using of servants and around the timepoint at 2315 uploaded files are 90% of servants used by this session.

ServentsPerSession error again after uploading 2542 of 3062 files. Can it be due to our server configuration?

I wouldn’t think so. It’s possible to increase the value, but that would just fail later if there are services that are not being closed.

Can you double-check that no stateful services (ServiceFactoryPrx.create*()) are being created without being closed?

Cheers,
~Josh

P.S. Diving into specifics, at least for the logs you gave me:

/tmp/27696 $  grep "Added servant" Blitz-0.log | cut -f2 -d/ | sort > added.txt
/tmp/27696 $ grep Unregister Blitz-0.log | cut -f2 -d/ | sort > unregister.txt
/tmp/27696 $ comm -2 -3 added.txt unregister.txt | wc
    1209    1209  116180
/tmp/27696 $ comm -1 -3 added.txt unregister.txt | wc
    7668    7668  714008

i.e. for the window of time, there were 1200 services which were added but never removed and 7668 services which were removed but not added (in another log??).

Following your instructions:

/tmp/27696 $  grep "Added servant" Blitz-0.log | cut -f2 -d/ | sort > added.txt
/tmp/27696 $ grep Unregister Blitz-0.log | cut -f2 -d/ | sort > unregister.txt

i get :

[omero_dev@omero3 log]$ comm -2 -3 added.txt unregister.txt |wc
    581     581   55983
[omero_dev@omero3 log]$ comm -1 -3 added.txt unregister.txt |wc
      0       0       0

Because i installed a fresh original server there is no other log file and all contents of Blitz-0.log belong to the import test.

Perfect, thanks. So if all your sessions have been properly closed and all timeouts reached, etc. etc., then this says that whatever action you took for the test, left over 500 servants unclosed. Multiply that times the number of actions and you can estimate when you’ll hit the session limit.

~J.

Ok, i think i should restart the test. I must confess i have killed the importer after there was the servant error. After the servant error on the server the importer try to transfer all other files and throws because of the servant error an Ice.ObjectNotExistsException.

Hi Josh,

i’ve tested the import again.This time 2542 of 3601 files have been imported. The importer GUI freeze after 2481 files was imported and shows no action in the log files and no data transfer. I’ve been waiting more than 24h, then i force the importer to quit again. The server seems something to do all the time.
Checking the status of stateful services shows:
Whole Blitz-0.log:

$ comm -2 -3 added.txt unregister.txt |wc
   1133    1133  109043
$ comm -1 -3 added.txt unregister.txt |wc
      0       0       0

The Blitz-0.log until the servantsPerSession reached:

$ comm -2 -3 added_partE.txt unregister_partE.txt |wc
  10006   10006  930153
$ comm -1 -3 added_partE.txt unregister_partE.txt |wc
      0       0       0

and the Blitz-0.log until the imported stops communication with server:

$ comm -2 -3 added_part.txt unregister_part.txt |wc
   3488    3488  287031
$ comm -1 -3 added_part.txt unregister_part.txt |wc
      0       0       0

So the server seems to try to close all unused servants after the communication stops.

What can i do to ensure that the import of many files can be handled by the server?

Hi Susanne,
I could replicate the issue here with a plain Insight 5.5.2.
Still investigating what the cause of this is.
Regards,
Dominik

The server configuration setting that Josh had in mind is probably omero.throttling.servants_per_session. As you do seem to get a good way through you could try the workaround of increasing it rather beyond its default of 10000. We’ve not run into this issue on IDR where we use the OMERO.cli import command so that too might be a viable workaround.

Separately we’ll note this as something that needs looking at as OMERO.importer should be letting you do this.

I’ve increased the servants_per_session value and repeat the import process for my 3601 files. All files have been imported, but the Omero.importer was freezing at the scanning step and the log file shows after transfering the 3601th file again a Ice.ObjectNotExistException.

I know there are less problems with the cli import, but our users are not so familiar with it, sorry.

I am glad to read that you are able to import now at least, thank you for the update. We are readying to test fixes for both the servant consumption and the importer UI freeze; we can’t guarantee that these fixes will be in the very next OMERO release but I’m cautiously optimistic.

Thank you for your hints! If I should test something, give me a notice.

Regards,
Susanne