Connection to OMERO server lost while importing SlideBook data

We have been troubleshooting some problems with OMERO reading SlideBook files using our SlideBook6Reader.jar. We resolved an issue that prevented data from being properly imported due to the sequence of close(bool) calls, which now enables (smaller) data to import. However, when importing larger data (e.g. >500 MB) we are experiencing server timeout / connection lost messages, resulting in data import failure.

The link below includes the SlideBook6Reader.jar file, along with error logs and the java source for SlideBook6Reader.class:

I’m not sure how to best go about troubleshooting this issue. We can confirm that files that fail can be opened just fine using Fiji under both Windows and Linux, so assume it’s some interaction with the sequence of calls OMERO is making. We have a test server running so can add and review whatever log might shed some light on this.

Thanks,

Brian

3 Likes

Hi @bodenste. Welcome to image.sc!

And thanks for the logs. Looking at omeroinsight.log I see:

/tmp/slidebook $grep -E "(^2019|fully)" omeroinsight.log  | tail -n 20
2019-06-11 09:59:26,430 INFO  [      o.f.i.transfers.UploadFileTransfer] ( Thread-14) Transferring C:\Users\bode\Desktop\St Jude 488 - 2.sld...
2019-06-11 10:06:15,631 ERROR [              loci.formats.FormatHandler] ( Thread-14) close (all)
2019-06-11 10:06:15,631 ERROR [              loci.formats.FormatHandler] ( Thread-14) close (all)
2019-06-11 10:06:15,635 DEBUG [ o.o.shoola.env.data.DataServicesFactory] ( Thread-14) Connection ErrorIce.UnknownLocalException
2019-06-11 10:06:23,691 DEBUG [                   omero.gateway.Gateway] (entQueue-0) joining the session
2019-06-11 10:06:29,587 ERROR [                   omero.gateway.Gateway] (entQueue-0) Failed to join the session Glacier2.CannotCreateSessionException
    reason = "Server not fully initialized"
    reason = "Server not fully initialized"
2019-06-11 10:06:29,591 ERROR [                   omero.gateway.Gateway] (entQueue-0) Failed to close the session Glacier2.CannotCreateSessionException
    reason = "Server not fully initialized"
    reason = "Server not fully initialized"
2019-06-11 10:06:29,891 ERROR [                   omero.gateway.Gateway] (entQueue-0) Failed to create connector omero.gateway.exception.DSOutOfServiceException: Failed to create connector
    reason = "Server not fully initialized"
    reason = "Server not fully initialized"
2019-06-11 10:06:30,920 WARN  [                   omero.gateway.Gateway] ( Thread-14) Cannot re-create. Returning null connector
2019-06-11 10:06:30,921 ERROR [              loci.formats.FormatHandler] ( Thread-14) close (all)
2019-06-11 10:06:30,921 ERROR [              loci.formats.FormatHandler] ( Thread-14) close (all)
2019-06-11 10:06:30,922 WARN  [                   omero.gateway.Gateway] ( Thread-14) Cannot re-create. Returning null connector
2019-06-11 10:07:02,925 INFO  [o.o.shoola.env.rnd.PixelsServicesFactory] (nitializer) Heap memory usage: max 954728448
2019-06-11 10:07:02,978 INFO  [         o.o.shoola.env.init.TaskBarInit] (nitializer) Loaded L&F: com.sun.java.swing.plaf.windows.WindowsLookAndFeel

which for me says:

  • you started uploading
  • insight lost its connection
  • the server supported “Server not fully initialized”
  • you either restarted or managed to connect, etc.

The “not fully initialized” is worrisome because that only happens when the server is starting up. Since you were already logged in, my guess is that either (a) your server ran out of memory and crashed (usually when it runs out of member it hangs rather than crash) (b) something segfaulted and the Java process exited completely.

The log files from OMERO_DIST/var/log might have more information. In case (b), there’s likely also a core dump somewhere that may tell us more.

Hope that helps.
~Josh

1 Like

Hi Josh,

Thanks for the note and apologies for the delay, I was out of the office end of last week. I cleared the logs and did a clean start with one upload with the same failure. The server log files are here:

I think you are exactly correct that it appears the server crashes and reboots in the middle of the data import. Where would I find the core dump? I can upload that as well.

Thanks again!

Brian

Assuming core dumps are turned on for your operating system, you can often find them in the home directory of the user running OMERO. ~Josh

Hi Josh,

Thanks for the help. We were able to enable the core dumps and debug the crash - turns out we had a couple of spurious calls to assert(), which compile out in Windows but obviously terminate the process in Linux. We’ve compiled them out and it now appears to be working. We have a new issue but I will start a second thread for that one.

Thanks again!

Brian

2 Likes

Great, thank you for letting us know, also for starting a new thread for your new issue!