KeepAlive failed while importing 50GB Slide Book data file

:~/OMERO.server/bin$ ./omero import -s omero-dev.intelligent-imaging.com -p 4064 -u brian.bodensteiner --no-upgrade-check --skip all  "file.sld"

Trying to import a 50 gb file, eventually program dies. From the terminal log:

..................................
2019-08-08 11:27:54,203 1253045    [3-thread-1] DEBUG  formats.importer.transfers.TransferState - checksums: client=454b92e50b000000,server=454b92e50b000000

2019-08-08 11:27:54,203 1253045    [3-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /home/omero/OMERO.server-5.5.1-ice36-b122/bin/../OS_LLSM_190704_MC191_2s_20ms_laser2_11h_0.33uM_Noc_20min_DMEMF12.sld

2019-08-08 11:27:54,761 1253603    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END

2019-08-08 11:27:55,107 1253949    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 6026

2019-08-08 11:28:05,711 1264553    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:29:05,712 1324554    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:30:05,713 1384555    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:31:05,719 1444561    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:32:05,721 1504563    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:33:05,722 1564564    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:34:05,724 1624566    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:35:05,735 1684577    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:36:05,736 1744578    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:37:05,737 1804579    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:38:05,739 1864581    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:39:05,740 1924582    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:40:05,741 1984583    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:41:05,742 2044584    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:42:05,743 2104585    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:43:05,797 2164639    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive ping.

2019-08-08 11:44:06,036 2224878    [1-thread-1] DEBUG      ome.formats.OMEROMetadataStoreClient - KeepAlive failed.

2019-08-08 11:44:06,043 2224885    [1-thread-1] ERROR  me.formats.importer.util.ClientKeepAlive - Exception while executing ping(), logging Connector out: 

java.lang.RuntimeException: Ice.ObjectNotExistException

    id.name = "9db46812-9f65-44be-8f31-858f92ded272"

    id.category = "session-24bcb505-919e-4ed2-a7c2-fa3e49a489c5"

    facet = ""

    operation = "keepAllAlive"

	at ome.formats.OMEROMetadataStoreClient.ping(OMEROMetadataStoreClient.java:773) ~[omero-blitz.jar:5.5.3]

Did you try in another terminal to run bin/omero sessions keepalive?
This should keep the current session alive for the duration of the import.

Cheers

Jmarie

import_log.txt (14.1 KB) keepalive_log.txt (767 Bytes)

Still crashing, have captured the output of the import and of keepalive with ‘script’ and have attached the two log files.
Thank you for the help

Hi @nicola,

sorry for the slow response. Testing timeout issues is always time consuming… I will keep trying to reproduce the problem you’re seeing, but in case you want to try yourself, the usual workflow I follow is:

  1. Open terminal A
  2. bin/omero sessions keepalive
  3. Open terminal B
  4. bin/omero import …

but I noticed that in your case the session was created in terminal B rather than A. That shouldn’t matter, but it’s the only idea I have so far.

g’night.
~Josh

Hi @nicola,

I’ve managed to reproduce similar errors by reducing server timeouts (and have made various improvements: https://github.com/ome/omero-blitz/pull/65), but I’m not seeing the timeout when I force a long-wait. I’m beginning to suspect that it’s more than just the time and rather resource exhaustion on the server. Would you or someone with access to your server be able to upload the server logs and possibly take stack traces of what the server is doing during the import? e.g. during the 20 minute window before the timeout occurs:

2019-08-13 03:09:21,559 1898089    [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 6220
2019-08-13 03:25:50,925 2887455    [1-thread-1] ERROR  me.formats.importer.util.ClientKeepAlive - Exception while executing ping(), logging Connector out: 

Thanks,
~Josh

Thanks, I will do it first thing Monday morning.
Nicola

gdb_08_16.txt (16.5 KB) import_08_16.txt (1.7 MB) Indexer-0_log.txt (1.3 MB) PixelData-0_log.txt (346.0 KB)

These are the logs from Aug 16. I am trying now to do a few stacks in gdb.
Had to break the uploads because of system limits blitz_08_16_2019_08AMa.txt (13.6 MB)

keepalive_08_16.txt (1.1 KB)

Last piece
blitz_08_16_2019_08AMb.txt (18.5 MB)

Parts a,b,c of the log file in ManagedRepository…

ManagedRepository___08-03-17.236_log_a.txt (10.1 MB) ManagedRepository___08-03-17.236_log_c.txt (7.9 MB) ManagedRepository___08-03-17.236_log_b.txt (9.6 MB)

“resource exhaustion” !!!

  we just upgraded the server from 4gb to 8gb of ram, and the

import worked fine. I will repeat the test tomorrow

Thanks

Nicola Papp

Ah, that sounds promising. Keep us posted. ~Josh