Cannot restart OMERO server 5.4.10

Recently found we could not login to OMERO via our web server or OMERO insight. Logged into our CENTOS7 OS and tried to use ‘omero admin stop’ or ‘omero admin restart’ but both timeout with ‘Failed to startup some components after 300 seconds’

‘omero admin diagnostics’ output says

================================================================================
OMERO Diagnostics (admin) 5.4.10-ice36-b105

WARNING:omero.util.UpgradeCheck:UPGRADE AVAILABLE:Please upgrade to 5.6.1. See http://downloads.openmicroscopy.org/latest/omero for the latest version.

Commands: java -version 1.8.0 (/usr/bin/java)
Commands: python -V 2.7.5 (/usr/bin/python)
Commands: icegridnode --version 3.6.4 (/usr/bin/icegridnode)
Commands: icegridadmin --version 3.6.4 (/usr/bin/icegridadmin)
Commands: psql --version 9.6.11 (/usr/bin/psql)

Server: icegridnode running
Server: Blitz-0 deactivating (enabled)
Server: DropBox inactive (disabled)
Server: FileServer inactive (disabled)
Server: Indexer-0 inactive (disabled)
Server: MonitorServer inactive (disabled)
Server: OMERO.Glacier2 inactive (disabled)
Server: OMERO.IceStorm inactive (disabled)
Server: PixelData-0 inactive (disabled)
Server: Processor-0 inactive (disabled)
Server: Tables-0 inactive (disabled)
Server: TestDropBox inactive (enabled)

Log dir: /home/omero/OMERO.server/var/log exists
Log files: Blitz-0.log 0.0 KB
Log files: Blitz-0.log.1 2.0 GB errors=188 warnings=53064
Log files: DropBox.log 52.0 KB errors=15 warnings=20
Log files: FileServer.log 7.0 KB
Log files: Indexer-0.log 15.0 MB errors=9 warnings=55
Log files: MonitorServer.log 25.0 KB
Log files: OMEROweb.lock 0.0 KB
Log files: OMEROweb.log 50.0 MB errors=270 warnings=612
Log files: PixelData-0.log 4.0 MB errors=6 warnings=26
Log files: Processor-0.log 3.0 MB errors=0 warnings=2
Log files: Processor-0.log.1 4.0 MB errors=0 warnings=3168
Log files: Processor-0.log.2 4.0 MB errors=0 warnings=2
Log files: Processor-0.log.3 4.0 MB errors=6 warnings=14
Log files: Processor-0.log.4 4.0 MB
Log files: Processor-0.log.5 4.0 MB
Log files: Processor-0.log.6 4.0 MB errors=1 warnings=11
Log files: Tables-0.log 90.0 KB errors=6 warnings=96
Log files: TestDropBox.log n/a
Log files: log_Blitz-0.log.backup 139.0 MB errors=85 warnings=1603
Log files: master.err 15.0 KB errors=7 warnings=14
Log files: master.out 0.0 KB
Log files: Total size 3023.91 MB

Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/omero/.local/bin:/home/omero/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

OMERO SSL port:4064
OMERO TCP port:4063

(and then hangs)

We have also checked the tail end of Blitz-0.log.1 but was not sure what to look for. Any help would be greatly appreciated!

The “Blitz-0 deactivating” is interesting. Does ps ax | grep Domero.name= report any OMERO java processes? If so then it might be worth kill-ing them manually, making sure they have actually gone, then have another go at starting the server and watching its logs.

If that fails, are master.out and Blitz-0.log completely empty? (From v5.5.0 this is clearer, it’s very well worth upgrading to v5.6.1.) If so, maybe at least send master.err and Blitz-0.log.1 along to us, zip them up and submit to http://qa.openmicroscopy.org.uk/qa/upload/ or make it so we can get them from somewhere, and we’ll see if those bear any clues.

Hi mtbc,

I did not get any results from ps ax | grep Domero.name= but i did ps ax | grep omero and there were a few idle omero database transactions. I have included a screenshot in the files i have uploaded. Yes, it looks look like both master.out and Blitz-0.log are both empty.

I was able to use the uploader you mention, but am unsure if you automatically get the upload? After it succesfully uploaded, I did not recieve any sort of link to give you. However, I have also uploaded the zip files here.

Thanks for your help!

Would it be worth restarting PostgreSQL perhaps? Maybe something’s gotten blocked up there. At this moment I’m guessing though, I didn’t see anything clear from the logs but thank you for sharing them.

When you’ve tried to start OMERO and it fails, does that ps command above report any OMERO java processes then?

Could this be a good time to simply take the database and binary repository and do the upgrade to OMERO 5.6? We now even provide Ansible roles to handle much of the setup if you want to migrate your OMERO state onto a fresh system.

Thanks - we did try restarting pgsql, but no changes.

A colleague took a look based on the message about the postgresql dump in our ps aux | grep “omero” and realized we had a trouble connecting to our /OMERO storage via a network setting and corrected this. Once this was corrected, it looked like our system was back online and we were able to commence with the upgrade to 5.6. Thanks so much!