Problems With Omero.Web after Upgrade to 5.5.0, Unable to Revert to 5.4.10

Today I thought it would be fun to try upgrading omero from 5.4.10 to 5.5.0. I ran the following steps:
Did a sql dump backup
Downloaded the new server files
Stopped Omero
mv OMERO.server OMERO.server-old
ln -s OMERO.server-5.5.0-ice36-b121 OMERO.server
cp OMERO.server-old/etc/grid/config.xml OMERO.server/etc/grid

bin/omero admin start
bin/omero web start

Both seem to start fine, omero.web copies a whole bunch of files.

But when I try to go to OMERO.web, it yields a 500 (internal server error). Yikes! From the logs, it looks like some of the plugins we’re using (OMERO.marshal?) do not support 5.5.0.

So I try to go back to 5.4.10

mv OMERO.server OMERO.server-5.5-abort
ln -s OMERO.server-5.4.10 OMERO.server

bin/omero admin start

And this basically times out. It spends ~ 300 seconds until it says:
Waiting on startup. Use CTRL-C to exit

Failed to startup some components after 300 seconds
Calling “stop” on remaining components
Waiting on shutdown. Use CTRL-C to exit

So then I go back to 5.5.0, and the core-omero server still works, but OMERO.web is still broken. Any idea what I did wrong trying to revert? I really just would like to revert back to 5.4.10, but I’m not sure what went wrong.

Thanks for the help,
Philip

Here are the logs when I try to restart 5.4.10:

-! 06/20/19 20:25:08.549 OMERO.Glacier2: warning: unable to contact permissions verifier `BlitzVerifier@BlitzAdapters'
   Reference.cpp:1637: Ice::NoEndpointException:
   no suitable endpoint available for proxy `BlitzVerifier -t -e 1.1 @ BlitzAdapters'
-! 06/20/19 20:25:08.549 OMERO.Glacier2: warning: unable to contact session manager `BlitzManager@BlitzAdapters'
   Reference.cpp:1637: Ice::NoEndpointException:
   no suitable endpoint available for proxy `BlitzManager -t -e 1.1 @ BlitzAdapters'
!! 06/20/19 20:25:23.598 error: communicator not destroyed during global destruction.

Hi Philip,

Did you stop 5.5 before restarting 5.4? If not, there may still be processes running that need cleaning up. Assuming this is the only OMERO instance running on this machine, First ‚killall icegridnode‘ and then restart 5.4.

~Josh

I ran that command it said “no process found”
I tried to start it again and I think I’m still getting the same issue.

What does the Blitz-0.log show for the 5.4 server?

Sorry for the wait, I tried restarting the server which did not seem to fix the problem.

Here are the blitz logs, this is probably over a few attempts:

2019-06-20 20:25:05,257 INFO  [                ome.services.blitz.Entry] (      main) Creating OMERO.blitz. Please wait...
2019-06-20 20:25:06,616 INFO  [        ome.services.util.ReadOnlyStatus] (      main) read-only status: db=false, repo=false
2019-06-20 20:25:06,643 INFO  [.s.ShutdownSafeEhcacheManagerFactoryBean] (      main) Initializing EHCache CacheManager
2019-06-20 20:25:06,734 INFO  [          ome.services.util.DBPatchCheck] (      main) Verified database patch: OMERO5.4__0
2019-06-20 20:25:06,771 INFO  [           ome.services.util.DBUserCheck] (      main) User root.id = 0
2019-06-20 20:25:06,771 INFO  [           ome.services.util.DBUserCheck] (      main) User guest.id = 1
2019-06-20 20:25:06,771 INFO  [           ome.services.util.DBUserCheck] (      main) Group system.id = 0
2019-06-20 20:25:06,771 INFO  [           ome.services.util.DBUserCheck] (      main) Group user.id = 1
2019-06-20 20:25:06,771 INFO  [           ome.services.util.DBUserCheck] (      main) Group guest.id = 2
2019-06-20 20:25:08,360 INFO  [  ome.services.fulltext.FullTextAnalyzer] (      main) Initialized FullTextAnalyzer
2019-06-20 20:25:08,519 INFO  [.s.ShutdownSafeEhcacheManagerFactoryBean] (      main) Shutting down EHCache CacheManager
2019-06-20 20:25:08,526 ERROR [                ome.services.blitz.Entry] (      main) Error on startup.
org.springframework.beans.factory.access.BootstrapException: Unable to return specified BeanFactory instance: factory key [OMERO.blitz], from group with resource name [classpath*:beanRefContext.xml]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'OMERO.blitz' defined in URL [jar:file:/home/omero/omero_versions/OMERO.server-5.4.10-ice36-b105/lib/server/blitz.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'ome.server' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ome.server' defined in URL [jar:file:/home/omero/omero_versions/OMERO.server-5.4.10-ice36-b105/lib/server/server.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [ome.system.OmeroContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'groupChmodStrategy' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'omeroSessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'omeroSessionFactory' defined in class path resource [ome/services/hibernate.xml]: Cannot resolve reference to bean 'sessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory' defined in class path resource [ome/services/hibernate.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: could not init listeners
        at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:409) ~[spring-beans.jar:3.0.1.RELEASE]
        at ome.system.OmeroContext.getInstance(OmeroContext.java:202) ~[common.jar:na]
        at ome.services.blitz.Entry.start(Entry.java:189) [blitz.jar:na]
        at ome.services.blitz.Entry.main(Entry.java:146) [blitz.jar:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'OMERO.blitz' defined in URL [jar:file:/home/omero/omero_versions/OMERO.server-5.4.10-ice36-b105/lib/server/blitz.jar!/beanRefContext.xml]: Cannot resolve reference to bean 'ome.server' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ome.server' defined in URL [jar:file:/home/omero/omero_versions/OMERO.server-5.4.10-ice36-b105/lib/server/server.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [ome.system.OmeroContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'groupChmodStrategy' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'omeroSessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'omeroSessionFactory' defined in class path resource [ome/services/hibernate.xml]: Cannot resolve reference to bean 'sessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory' defined in class path resource [ome/services/hibernate.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: could not init listeners
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:586) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:140) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:984) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:888) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:479) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1048) ~[spring-context.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.access.SingletonBeanFactoryLocator.useBeanFactory(SingletonBeanFactoryLocator.java:397) ~[spring-beans.jar:3.0.1.RELEASE]
        ... 3 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ome.server' defined in URL [jar:file:/home/omero/omero_versions/OMERO.server-5.4.10-ice36-b105/lib/server/server.jar!/beanRefContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [ome.system.OmeroContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'groupChmodStrategy' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'omeroSessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'omeroSessionFactory' defined in class path resource [ome/services/hibernate.xml]: Cannot resolve reference to bean 'sessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory' defined in class path resource [ome/services/hibernate.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: could not init listeners
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:281) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:984) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:888) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:479) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:189) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322) ~[spring-beans.jar:3.0.1.RELEASE]
        ... 16 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [ome.system.OmeroContext]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'groupChmodStrategy' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'omeroSessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'omeroSessionFactory' defined in class path resource [ome/services/hibernate.xml]: Cannot resolve reference to bean 'sessionFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory' defined in class path resource [ome/services/hibernate.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: could not init listeners
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:141) ~[spring-beans.jar:3.0.1.RELEASE]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:107) ~[spring-beans.jar:3.0.1.RELEASE]
... 

There’s more but I hit the forum limit

No worries, always good to retry. Unfortunately this part of the log doesn’t tell me what’s going wrong. If you can get the whole file somewhere, I’ll take a look tomorrow. Night. ~Josh

Here is the full log file, hope this helps:
https://pastebin.com/UhUyt1sD
Thanks again for all the help.

Caused by: org.apache.lucene.index.CorruptIndexException: Unknown format version: -11

Starting the 5.5 server modified the search index of in /OMERO/FullText. You will need to either restore from your backup or wipe the index:

https://docs.openmicroscopy.org/omero/5.4.10/sysadmins/search.html#re-indexing

~Josh

Hi Philip

Could you check the version of omero-marshal you have installed?
OMERO 5.5 comes with omero-marshal 0.5.4, this is the version that you will need in order to use OMERO.web 5.5.0.

Thanks

Jmarie

I have some good news…
Before attempting to wipe and re-index lucene, I thought it would be good to try updating OMERO.marshal. I ran pip install omero-marshal --upgrade, which updated omero-marshal to 0.6.0. Awesomely, this did seem to fix the error with Omero.web. Should I downgrade to omero-marshal 0.5.4 from 0.6.0?

Thanks again for all the help.

No need to downgrade, you can use OMERO.marshal 0.6.0
Glad that everything is working

Jmarie