Image import error using NFS mount to OMERO server container

Hi,

I am trying to deploy a Highly Available OMERO environment, with two OMERO servers mounted to the same NFS file system. I am using AWS EFS as the NFS storage. The deployment went well. But I got the errors when I tried to import images. I got different errors when I tried to load image to either one.

One error message is:

2021-03-11 21:02:20,306 567        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.8
2021-03-11 21:02:20,339 600        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.5.1 revision: 6f50e4d52c9d96112635fd8b2dde737f31041cf0 date: 7 July 2020
2021-03-11 21:02:23,694 3955       [      main] WARN                    ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to 5.6.1. See http://downloads.openmicroscopy.org/latest/omero for the latest version.
2021-03-11 21:02:23,869 4130       [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import: Directory exists but is not registered: CheckedPath(root_0)
omero.ResourceError: null
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]
	at java.base/java.lang.Class.newInstance(Class.java:584) ~[na:na]
	at IceInternal.BasicStream.createUserException(BasicStream.java:2785) ~[ice.jar:na]
	at IceInternal.BasicStream.access$300(BasicStream.java:14) ~[ice.jar:na]
	at IceInternal.BasicStream$EncapsDecoder11.throwException(BasicStream.java:3620) ~[ice.jar:na]
	at IceInternal.BasicStream.throwException(BasicStream.java:2291) ~[ice.jar:na]
	at IceInternal.OutgoingAsync.throwUserException(OutgoingAsync.java:399) ~[ice.jar:na]
	at omero.grid.ManagedRepositoryPrxHelper.end_importFileset(ManagedRepositoryPrxHelper.java:175) ~[omero-blitz.jar:5.5.8]
	at omero.grid.ManagedRepositoryPrxHelper.importFileset(ManagedRepositoryPrxHelper.java:46) ~[omero-blitz.jar:5.5.8]
	at omero.grid.ManagedRepositoryPrxHelper.importFileset(ManagedRepositoryPrxHelper.java:33) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary.createImport(ImportLibrary.java:468) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:613) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:354) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:328) ~[omero-blitz.jar:5.5.8]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

The other error message is:

2021-03-11 21:07:31,869 565        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.8
2021-03-11 21:07:31,904 600        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.5.1 revision: 6f50e4d52c9d96112635fd8b2dde737f31041cf0 date: 7 July 2020
2021-03-11 21:07:35,171 3867       [      main] WARN                    ome.system.UpgradeCheck - UPGRADE AVAILABLE:Please upgrade to 5.6.1. See http://downloads.openmicroscopy.org/latest/omero for the latest version.
2021-03-11 21:07:35,182 3878       [2-thread-1] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.RuntimeException: Cannot exclusively use the managed repository.

Likely no ManagedRepositoryPrx is being returned from the server.
This could point to a recent server crash. Ask your server administrator
to check for stale .lock files under the OMERO data directory. This
is particularly likely on a server using NFS.

	at ome.formats.importer.ImportLibrary.checkManagedRepo(ImportLibrary.java:894) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary.createImport(ImportLibrary.java:432) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:613) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:354) ~[omero-blitz.jar:5.5.8]
	at ome.formats.importer.ImportLibrary$1.call(ImportLibrary.java:328) ~[omero-blitz.jar:5.5.8]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Could anyone help me out here? I am happy to share more information if needed.

Best,
Steve

By the way, I also found the following error in Blitz-0.log file:

2021-03-12 02:10:07,359 WARN  [        ome.services.util.ServiceHandler] (.Server-10) Unknown exception thrown.

java.lang.NullPointerException: null
        at ome.services.blitz.repo.ManagedRepositoryI.makeCheckedDirs(ManagedRepositoryI.java:1644) ~[omero-blitz.jar:5.5.8]
        at ome.services.blitz.repo.PublicRepositoryI.makeDir(PublicRepositoryI.java:679) ~[omero-blitz.jar:5.5.8]
        at ome.services.blitz.repo.RepositoryDaoImpl$14.doWork(RepositoryDaoImpl.java:762) ~[omero-blitz.jar:5.5.8]
        at ome.services.blitz.repo.RepositoryDaoImpl$14.doWork(RepositoryDaoImpl.java:750) ~[omero-blitz.jar:5.5.8]
        at jdk.internal.reflect.GeneratedMethodAccessor267.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop.jar:4.3.14.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop.jar:4.3.14.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop.jar:4.3.14.RELEASE]
        at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568) ~[omero-server.jar:5.6.1]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE]
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154) ~[omero-server.jar:5.6.1]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE]
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119) ~[spring-orm.jar:4.3.14.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx.jar:4.3.14.RELEASE]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx.jar:4.3.14.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx.jar:4.3.14.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE]
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249) ~[omero-server.jar:5.6.1]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE]
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121) ~[omero-server.jar:5.6.1]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop.jar:4.3.14.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop.jar:4.3.14.RELEASE]
        at com.sun.proxy.$Proxy73.doWork(Unknown Source) ~[na:na]
        at ome.services.util.Executor$Impl.execute(Executor.java:447) ~[omero-server.jar:5.6.1]
        at ome.services.blitz.repo.RepositoryDaoImpl.makeDirs(RepositoryDaoImpl.java:749) ~[omero-blitz.jar:5.5.8]
        at ome.services.blitz.repo.PublicRepositoryI.makeDir(PublicRepositoryI.java:654) ~[omero-blitz.jar:5.5.8]
        at ome.services.blitz.repo.ManagedRepositoryI.createTemplatePath(ManagedRepositoryI.java:1374) ~[omero-blitz.jar:5.5.8]
        at ome.services.blitz.repo.ManagedRepositoryI.internalImport(ManagedRepositoryI.java:283) ~[omero-blitz.jar:5.5.8]
        at ome.services.blitz.repo.ManagedRepositoryI.importFileset(ManagedRepositoryI.java:257) ~[omero-blitz.jar:5.5.8]
        at omero.grid._ManagedRepositoryTie.importFileset(_ManagedRepositoryTie.java:66) ~[omero-blitz.jar:5.5.8]
        at omero.grid._ManagedRepositoryDisp.___importFileset(_ManagedRepositoryDisp.java:365) ~[omero-blitz.jar:5.5.8]
        at omero.grid._ManagedRepositoryDisp.__dispatch(_ManagedRepositoryDisp.java:564) ~[omero-blitz.jar:5.5.8]
        at IceInternal.Incoming.invoke(Incoming.java:221) ~[ice.jar:na]
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536) ~[ice.jar:na]
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145) ~[ice.jar:na]
        at Ice.ConnectionI.message(ConnectionI.java:1056) ~[ice.jar:na]
        at IceInternal.ThreadPool.run(ThreadPool.java:395) ~[ice.jar:na]
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12) ~[ice.jar:na]

Hi Steve,

We don’t test or support OMERO in a multi-write configuration. Obviously, if you are interested in investigating and helping us make it work, you’d have our appreciation!

On import, the server creates directories for the users. These are also registered in the PostgreSQL DB. From this, it sounds like the two OMERO’s are not using the same database. Is that right?

This is a race condition between the two instances trying to own a lock on the NFS directory. One of the OMERO’s is saying that it doesn’t “own” the directory.

So a few suggestions / options:

  • Could you tell us more about what you are trying to achieve? Maybe there’s a shorter path.
  • For example, you might be able to run the additional servers in a read-only mode, which is what we do with the IDR. (An ansible playbook is available in the idr/deployment repo.)
  • Alternatively, I can help you to work around the issues you listed above, but again: this isn’t a configuration that we’ve tested.

All the best,
~Josh

Thank you very much, Josh!

I want to deploy a high availability environment of OMERO, like this arch

but it seems I can only do this for read-only nodes, right? I got the AWS cloudformation (GitHub - flamingofugang/OMERO-on-AWS: This repo explains how to deploy OMERO on AWS ECS with different options) from this OMERO docker compose file: GitHub - ome/docker-example-omero: OMERO Docker compose example

Is there an easy way that I can make them read-only in docker compose file?

Best,
Steve

Hi Steve,

Sure. Please see docker-example-omero-readonly with the critical line being:

This and other examples are linked from GitHub - ome/omero-deployment-examples: Examples playbooks for installing OMERO with Ansible and Docker. If you would like to contribute back an AWS example to add to the Ansible & Docker examples, we’d love to see it!

~Josh

1 Like

Thank you very much, Josh! I tried to deploy the read-only OMERO server on AWS. I can deploy more than one read-only OMERO server to achieve the high availability.

However, when I tried to deploy one readwrite OMERO server, and one readonly OMERO server in one cluster, I got trouble. Since those two OMERO server containers cannot listen to the same port number for a given cluster, I made readonly listen to port 4063 and readwrite listen to port 4064. I have my AWS CloudFormation template attached here. It can be deployed without any issue.convertedOMEROonFargate.yml (27.7 KB)

But when I access the OMERO web portal, only the readwrite at port 4064 can work. The login to readonly at port 4063 got server no response error: Screen Shot 2021-03-28 at 7.22.19 PM

I checked the error log in the container: /opt/omero/server/OMERO.server/var/log/master.err, and got:
bash-4.2$ cat master.err
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions (file:/opt/omero/server/OMERO.server-5.6.3-ice36-b228/lib/server/javassist.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte,int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of javassist.util.proxy.SecurityActions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions (file:/opt/omero/server/OMERO.server-5.6.3-ice36-b228/lib/server/javassist.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte,int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of javassist.util.proxy.SecurityActions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions (file:/opt/omero/server/OMERO.server-5.6.3-ice36-b228/lib/server/javassist.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte,int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of javassist.util.proxy.SecurityActions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
-! 03/29/21 01:54:01.954 OMERO.Glacier2: warning: unable to contact permissions verifier BlitzVerifier@BlitzAdapters' Reference.cpp:1637: Ice::NoEndpointException: no suitable endpoint available for proxy BlitzVerifier -t -e 1.1 @ BlitzAdapters’
-! 03/29/21 01:54:01.955 OMERO.Glacier2: warning: unable to contact session manager BlitzManager@BlitzAdapters' Reference.cpp:1637: Ice::NoEndpointException: no suitable endpoint available for proxy BlitzManager -t -e 1.1 @ BlitzAdapters’
!! 03/29/21 01:54:17.723 error: communicator not destroyed during global destruction.!! 03/29/21 01:54:40.012 error: communicator not destroyed during global destruction

Have you seen this before? Do you know how to fix it?

Thx so much for all of your help! We did see a lot of interests to deploy OMERO on AWS. I think this work will be beneficial to many others :slight_smile:

1 Like

Hi Steve.

Thanks for sharing your CloudFormation template. Hopefully this will come in use for others.

I imagine the problem you are running into is a conflict on ports. 4063 is likely also in use by the read-write server. When running multiple OMERO instances on a single host we tend to use the omero.ports.prefix setting which will prepend the value all the standard ports. For example, 4061, 4063, and 4064 become 14061, 14063, 14064.

This won’t work for more than a handful of OMERO instances, but it’s a good place to start.

~Josh

Thx a lot, Josh! I tried to use port 5064 for readonly container, and got the same error…did I miss anything?

Best,
Steve

Hi Steve,

can you share the portion of your updated formation file? Also, can you check which ports are open on each of your VMs?

~Josh

Hi Josh,

Please find the Cloudformation for two OMERO servers attached. You will need a EC2 key pair, VPC, and two public subnest to deploy it. The network infrastructure can be created using this CloudFormation template: AWS CloudFormation VPC template - AWS CodeBuild

By the way, I have the single OMERO server in a HA deployment here: GitHub - aws-samples/digital-pathology-on-aws. I hope I can add the two OMERO servers one there as well :slight_smile:

Best,
SteveconvertedOMEROonFargate.yml (28.1 KB)

2 Likes

Hi Steve,

Thanks again for sharing all the configuration files. We’re looking forward to giving them a try, but won’t get to it this week. Just to clarify: are the two OMERO systems on the same virtual machine? If so, just changing the one port for the read-only server won’t suffice since all three of the ports have the potential to conflict.

… looking through the file it looks more like they are in containers and therefore shouldn’t conflict. Can you try setting omero.web.secure to true and see if that fixes your web login. If so, I think you will be better servered by using omero.ports.prefix and then mapping 14063 and 14064 of the read-only server externally.

~Josh.

Thx a lot, Josh!

I made the following change to add the environment variables in the read-only container:

  • Name: CONFIG_omero_ports_prefix
    Value: 1

and omero web container:

  • Name: CONFIG_omero_web_secure
    Value: true

But I got the same error for read only port
Please find the modified cloudformation template attached.

Best,
SteveconvertedOMEROonFargate.yml (28.2 KB)

Hi Steve,

I don’t see any references to “14064” or “14063” in your yml file. The omero.ports.prefix command will change all ports for that instance. Can you take a look at what ports you have open in each instance?

In my case:

$ ssh omeroreadonly-1 sudo netstat -nape | grep -E "406.*LISTEN"
tcp        0      0 127.0.0.1:4061          0.0.0.0:*               LISTEN      546        28675      2247/icegridnode
tcp6       0      0 :::4064                 :::*                    LISTEN      546        26890      2335/glacier2router
tcp6       0      0 :::4065                 :::*                    LISTEN      546        26892      2335/glacier2router
tcp6       0      0 :::4063                 :::*                    LISTEN      546        26891      2335/glacier2router

$ ssh omeroreadwrite sudo netstat -nape | grep -E "406.*LISTEN"
tcp        0      0 127.0.0.1:4061          0.0.0.0:*               LISTEN      546        249423893  3982/icegridnode
tcp        0      0 127.0.0.1:4063          0.0.0.0:*               LISTEN      546        249411563  4033/glacier2router
tcp        0      0 127.0.0.1:4064          0.0.0.0:*               LISTEN      546        249411562  4033/glacier2router
tcp        0      0 127.0.0.1:4065          0.0.0.0:*               LISTEN      546        249411564  4033/glacier2router

(port 4065 is for websockets)

~Josh