OMERO.server update 5.6.1: Server Error.(500)

Hi,
after updating to OMERO 5.6.1 (5.6 was running without problems before). We get an error when opening images in the i.viewer, omero.viewer or on preview tab. For some reasons a few images are working.

Traceback (most recent call last):

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omeroweb/decorators.py", line 488, in wrapped
    retval = f(request, *args, **kwargs)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omeroweb/decorators.py", line 538, in wrapper
    context = f(request, *args, **kwargs)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omeroweb/webclient/views.py", line 1622, in load_metadata_preview
    rdefId = manager.image.getRenderingDefId()

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 7780, in wrapped
    if not self._prepareRenderingEngine() \

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8056, in _prepareRenderingEngine
    self._re = self._prepareRE(rdid=rdid)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 8035, in _prepareRE
    re.load(ctx)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4749, in __call__
    return self.handle_exception(e, *args, **kwargs)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omeroweb/webclient/webclient_gateway.py", line 2126, in handle_exception
    e, *args, **kwargs)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omero/gateway/__init__.py", line 4746, in __call__
    return self.f(*args, **kwargs)

  File "/opt/omero/venv_web/lib64/python3.6/site-packages/omero_api_RenderingEngine_ice.py", line 1192, in load
    return _M_omero.api.RenderingEngine._op_load.invoke(self, ((), _ctx))

omero.ResourceError: exception ::omero::ResourceError
{
    serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /opt/omero/data_dir/ManagedRepository/Thomas_2/2020-03/02/10-37-05.121/Image 15_2 SIMApotome 561 488 simul Leap_SIM.czi
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:646)
	at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:564)
	at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
	at jdk.internal.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:568)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy81.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:447)
	at ome.services.util.Executor$Impl.execute(Executor.java:392)
	at ome.services.RenderingBean.getPixelBuffer(RenderingBean.java:2202)
	at ome.services.RenderingBean.load(RenderingBean.java:417)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy110.load(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy110.load(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
	at ome.services.throttling.Callback.run(Callback.java:56)
	at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
	at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
	at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at omero.cmd.CallContext.invoke(CallContext.java:85)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy112.load_async(Unknown Source)
	at omero.api._RenderingEngineTie.load_async(_RenderingEngineTie.java:248)
	at omero.api._RenderingEngineDisp.___load(_RenderingEngineDisp.java:1197)
	at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:2382)
	at IceInternal.Incoming.invoke(Incoming.java:221)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
	at Ice.ConnectionI.message(ConnectionI.java:1056)
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:834)

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /opt/omero/data_dir/ManagedRepository/Thomas_2/2020-03/02/10-37-05.121/Image 15_2 SIMApotome 561 488 simul Leap_SIM.czi
}

<WSGIRequest: GET '/webclient/metadata_preview/image/10698/?_=1585306940180'>

Do you have any ideas?
Best,
Thomas

Maybe some background information:
OMERO runs on an open-stack VM. The data partition is mounted via an nfs share.
I took a snapshot of the server before and created a new one. So the data partition is also mounted on our normal server at the same time.

The snapshot VM was perfectly working prior to the update, so the snapshot was fine.

Maybe i have can provide some more informations on that issue.
In the Blitz-0.log file we can see the following error message when opening images:

[   o.s.b.repo.ManagedReaderSecurityCheck] (.Server-38) reader for /opt/omero/data_dir/ManagedRepository/Thomas_2/2020-03/02/10-37-05.121/Image 15_2 SIMApotome 561 488 simul Leap_SIM.czi accesses data outside managed repository:
	/opt/omero/data_dir/ManagedRepository/Thomas_2/2020-03/02/10-37-05.121/Image 15_2 SIMApotome 561 488 simul Leap_SIM.czi
[ome.io.nio.PixelsService] (.Server-38) Error instantiating pixel buffer: /opt/omero/data_dir/ManagedRepository/Thomas_2/2020-03/02/10-37-05.121/Image 15_2 SIMApotome 561 488 simul Leap_SIM.czi
java.lang.RuntimeException: ome.conditions.SecurityViolation: reader for /opt/omero/data_dir/ManagedRepository/Thomas_2/2020-03/02/10-37-05.121/Image 15_2 SIMApotome 561 488 simul Leap_SIM.czi accesses data outside managed repository:
	/opt/omero/data_dir/ManagedRepository/Thomas_2/2020-03/02/10-37-05.121/Image 15_2 SIMApotome 561 488 simul Leap_SIM.czi

Hope that helps to resolve the issue.

Aha, that’s very helpful, thank you. From the server’s environment can you tell us the output from,

omero fs repos
omero config get omero.data.dir
omero config get omero.managed.dir

Is there some clear relationship between which images can be opened and which can’t? I am wondering if, for example, it depends on who is the owner and who is the viewer - indeed, what the relationship is between them.

No, i can’t see one. Even from the same import session one image was working and another not.
I tested it again right now, and no image is working. Even the one which was working before now shows the error.

omero fs repos:

# | Id    | UUID                                 | Type   | Path                                                 
---+-------+--------------------------------------+--------+------------------------------------------------------
 0 | 58    | ScriptRepo                           | Script | /opt/omero/OMERO.server-5.6.1-ice36-b225/lib/scripts 
 1 | 59    | 3f5040c8-9fc4-4063-9a16-0a5acc5f3484 | Public | /home/omero/OMERO_HDD                                
 2 | 60    | c8cb2e9a-8f36-4300-a6f4-57bb14987ba0 | Public | /opt/omero/data_dir/ManagedRepository                
 3 | 1151  | afee5439-5bd6-4b04-a094-95e2dcb61a72 | Public | /home/omero/OMERO_HDD                                
 4 | 1152  | 52fe9c5f-a3df-435b-8067-cad70a4e47ca | Public | /home/omero/OMERO_HDD/ManagedRepository              
 5 | 1251  | 74cc8662-773a-4103-ac0a-b4927f82adae | Public | /home/omero/OMERO_HDD/ManagedRepository              
 6 | 1252  | 08320c5b-e537-471f-9864-42044244ab57 | Public | /home/omero/OMERO_HDD                                
 7 | 23217 | 900b17f7-f58c-4202-8716-b80a6a5dfa83 | Public | /opt/omero/data_dir                                  

omero data dir:

/opt/omero/data_dir/

I do not get an output from omero.managed.dir

omero fs repos
 # | Id    | UUID                                 | Type   | Path                                                 
---+-------+--------------------------------------+--------+------------------------------------------------------
 0 | 58    | ScriptRepo                           | Script | /opt/omero/OMERO.server-5.6.1-ice36-b225/lib/scripts 
 1 | 59    | 3f5040c8-9fc4-4063-9a16-0a5acc5f3484 | Public | /home/omero/OMERO_HDD                                
 2 | 60    | c8cb2e9a-8f36-4300-a6f4-57bb14987ba0 | Public | /opt/omero/data_dir/ManagedRepository                
 3 | 1151  | afee5439-5bd6-4b04-a094-95e2dcb61a72 | Public | /home/omero/OMERO_HDD                                
 4 | 1152  | 52fe9c5f-a3df-435b-8067-cad70a4e47ca | Public | /home/omero/OMERO_HDD/ManagedRepository              
 5 | 1251  | 74cc8662-773a-4103-ac0a-b4927f82adae | Public | /home/omero/OMERO_HDD/ManagedRepository              
 6 | 1252  | 08320c5b-e537-471f-9864-42044244ab57 | Public | /home/omero/OMERO_HDD                                
 7 | 23217 | 900b17f7-f58c-4202-8716-b80a6a5dfa83 | Public | /opt/omero/data_dir                                  
(8 rows)

The output from omero config get omero.managed.dir was empty.

omero config get omero.data.dir
    /opt/omero/data_dir/

Maybe it has something to do with the location of the data dir. The path /opt/omero/data_dir is a symlink to /home/omero/OMERO_HDD/data

ls -la /opt/omero/ | grep data
lrwxrwxrwx. 1 root      root         26 30. Jan 15:34 data_dir -> /home/omero/OMERO_HDD/data

We changed the location to that symlink when we upgraded from 5.5 to 5.6 some months ago. Maybe that is the root of the problem.

Ok, i will hand over to @Christian
He definitely knows better what he is doing then I do. :smiley:

As you have the snapshot would it be fairly quick to simply repeat the upgrade? The guess is that the server got confused if it was started before that data directory was enough accessible by it. (One would normally have a Managed somewhere in that “Type” column but it can get changed if the directory seems bad.) We can try to figure how to de-confuse it but that might be the longer, harder course from where you now are.

Yeah we can try to make a new snapshot. I think there was problem with the mount point on the snapshot machine on first start. But i can’t say for sure because Thomas was doing it.

Btw: When i issue the command omero fs repos on the production system there is one Managed repository. Its the one with ID 60.

First sorry for my late reply. I had to take care of my kids.

I created a new VM from the Snapshot.
Server: 5.6
Web: 5.6.1

I had to mount the data directory. Now everything works and I do not get any errors.

omero fs repos
 # | Id    | UUID                                 | Type    | Path                                                 
---+-------+--------------------------------------+---------+------------------------------------------------------
 0 | 58    | ScriptRepo                           | Script  | /opt/omero/OMERO.server-5.6.0-ice36-b136/lib/scripts 
 1 | 59    | 3f5040c8-9fc4-4063-9a16-0a5acc5f3484 | Public  | /home/omero/OMERO_HDD                                
 2 | 60    | c8cb2e9a-8f36-4300-a6f4-57bb14987ba0 | Managed | /opt/omero/data_dir/ManagedRepository                
 3 | 1151  | afee5439-5bd6-4b04-a094-95e2dcb61a72 | Public  | /home/omero/OMERO_HDD                                
 4 | 1152  | 52fe9c5f-a3df-435b-8067-cad70a4e47ca | Public  | /home/omero/OMERO_HDD/ManagedRepository              
 5 | 1251  | 74cc8662-773a-4103-ac0a-b4927f82adae | Public  | /home/omero/OMERO_HDD/ManagedRepository              
 6 | 1252  | 08320c5b-e537-471f-9864-42044244ab57 | Public  | /home/omero/OMERO_HDD                                
 7 | 23217 | 900b17f7-f58c-4202-8716-b80a6a5dfa83 | Public  | /opt/omero/data_dir                        

For omero.managed.dir I get no output on the new server made from the snapshot.

As @Christian already wrote, we have now, before the update, a managed in the Type column (ID 60). So this has not changed due to the snap and creating the server.

I’m glad things are working well for you now, but are you still back on the previous release or have you yet managed to upgrade okay to OMERO.server 5.6.1, OMERO.web 5.6.3? It should go fine if the data directory is mounted when the server starts.

Christian has done the update now and everything works fine.
Thanks for the hint with omero fs repo. That the OMERO data partition showed public was the problem. So, when we make the update from a snapshot, we always have to make sure that the OMERO data partition is displayed as manged when we output omero fs repo.

However, I don’t know exactly what this depends on. I have set up a new VM about 8 times over the weekend and only got the managed one time, otherwise always public. :man_shrugging:

We will update the productive server tomorrow. Here I will unmount the OMERO data partition before the snapshot. Hopefully this helps.

Thanks a lot for your help!
Best,
Thomas

Great, thank you for the good news. You probably lose the “managed” if you start the server without the data partition properly mounted or perhaps also if some other running server also has it mounted.

Thanks for the hints :+1:
I will post my step-by-step procedure for creating the snap and a second VM here if it works. Maybe it will help someone who also works with openstack.

I did the update yesterday on our productive system. Everything worked out fine. :grinning: :+1:
If somebody has a similar problem in the future, here is how I proceeded:

Background:
OMERO runs on an OpenStack virtual machine. Omero data partition is mounted in two virtual machines as an NFS share.

Problem:
The output of omero fs repos showed public after creating a virtual machine from a snapshot. The omero data partition should be of Type: managed So, when we make the update from a snapshot, we always have to make sure that the OMERO data partition is displayed as manged when we output omero fs repos .

Creating the Snapshot:

  • Stopping omero.web & omero.server in all VMs
  • Stopping all VMs
  • Snapshot productive VM
  • New VM from the Snapshot
  • Add new IP rule to the NFS-share
  • Mount (if necessary) the NFS-share
  • Start Omero.server and Omero.web
  • Check with omero fs repos (omero server environment) if Typeof the omero data partition is managed
  • Update as described in the OMERO DOCs
  • Switch floating IP

@mtbc Thanks again for you help!
All the best,
Thomas

1 Like

Thank you very much for telling us how it all went, we’re sure glad to know you have OMERO 5.6.1 running fine now.