OMERO server, bandwidth problems

Hi everyone,

I have problems with the OMERO installation we assembled at our institute for testing purposes. OMERO.server works, OMERO.web works, it is possible to upload images from OMERO.insight, to visualize them, to download them, but there is still a big problem mentioned by the “beta-testers”: the time to upload images to the server is very long compared to the actual server we use (a server based on windows server, basic, for data storage, not really fine-tuned for imaging). Also, when trying to download images from the OMERO server to a PC, i experienced failures: for example, the downloading of a czi image of about 90 Mb fails at about 72 Mbs, and trying to download a metamorph image (nd + tif files) fails with a 504 error.

So, what are the minimal recommandations for an OMERO server to work properly?
I’m assume it depends of the purpose of the server and of the number of users, but i’m pretty sure there is a problem: we have currently at least between 5-10 persons (maybe less) who are using it for testing, and certainly not at the same time…

On my side, i’ll try to acquire some informations on the infrastructure data for both servers (the omero testing server and the current one based on Windows). If you have an idea of the critical values i should examine in priority, feel free to tell me.

Thanks by advance, Marc.

Hi @mmongy
Could you try a local import of the same file, so we can rule out network problems?

  1. Copy the file to a temp directory on your OMERO.server
  2. Do a local CLI import: /opt/omero/server/OMERO.server/bin/omero import ... and show us the CLI logs, they should include timestamps.

Hi manics,

In fact, my senior at the informatics department helped me to solve the problem:

  • He modified /etc/nginx/conf.d/omeroweb.conf like that:
    proxy_connect_timeout 3600;
    proxy_send_timeout 3600;
    proxy_read_timeout 3600;
    send_timeout 3600;
  • But the real problem was on the physical switch, which shut down the port when the connection is at 1 gigabyte/sec.

The problem of import to OMERO is solved, but there are still problems at downloading: this time i get a 502 Bad Gateway error instead of the 504 error.

Hi @mmongy. Could you check your nginx/error.log and OMEROweb.log for warnings or errors?

Hi @manics manics,

Here are the logs:

—NGINX error log:

2020/07/29 10:30:22 [error] 5716#5716: *1170 upstream prematurely closed connection while reading response header from upstream, client: 000.000.000.000, server: omero.xxx.xxxxx, request: “GET /webgateway/archived_files/download/?image=551&zipname=100x%20oil%20SR.nd.zip HTTP/1.1”, upstream: “http://127.0.0.1:4080/webgateway/archived_files/download/?image=551&zipname=100x%20oil%20SR.nd.zip”, host: “omero.xxx.xxxxx”, referrer: “http://omero.xxx.xxxxx/webclient/download_placeholder/?ids=image-551&fileCount=&name=100x%20oil%20SR.nd

~

—OMEROweb.log

2020-07-29 09:30:26,791 INFO [ omeroweb.webclient.views] (proc.14783) ():112 INIT ‘14783’
2020-07-29 09:30:26,795 INFO [ root] (proc.14783) process_custom_settings():1009 Processing custom settings for module omeroweb.api.api_settings
2020-07-29 09:30:26,796 INFO [ root] (proc.14783) process_custom_settings():1009 Processing custom settings for module omero_iviewer.iviewer_settings
2020-07-29 09:30:26,847 INFO [ omero.gateway] (proc.14783) connect():2274 created connection (uuid=5ea29912-8349-4e60-9c11-cf276297bda3)
2020-07-29 09:30:26,849 INFO [ omero.gateway] (proc.14783) close():1987 closed connection (uuid=5ea29912-8349-4e60-9c11-cf276297bda3)
~
~

But something is odd, the timestamp for OMEROweb.log antidated by 1 hour compared to the local time.

And yesterday, my senior noticed a weird process (with ps -aux), with a 60 sec timeout. He is almost sure it is the cause, because he measured the time between clicking “Create zip” and the displaying of the 502 error Bad Gateway (60 seconds)

/opt/omero/web/venv3/bin/python3 /opt/omero/web/venv3/bin/gunicorn -D -p /opt/omero/web/omero-web/var/django.pid --bind 127.0.0.1:4080 --workers 5 --threads 1 –timeout 60 --max-requests 0 omeroweb.wsgi:application

In general the default configuration of OMERO.web and gunicorn isn’t suitable for downloads that take a long time to prepare or transfer. You could try increasing omero.web.wsgi_timeout

1 Like

Thanks. I tuned it to 300 seconds. The download went well.

1 Like