Large file download issue in OMERO 5.6.3

Hello all!

Several OMERO users report issues when downloading large files (> 4GB) using OMERO.web.

The reported issues when it comes to file downloads are:

a) Large files via omero.web (starts but stops between 4 and 4.2gb)

b) Any files via omero.insight client - download button shows no dropdown.

No issues with file downloads were reported before the upgrade from 5.5.1 to 5.6.3.

Async workers were enabled as per: https://docs.openmicroscopy.org/omero/5.4.0/sysadmins/unix/install-web/web-deployment.html#experimental-async-workers

I am happy to provide our nginx config if necessary.

Any ideas or someone has found any similar issues?

Kind regards and thanks!

Hi and Welcome to the forum,

We’re not aware of any changes that would cause the problems you’re describing.

Do you know if OMERO.insight download worked with these files before the upgrade?

When downloading via the web client, are you downloading a single file at a time (not generating zips with multiple files)?

When you say “starts but stops between 4 and 4.2 GB”, you mean that 4.2 GB of the file downloads before it stops? How big is the file in total (approx)? How long does it take to get to 4.2 GB? Is it possible that you’re seeing a timeout that didn’t occur before because the download rate is slower now?

nginx config could be useful, and it might also be useful to see the output of:

$ omero config get

(remove any sensitive settings).

If you can think of any infrastructure (hardware or software) changes you made when upgrading, that might also help.

Regards,

Will.

Hi Will,

I’m the user mentioned above.

The file to be downloaded is a single file - granted a large one, on the order of 60-80gb which is not unusual for us these days.

It works fine in both omero.web and omero.insight if pointed to our 5.5 instance, but when trying to perform the same download from the 5.6 instance it begins the download (in web) but stops and fails around the 4gb mark. In chrome it restarts from zero three times before reporting that it’s failed - I’m not sure if firefox restarts or not but it also fails the download.

Similarly, using omero.insight 5.5.9 connecting to our omero 5.5 instance I can download files without issue, but when connecting to our 5.6 the download options are either greyed out in the right-click context menus, or the popup doesn’t appear when clicking the download button.

Thanks,
-Lachie

Hi both,

I wonder if the web download is failing because of a timeout or just because it’s slow. Before the failure, do you notice a difference in the rate of download between 5.5 and 5.6?
How long does it take to get to 4 GB?

Web logs from the server could be helpful to see.

For Insight, you sometimes see download options greyed-out and sometimes the popup doesn’t appear. Do you see both these things happen “randomly” on the same image, or do you get different behaviours consistently with different images?

You could see if there are any useful errors in the Insight logs (in your home directory omero/log/omeroinsight.log).

Also if you have the nginx and other server configs, that could be handy.
Are the 5.5 and 5.6 servers installed on similar hardware, network etc?

Thanks,

Will

I’ve had a look and can’t see a significant difference in speed of the download between the two instances, takes about a minute to get to 4GB before failing/restarting.

The download is greyed out and doesn’t appear in the drop down for any image - although I’ve just noticed the upload option is similarly greyed out so I’ll follow up on that with @miesav

Hello Will,

Currently, we haven’t updated our production instance of Omero yet. We are testing on a clone of the production server, which has 8 CPUs and 16 GB of RAM.

Before updating to 5.6.3, we updated psql to version 11 and updated the system from CentOS 7.1 to 7.7.

Our configurations:

omero.data.dir=/OMERO/

omero.db.name=omero_database

omero.db.pass=********

omero.db.user=omero_db_user

omero.fs.importArgs=--transfer=upload_rm

omero.jvmcfg.percent.blitz=85

omero.ldap.base=ou=Unix,dc=wehi,dc=edu,dc=au

omero.ldap.config=true

omero.ldap.group_filter=(cn=grp_img_*)

omero.ldap.group_mapping=name=cn

omero.ldap.new_user_group=:filtered_dn_attribute:memberOf

omero.ldap.sync_on_login=true

omero.ldap.urls=ldap://ldapserver #removed

omero.ldap.user_mapping=omeName=uid,firstName=givenName,lastName=sn,email=mail

omero.policy.binary_access=-read,+write,+image,-plate

omero.security.default_permissions=rwrw--

omero.web.application_server=wsgi-tcp

omero.web.application_server.max_requests=0

omero.web.apps=["omero_iviewer", "omero_figure", "omero_fpbioimage"]

omero.web.open_with=[["Image viewer", "webgateway", {"supported_objects": ["image"], "script_url": "webclient/javascript/ome.openwith_viewer.js"}], ["omero_iviewer", "omero_iviewer_index", {"supported_objects": ["images", "dataset", "well"], "script_url": "omero_iviewer/openwith.js", "label": "OMERO.iviewer"}], ["omero_figure", "new_figure", {"supported_objects": ["images"], "target": "_blank", "label": "OMERO.figure"}], ["omero_fpbioimage", "fpbioimage_index", {"supported_objects": ["image"], "script_url": "fpbioimage/openwith.js", "label": "FPBioimage"}]]

omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Figure", "figure_index", {"target": "_blank", "title": "Open Figure in new tab"}]]

omero.web.view=omero_iviewer.views.index

omero.web.wsgi_worker_class=gevent

omero.web.wsgi_worker_connections=1000
upstream omeroweb {
    server 127.0.0.1:4080 fail_timeout=0;
}

server {
    listen 80;
    server_name localhost;

    sendfile on;
    proxy_max_temp_file_size 0;

    # maintenance page serve from here
    location @maintenance {
        root /opt/omero/web/omero-web/etc/templates/error;
        try_files $uri /maintainance.html =502;
    }

    # weblitz django apps serve media from here
    location /static {
        alias /opt/omero/web/omero-web/var/static;
    }

    location @proxy_to_app {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_buffering off;

        proxy_pass http://omeroweb;
    }

    location / {

        error_page 502 @maintenance;
        # checks for static file, if not found proxy to app
        try_files $uri @proxy_to_app;
    }

}

Kind regards.

Hi @miesav,

You show your configuration as a single block. Can you break it down into which configuration is in the /opt/omero/web/omero-web directory and which is in (I assume) /opt/omero/server/omero-server?

All the best,
~Josh

Hello Josh,

omero-web config:

omero.web.application_server=wsgi-tcp
omero.web.application_server.max_requests=0
omero.web.wsgi_worker_class=gevent
omero.web.wsgi_worker_connections=1000
omero.web.wsgi_workers=17

omero-server config (running under the user ‘omero’, which was running 5.5.1. This configuration was copied into 5.6.3 from 5.5.1):

omero.data.dir=/OMERO/
omero.db.name=omero_database
omero.db.pass=********
omero.db.user=omero_db_user
omero.fs.importArgs=--transfer=upload_rm
omero.jvmcfg.percent.blitz=85
omero.ldap.base=ou=Unix,dc=wehi,dc=edu,dc=au
omero.ldap.config=true
omero.ldap.group_filter=(cn=grp_img_*)
omero.ldap.group_mapping=name=cn
omero.ldap.new_user_group=:filtered_dn_attribute:memberOf
omero.ldap.sync_on_login=true
omero.ldap.urls=ldap://
omero.ldap.user_mapping=omeName=uid,firstName=givenName,lastName=sn,email=mail
omero.policy.binary_access=-read,+write,+image,-plate
omero.security.default_permissions=rwrw--
omero.web.application_server=wsgi-tcp
omero.web.application_server.max_requests=0
omero.web.apps=["omero_iviewer", "omero_figure", "omero_fpbioimage"]
omero.web.open_with=[["Image viewer", "webgateway", {"supported_objects": ["image"], "script_url": "webclient/javascript/ome.openwith_viewer.js"}], ["omero_iviewer", "omero_iviewer_index", {"supported_objects": ["images", "dataset", "well"], "script_url": "omero_iviewer/openwith.js", "label": "OMERO.iviewer"}], ["omero_figure", "new_figure", {"supported_objects": ["images"], "target": "_blank", "label": "OMERO.figure"}], ["omero_fpbioimage", "fpbioimage_index", {"supported_objects": ["image"], "script_url": "fpbioimage/openwith.js", "label": "FPBioimage"}]]
omero.web.ui.top_links=[["Data", "webindex", {"title": "Browse Data via Projects, Tags etc"}], ["History", "history", {"title": "History"}], ["Help", "http://help.openmicroscopy.org/", {"target": "new", "title": "Open OMERO user guide in a new tab"}], ["Figure", "figure_index", {"target": "_blank", "title": "Open Figure in new tab"}]]
omero.web.view=omero_iviewer.views.index
omero.web.wsgi_worker_class=gevent
omero.web.wsgi_worker_connections=1000

Kind regards,

Hi. Could you please show us your OMEROweb.log, Nginx error.log and Nginx access.log corresponding to the download? Thanks!

Hello OME Team,

We tested for a short window of time. Indeed it seems like a timeout somewhere:

nginx/error.log

2020/04/08 10:10:52 [error] 22422#0: *20 upstream prematurely closed connection while reading upstream, client: 10.1.6.101, server: localhost, request: "GET /webgateway/archived_files/download/135785/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webgateway/archived_files/download/135785/", host: "omero-dev.wehi.edu.au", referrer: "http://omero-dev.wehi.edu.au/webclient/"
2020/04/08 10:11:54 [error] 22419#0: *23 upstream prematurely closed connection while reading upstream, client: 10.1.6.101, server: localhost, request: "GET /webgateway/archived_files/download/135785/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webgateway/archived_files/download/135785/", host: "omero-dev.wehi.edu.au", referrer: "http://omero-dev.wehi.edu.au/webclient/"
2020/04/08 10:13:06 [error] 22419#0: *58 upstream prematurely closed connection while reading upstream, client: 10.1.6.101, server: localhost, request: "GET /webgateway/archived_files/download/136882/ HTTP/1.1", upstream: "http://127.0.0.1:4080/webgateway/archived_files/download/136882/", host: "omero-dev.wehi.edu.au", referrer: "http://omero-dev.wehi.edu.au/webclient/"

nginx-access.log.txt (95.8 KB) OMEROweb.log.txt (54.9 KB)

So far the only issue we see is the download through the OMERO.web. It does appear to be an nginx issue. Will dig around a bit more.

Also a bit of a separate question. The documentation about migrating Omero to python 3 mentions this (https://docs.openmicroscopy.org/omero/5.6.0/sysadmins/python3-migration.html):

Once you have your installation in place, you will need to follow the standard upgrade instructions, working from a copy of your data.

Unfortunately, our Omero data directory is on the large side (~40TB), which makes it hard for us to copy or retrieve from backups. Do you have any recommendations about testing new versions of Omero in this case? Is it possible to create a subset of the OMERO data dir?

Kind regards and thanks!

:+1:

This assumes that you have your production server on Python 2 and haven’t yet set up a Python 3 environment. We were concerned about someone breaking their working environment without intending to.

You can start up a test OMERO.server with a copy of your database but without a copy of /OMERO/ManagementRepository. OMERO should work fine. You should only notice the missing data when you try to open an image in the testing instance since that will throw an exception. (You can also omit /OMERO but then you will also lose thumbnails and full text search, which may still be ok for a test.)

All the best,
~Josh

We have 60TB on 180TB Disks. To test a new version I created a snapshot and mounted that ‘ro’ with a ‘rw’ overlay filesystem to /OMERO_test. That works…

best regards
Andreas

1 Like

I have a question about this @joshmoore …do you see any harm in omero.server and omero.web sharing a config file? that’s how I have things currently set up for convenience (since I have one service account running both OMERO.web and OMERO.server on the same machine). Easy enough to change, though.

Not particularly, no. More containment tends to be better though. For example, there’s no need for the OMERO.web process to have access to the omero.db.password property.

The primary reason for separating the two components from one another is to deal with the differing release cycles. We intend OMERO.web to be released more proactively and the separation prevents needing to restart OMERO.server at the same time. (Note, however, that that’s largely independent of whether they use the same config or not.)

All the best,
~Josh

1 Like

A post was split to a new topic: OMERO.figure for 5.6.3

@miesav As we might have a similar problem (Download stops after ~60sec) I would like to ask if you have found a solution for the problem.

All the best,
Thomas

Hi miesav,

did you succeed in resolving the issue?
We had the same problem with our OMERO installation. I partially solved it by increasing the wscgi timeout of omero web to 30mins (default value is 60s), e.g.
omero config set omero.web.wsgi_timeout 1800.
I don’t know whether this has other (negative) effects but downloads that take longer than 60s work now.

Kind regards
Christian

Sorry if this is a bit late, but adding these to the nginx virtual server configuration helped in case someone else runs into this:

server {

    ...
    client_max_body_size 0;
    proxy_read_timeout 2h;
    fastcgi_read_timeout 2h;
    ...

}

The default value is 1 minute.

Cheers,

Miguel

1 Like