Error 504 when many users log in to OMERO.web simultaneously

Hi,
we have added about 350 students to a group to offer a virtual microscopy course. Unfortunately our server crashes as soon as several users try to log in at the same time. The exercises take place with about 80 students at a time, so I assume that not more than these 80 will try to log in at a time.

If the server / OMERO.web crashes, we get a 504 timeout error.

What possibilities do I have to provide access for more students at the same time? Which error logs do you need? I can’t figure out the entries in /var/log/nginx error.log:

2020/11/10 17:52:45 [error] 2387#0: *1130408 connect() failed (111: Connection refused) while connecting to upstream, client: 37.201.170.108, server: localhost, request: "GET /webclient/activities/?_=1605027084564 HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/activities/?_=1605027084564", host: "omero-imaging.uni-muenster.de", referrer: "http://omero-imaging.uni-muenster.de/webclient/"              
2020/11/10 18:01:34 [error] 2389#0: *1130436 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 37.201.171.215, server: localhost, request: "GET /webclient/keepalive_ping/?_=1604994680356 HTTP/1.1", upstream: "http://127.0.0.1:4080/webclient/keepalive_ping/?_=1604994680356", host: "omero-imaging.uni-muenster.de", referrer: "http://omero-imaging.uni-muenster.de/webclient/userdata/?experimenter=-1"

We running
OMERO.web 5.8.1
OMERO.server 5.6.2

Any help is welcome :wink:

Best,
Thomas

Hi Thomas

The logs you should check include:

  • OMERO.web/var/log/OMEROweb.log
  • OMERO.server/var/log/Blitz-0.log

Could you also show us the output of

  • omero config get for your OMERO.server and OMERO.web
  • omero admin diagnostics for your OMERO.server

Thanks!

2 Likes

Dear Simon,

Thanks for your fast reply. I am at home now and con show you the logs.
Diagnostics:

Commands:   java -version                  11.0.6    (/usr/bin/java)                                                    
Commands:   python -V                      3.6.8     (/opt/omero/venv_server/bin/python -- 2 others)                    
Commands:   icegridnode --version          3.6.5     (/usr/bin/icegridnode)                                             
Commands:   icegridadmin --version         3.6.5     (/usr/bin/icegridadmin)                                            
Commands:   psql --version                 11.6      (/usr/bin/psql)                                                    
Commands:   openssl version                1.0.2     (/usr/bin/openssl)                                                                                                                                                                         
Server:     icegridnode                    running                                                                      
Server:     Blitz-0                        active (pid = 14470, enabled)                                                
Server:     DropBox                        inactive (disabled)                                                          
Server:     FileServer                     inactive (disabled)                                                          
Server:     Indexer-0                      active (pid = 14499, enabled)                                                
Server:     MonitorServer                  inactive (disabled)                                                          
Server:     OMERO.Glacier2                 active (pid = 14502, enabled)                                                
Server:     OMERO.IceStorm                 active (pid = 14506, enabled)                                                
Server:     PixelData-0                    active (pid = 14513, enabled)                                                
Server:     Processor-0                    active (pid = 14524, enabled)                                                
Server:     Tables-0                       inactive (disabled)                                                          
Server:     TestDropBox                    inactive (enabled)                                                                                                                                                                                  

Log dir:    /opt/omero/OMERO.server/var/log exists                                                                      
Log files:  Blitz-0.log                    266.6 MB      errors=0    warnings=79                                        
Log files:  Blitz-0.log.1                  524.3 MB      errors=5    warnings=502                                       
Log files:  Blitz-0.log.2                  524.3 MB      errors=0    warnings=152                                       
Log files:  Blitz-0.log.3                  524.4 MB      errors=0    warnings=115                                       
Log files:  Blitz-0.log.4                  524.3 MB      errors=1    warnings=134                                       
Log files:  Blitz-0.log.5                  524.3 MB      errors=0    warnings=180                                       
Log files:  Blitz-0.log.6                  524.3 MB      errors=3    warnings=426                                       
Log files:  Blitz-0.log.7                  524.3 MB      errors=13   warnings=296                                       
Log files:  Blitz-0.log.8                  524.4 MB      errors=5    warnings=998                                       
Log files:  Blitz-0.log.9                  524.4 MB      errors=6    warnings=891                                       
Log files:  DropBox.log                    n/a                                                                          
Log files:  FileServer.log                 n/a                                                                          
Log files:  Indexer-0.log                  13.0 MB       errors=16   warnings=2699                                      
Log files:  MonitorServer.log              n/a                                                                          
Log files:  PixelData-0.log                3.0 MB        errors=51   warnings=94                                        
Log files:  Processor-0.log                1.6 MB        errors=0    warnings=18                                        
Log files:  Processor-0.log.1              5.0 MB        errors=7    warnings=51                                        
Log files:  Processor-0.log.2              5.0 MB        errors=5    warnings=20                                        
Log files:  Processor-0.log.3              5.0 MB        errors=0    warnings=2                                         
Log files:  Processor-0.log.4              5.0 MB                                                                       
Log files:  Processor-0.log.5              5.0 MB                                                                       
Log files:  Processor-0.log.6              5.0 MB                                                                       
Log files:  Processor-0.log.7              5.0 MB        errors=2    warnings=16                                        
Log files:  Processor-0.log.8              5.0 MB        errors=0    warnings=12                                        
Log files:  Tables-0.log                   n/a                                                                          
Log files:  TestDropBox.log                n/a                                                                          
Log files:  master.err                     13.5 MB       errors=0    warnings=26                                        
Log files:  master.out                     empty                                                                        
Log files:  Total size                     5060.85 MB                                                                                                                                                                                                                                                                                                                   
Environment:OMERO_HOME=(unset)                                                                                          
Environment:OMERODIR=/opt/omero/OMERO.server                                                                            
Environment:OMERO_NODE=(unset)                                                                                          
Environment:OMERO_MASTER=(unset)                                                                                        
Environment:OMERO_USERDIR=(unset)                                                                                       
Environment:OMERO_TMPDIR=(unset)                                                                                        
Environment:PATH=/opt/omero/venv_server/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/centos/.local/bin:/home/centos/bin                                                                                                          
Environment:PYTHONPATH=(unset)                                                                                          
Environment:ICE_HOME=(unset)                                                                                            
Environment:LD_LIBRARY_PATH=(unset)                                                                                     
Environment:DYLD_LIBRARY_PATH=(unset)                                                                                                                                                                                                           
OMERO SSL port:4064                                                                                                     
OMERO TCP port:4063                                                                                                     
OMERO data dir:'/opt/omero/data_dir'          Exists? True      Is writable? True                                       
OMERO temp dir:'/home/omero/omero/tmp'        Exists? True      Is writable? True   (Size: 534399020)                                                                                                                                           
JVM settings: Blitz-${index}                -Xmx5034m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions               
JVM settings: Indexer-${index}              -Xmx3356m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions               
JVM settings: PixelData-${index}            -Xmx5034m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions               
JVM settings: Repository-${index}           -Xmx3356m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

omero config get :

omero.data.dir=/opt/omero/data_dir
omero.db.name=omero_database
omero.db.pass=********
omero.db.user=***
omero.ldap.base=
omero.ldap.config=true
omero.ldap.new_user_group=Demo
omero.ldap.password=********
omero.ldap.referral=follow
omero.ldap.urls=
omero.ldap.user_mapping=omeName=cn,firstName=givenName,lastName=sn,email=mail
omero.ldap.username=
omero.mail.config=true
omero.mail.from=*******
omero.mail.host= ****
omero.web.public.enabled=True
omero.web.public.password=*******
omero.web.public.url_filter=^/(webadmin/myphoto/|webclient/(?!(script_ui|ome_tiff|figure_script))|webgateway/(?!(archived_files|download_as))|iviewer|api)
omero.web.public.user=Public-User

I deleted some values (mainly ldab and mail) as I guess it is not important here. If you need it just tell me.

If I look into OMERO.web/var/log/OMEROweb.log, I get a lot entries ;-). Unfortunately I am not an experienced centos user (I hope my system admin will have look in here tomorrow) and I don’t know exactly what I am looking for. I tried grep "error" and got this:

2020-11-05 15:23:17,869 ERROR [                 omeroweb.feedback.views] (proc.20600) handler500():163 handler500: Server error
2020-11-09 08:40:10,854 ERROR [                 omeroweb.feedback.views] (proc.32726) handler500():163 handler500: Server error
2020-11-09 15:19:12,265 ERROR [                 omeroweb.feedback.views] (proc.15414) handler500():163 handler500: Server error
2020-11-10 06:28:24,651 ERROR [                 omeroweb.feedback.views] (proc.24308) handler500():163 handler500: Server error
2020-11-10 06:28:27,784 ERROR [                 omeroweb.feedback.views] (proc.23320) handler500():163 handler500: Server error
2020-11-10 06:28:48,424 ERROR [                 omeroweb.feedback.views] (proc.23320) handler500():163 handler500: Server error
2020-11-10 09:16:05,862 ERROR [                 omeroweb.feedback.views] (proc.13359) handler500():163 handler500: Server error
2020-11-10 09:16:09,133 ERROR [                 omeroweb.feedback.views] (proc.04365) handler500():163 handler500: Server error
2020-11-10 09:16:12,993 ERROR [                 omeroweb.feedback.views] (proc.04365) handler500():163 handler500: Server error
2020-11-10 09:16:21,021 ERROR [                 omeroweb.feedback.views] (proc.14990) handler500():163 handler500: Server error
2020-11-10 09:16:33,186 ERROR [                 omeroweb.feedback.views] (proc.14990) handler500():163 handler500: Server error
2020-11-10 09:16:33,306 ERROR [                 omeroweb.feedback.views] (proc.15557) handler500():163 handler500: Server error

Here are some other entries:

2020-11-10 11:04:33,779  INFO [                           omero.gateway] (proc.23118) connect():2274 created connection (uuid=b0c2ed8a-6569-4af5-afb7-b746f3e2bb01)
2020-11-10 11:04:33,782  INFO [                           omero.gateway] (proc.27445) _unregister_service():1646 Unregistered b0c2ed8a-6569-4af5-afb7-b746f3e2bb01/1436b8f1-5967-4a86-b12e-ffbd5119da05omero.api.RenderingEngine -t -e 1.1:tcp -h "2001:4cf0:8:0:0:0:0:50%eth0" -p 38352 -t 60000:tcp -h 10.14.29.83 -p 38352 -t 60000                                                                                                                                                    
2020-11-10 11:04:33,784  INFO [                           omero.gateway] (proc.27445) close():1987 closed connection (uuid=b0c2ed8a-6569-4af5-afb7-b746f3e2bb01)                                                                                                                                                            
2020-11-10 11:04:33,798  INFO [                           omero.gateway] (proc.27445) connect():2274 created connection (uuid=b0c2ed8a-6569-4af5-afb7-b746f3e2bb01)                                                                                                                                                         
2020-11-10 11:04:33,806  INFO [                           omero.gateway] (proc.20459) _unregister_service():1646 Unregistered b0c2ed8a-6569-4af5-afb7-b746f3e2bb01/ffc55906-7909-4bca-a001-3ccd022df07bomero.api.RenderingEngine -t -e 1.1:tcp -h "2001:4cf0:8:0:0:0:0:50%eth0" -p 38352 -t 60000:tcp -h 10.14.29.83 -p 38352 -t 60000                                                                                                                                                    
2020-11-10 11:04:33,809  INFO [                           omero.gateway] (proc.20459) close():1987 closed connection (uuid=b0c2ed8a-6569-4af5-afb7-b746f3e2bb01)                                                                                                                                                            
2020-11-10 11:04:33,820  INFO [                           omero.gateway] (proc.20459) connect():2274 created connection (uuid=b0c2ed8a-6569-4af5-afb7-b746f3e2bb01)                                                                                                                                                         
2020-11-10 11:04:34,035  INFO [                           omero.gateway] (proc.23118) _register_service():1637 Registered b0c2ed8a-6569-4af5-afb7-b746f3e2bb01/686e274b-16bc-42e3-b334-fba3f6ac676eomero.api.RenderingEngine -t -e 1.1:tcp -h "2001:4cf0:8:0:0:0:0:50%eth0" -p 38352 -t 60000:tcp -h 10.14.29.83 -p 38352 -t 60000                                                                                                                                                        
2020-11-10 11:04:34,054  INFO [                           omero.gateway] (proc.27445) _register_service():1637 Registered b0c2ed8a-6569-4af5-afb7-b746f3e2bb01/82ebf966-8d2f-4c9b-94e7-e29b116494bdomero.api.RenderingEngine -t -e 1.1:tcp -h "2001:4cf0:8:0:0:0:0:50%eth0" -p 38352 -t 60000:tcp -h 10.14.29.83 -p 38352 -t 60000                                                                                                                                                        
2020-11-10 11:04:34,131  INFO [                           omero.gateway] (proc.23118) _unregister_service():1646 Unregistered b0c2ed8a-6569-4af5-afb7-b746f3e2bb01/686e274b-16bc-42e3-b334-fba3f6ac676eomero.api.RenderingEngine -t -e 1.1:tcp -h "2001:4cf0:8:0:0:0:0:50%eth0" -p 38352 -t 60000:tcp -h 10.14.29.83 -p 38352 -t 60000                                                                                                                                                    
2020-11-10 11:04:34,133  INFO [                           omero.gateway] (proc.23118) close():1987 closed connection (uuid=b0c2ed8a-6569-4af5-afb7-b746f3e2bb01)                                 

I guess the server were down at this time.

And here is the outcome of the grep "Error" Blitz-0.log

2020-11-10 17:53:20,492 INFO  [        ome.services.util.ServiceHandler] (l.Server-8)  Args:    [Error]                                                       
2020-11-10 17:53:22,007 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Args:    [Error]                                                       
2020-11-10 17:53:23,201 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Args:    [Error]                                                       
2020-11-10 17:53:24,395 INFO  [        ome.services.util.ServiceHandler] (l.Server-8)  Args:    [Error]                                                       
2020-11-10 17:53:25,466 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Args:    [Error]                                                       
2020-11-10 17:53:26,631 INFO  [        ome.services.util.ServiceHandler] (l.Server-4)  Args:    [Error]                                                       
2020-11-10 17:53:27,906 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Args:    [Error]                                                       
2020-11-10 17:53:29,112 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Args:    [Error]                                                       
2020-11-10 17:53:30,282 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Args:    [Error]                                                       
2020-11-10 17:53:31,556 INFO  [        ome.services.util.ServiceHandler] (l.Server-8)  Args:    [Error]                                                       
2020-11-10 17:53:32,710 INFO  [        ome.services.util.ServiceHandler] (l.Server-3)  Args:    [Error]                                                       
2020-11-10 17:53:34,279 INFO  [        ome.services.util.ServiceHandler] (l.Server-5)  Args:    [Error]                                                       
2020-11-10 17:53:35,526 INFO  [        ome.services.util.ServiceHandler] (l.Server-5)  Args:    [Error]                                                       
2020-11-10 17:53:36,646 INFO  [        ome.services.util.ServiceHandler] (l.Server-4)  Args:    [Error]                                                       
2020-11-10 17:53:37,787 INFO  [        ome.services.util.ServiceHandler] (l.Server-2)  Args:    [Error]                                                       
2020-11-10 17:53:38,949 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Args:    [Error]                                                       
2020-11-10 17:53:39,980 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Args:    [Error]                                                       
2020-11-10 17:53:41,155 INFO  [        ome.services.util.ServiceHandler] (l.Server-3)  Args:    [Error]                                                       
2020-11-10 17:53:42,365 INFO  [        ome.services.util.ServiceHandler] (l.Server-8)  Args:    [Error]                                                       
2020-11-10 17:53:43,423 INFO  [        ome.services.util.ServiceHandler] (l.Server-7)  Args:    [Error] 
2020-11-10 20:41:00,953 INFO  [                 org.perf4j.TimingLogger] (1-thread-3) start[1605037260951] time[2] tag[omero.call.success.ome.services.sessions.SessionManagerImpl$6.doWork]                                                                                                                                
2020-11-10 20:41:00,954 INFO  [        ome.services.util.ServiceHandler] (1-thread-3)  Rslt:    (ome.model.meta.Experimenter:Id_52, ome.model.meta.ExperimenterGroup:Id_904, (), ... 5 more)                                                                                                                                
2020-11-10 20:41:00,954 INFO  [        ome.services.util.ServiceHandler] (1-thread-3)  Executor.doWork -- ome.services.sessions.SessionManagerImpl.reload[3bb36eb8-6edf-4de4-a0db-dd23a2fd2e82]                                                                                                                             
2020-11-10 20:41:00,954 INFO  [        ome.services.util.ServiceHandler] (1-thread-3)  Args:    [null, InternalSF@418957165]                                  2020-11-10 20:41:00,954 INFO  [         ome.security.basic.EventHandler] (1-thread-3)  Auth:    user=0,group=0,event=null(Sessions),sess=8b621f25-12ea-48ea-b704-73534acd1a66 

At 20:41 the server was down again.
Here are the last entries of the OMEROweb.log:

2020-11-10 19:53:05,763  INFO [                           omero.gateway] (proc.07594) close():1987 closed connection (uuid=fb57a2f8-4295-4dba-868a-5d5612766467)                                                                                                                                                            
2020-11-10 19:53:05,773  INFO [                           omero.gateway] (proc.07594) connect():2274 created connection (uuid=fb57a2f8-4295-4dba-868a-5d5612766467)                                                                                                                                                         
2020-11-10 19:53:06,563  INFO [                           omero.gateway] (proc.07594) close():1987 closed connection (uuid=fb57a2f8-4295-4dba-868a-5d5612766467)                                                                                                                                                            
2020-11-10 19:53:06,585  INFO [                           omero.gateway] (proc.07594) connect():2274 created connection (uuid=055c6102-61ee-4f19-b813-35c50b765860)                                                                                                                                                         
2020-11-10 19:53:06,587  INFO [                           omero.gateway] (proc.07594) close():1987 closed connection (uuid=055c6102-61ee-4f19-b813-35c50b765860)                                                                                                                                                            
2020-11-10 19:53:06,599  INFO [                           omero.gateway] (proc.07594) connect():2274 created connection (uuid=1aa6bf36-3d85-40cd-a64d-359d8edacc3e)                                                                                                                                                         
2020-11-10 19:53:06,601  INFO [                           omero.gateway] (proc.07594) close():1987 closed connection (uuid=1aa6bf36-3d85-40cd-a64d-359d8edacc3e)                                                                                                                                                            
2020-11-10 19:53:06,612  INFO [                           omero.gateway] (proc.07594) connect():2274 created connection (uuid=fb57a2f8-4295-4dba-868a-5d5612766467)                                                                                                                                                         
2020-11-10 19:53:07,285  INFO [                           omero.gateway] (proc.07594) close():1987 closed connection (uuid=fb57a2f8-4295-4dba-868a-5d5612766467)                                                                                                                                                            
2020-11-10 19:53:07,299  INFO [                           omero.gateway] (proc.07594) connect():2274 created connection (uuid=fbe6c551-aa7d-47c6-81da-c58c4a144136) 

Thanks a lot for the help!
All the best,
Thomas

@T-Zobel immediate thought while glancing over your configuration is that you might want to increase the value of omero.db.poolsize.

See https://github.com/ome/prod-playbooks/blob/3027ede41396c49cb8ae4c5dede19a58a718b6e6/learning.yml#L55 for an example of the production configuration for the learning instance running at the University of Dundee.

@s.besson
Thanks for your help at this time!
I changed the omero.db.poolsize to 100 now. Is there a way to test if it helps? Or do I have to wait for the next course :slight_smile:

EDIT: Should I change the other values as well (omero.jvmcfg.percent.blitz: 50 omero.jvmcfg.percent.indexer: 20 omero.jvmcfg.percent.pixeldata: 30)
And where do I have to configure my database installation to accept as many connections?

After setting the db.poolsize to 100 and restarting the server, we got again a 504 timeout. I assume that many students want to view or download the data at the moment to prepare for the course.

Hi,
I had some time to read the docs and found omero.threads.background_threads. As I assume many students would like to download data at the same time, should I increase the max background threads? Is downloading a background task?

Hi

Multiple concurrent downloads are known to be a problem with OMERO.web due to how the Django threading is handled, so it’s probably worth spending some time to tune your configuration.

Roughly how many concurrent downloads are you expecting and what size are they? Does the server and web recover after a while, or is a restart always necessary of one or both components? If it’s the latter that may indicate a deeper problem.

There’s a bit of related (technical!) background information on https://github.com/ome/omero-web/issues/241

You could try increasing omero.web.wsgi_workers

Today I tried to optimize the system as described under Performance and monitoring.
Here is my new config:

omero.db.name=omero_database
omero.db.pass=********
omero.db.poolsize=200
omero.db.user=db_min
omero.jvmcfg.percent=90
omero.jvmcfg.percent.blitz=30
omero.jvmcfg.percent.indexer=30
omero.jvmcfg.percent.pixeldata=20
omero.jvmcfg.system_memory=32000
omero.ldap.base=OU=Projekt-Benutzer,DC=WWU,DC=DE
omero.ldap.config=true
omero.ldap.new_user_group=Demo
omero.ldap.password=********
omero.ldap.referral=follow
omero.ldap.urls=ldap://wwu.de
omero.ldap.user_mapping=omeName=cn,firstName=givenName,lastName=sn,email=mail
omero.ldap.username=cn=imaging,ou=Projekt-Benutzer,dc=wwu,dc=de
omero.mail.config=true
omero.mail.from=tzobe_01@uni-muenster.de
omero.mail.host=mail.uni-muenster.de
omero.threads.background_threads=50
omero.threads.max_threads=100
omero.web.public.enabled=True
omero.web.public.password=********
omero.web.public.url_filter=^/(webadmin/myphoto/|webclient/(?!(script_ui|ome_tiff|figure_script))|webgateway/(?!(archived_files|download_as))|iviewer|api)
omero.web.public.user=Public-User
omero.web.wsgi_workers=12

I touched these settings:
omero.jvmcfg.percent 90
omero config set omero.jvmcfg.percent.blitz 30
omero config set omero.jvmcfg.percent.indexer 30
omero config set omero.jvmcfg.percent.pixeldata 20
omero config set omero.jvmcfg.system_memory 32000
omero config set omero.db.poolsize 200
omero.threads.max_threads 100
omero.threads.background_threads 50
omero.web.wsgi_workers 12

The downloads are very different. From small images without z up to large tilescans inkluding z (>10GB). I will think about another way to share the data. It could be that >20 people try to download. Hard to guess.

Does the server and web recover after a while, or is a restart always necessary of one or both components?

I´ve waited 3 times more then 10 min (longest 25min) and it did not recover. I am allways restarting both. Actually I have to stop web, restart the server, start web again. If I do it in another order, I might have to start the web server twice.

Hi Thomas,

I could provide you with our plugin for sharing data and speeding up web downloads. However, you would have to configure your nginx server to use this plugin.

Best,
Susanne

1 Like

Hi Susanne,

I think it would be worth a try. I will write you an email.
2nd option would be to put the data on a cloud/network drive for now. But I would like to have evrything in one place.

Best,
Thomas

I’m sure we’d all be interested as well! :wink:

Hi Josh,
that was smart move :wink:
See you later ?

Something else…
Susanne gave me the tip to check what the server is doing with htop.
htop shows me the following:
tasks: >70
Threads: >250
Running: mostly 1, shortly up to 8, 3 for some time now
Of the 270 threads, at least 200 are from the user omero, omero-web, nginx.
By the way, only OMERO runs on the VM.

There is no student course at the moment. If a course is running, it is possible that 60-80 students using omero-web at the same time.

Are these the threads I set with omero.db.poolsize (now 400) and omero.threads.max_threads (now 300)? And is it normal that these are already so high, although hardly anyone uses omero (at 7am in the morning)? Threads and processor activity went up at about 8:30 am.

Hi @T-Zobel,

What do you mean by “tasks”? Each Java and Python process could easily have a couple dozen threads even when idling. As long as they aren’t using load, I wouldn’t worry about that. How many connections do you have enabled in PostgreSQL itself?

And just catching up: your primary concern at the moment is still downloads with the server becoming unresponsive if too much data is downloaded, or are there still other issues?

~Josh

Hi @joshmoore,

after increasing omero.db.poolsize, omero.web.wsgi_workers and so on, the server runs more stable. However, I still get from time to time the 504 timeout nginx error and yes, the server becoming unresposive from time to time. That´s my primary concern.
I am not sure if this coming from downloading data. I told the students not to download anymore, but I guess they do. I had no time to install @sukunis plugin yet. We will report if we dit.

Today I had 65 individual users at the same time online (omero sessions who) and 115 sessions in total (some multiple and quite many from the public user).

What do you mean by “tasks”?

That are the values htop is giving me.

How many connections do you have enabled in PostgreSQL itself?

Using htop I got > 20 postgres entries some time before I got the 504 Error again.

Best,
Thomas

In what other ways is the server responsive? e.g. insight & CLI as well?