Problem after installation of OMERO.server 5.5.1 (Ubuntu 18.04 Server)

Hi everyone.

After following the procedure to install OMERO.server (https://docs.openmicroscopy.org/omero/5.5.1/sysadmins/unix/install-web/walkthrough/omeroweb-install-with-server-ubuntu1804-ice3.6.html), i am stuck with this error when starting the server:

mmongy@ubuntu-server-test-2:/home/omero$ OMERO.server/bin/omero admin start
Traceback (most recent call last):
  File "OMERO.server/bin/omero", line 130, in <module>
    rv = omero.cli.argv()
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1706, in argv
    cli.invoke(args[1:])
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1159, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1236, in onecmd
    self.execute(line, previous_args)
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1318, in execute
    args.func(args)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/prefs.py", line 74, in open_and_close_config
    return func(*args, **kwargs)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/admin.py", line 803, in start
    self.startasync(args, config)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/prefs.py", line 74, in open_and_close_config
    return func(*args, **kwargs)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/admin.py", line 754, in startasync
    self.checkice()
  File "/home/omero/OMERO.server/lib/python/omero/plugins/admin.py", line 1652, in checkice
    popen = self.ctx.popen(["icegridnode", "--version"])
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1409, in popen
    stdout=stdout, stderr=stderr)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Any ideas?

Cheers, Marc.

Hi @mmongy. Welcome to image.sc!

Could you include the output from https://docs.openmicroscopy.org/omero/5.5.1/sysadmins/unix/server-installation.html?highlight=diagnostics#omero-diagnostics ? I’m assuming the issue is that you are missing one of the Ice executables like icegridnode.

Cheers,
~Josh

Thanks for the answer. Here the diagnostic output:

mmongy@ubuntu-server-test-2:/home/omero$ OMERO.server/bin/omero admin diagnostics

================================================================================
OMERO Diagnostics (admin) 5.5.1-ice36-b122
================================================================================
        
Commands:   java -version                  1.8.0     (/usr/bin/java)
Commands:   python -V                      2.7.15    (/usr/bin/python)
Commands:   icegridnode --version          not found
Commands:   icegridadmin --version         not found
Commands:   psql --version                 10.10     (/usr/bin/psql)
Commands:   openssl version                1.1.111   (/usr/bin/openssl)

No icegridadmin available: Cannot check server list

Log dir:    /home/omero/OMERO.server/var/log 
No logs available

Environment:OMERO_HOME=(unset)             
Environment:OMERO_NODE=(unset)             
Environment:OMERO_MASTER=(unset)           
Environment:OMERO_USERDIR=(unset)          
Environment:OMERO_TMPDIR=(unset)           
Environment:PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin 
Environment:PYTHONPATH=(unset)             
Environment:ICE_HOME=(unset)               
Environment:LD_LIBRARY_PATH=(unset)        
Environment:DYLD_LIBRARY_PATH=(unset)      

Traceback (most recent call last):
  File "OMERO.server/bin/omero", line 130, in <module>
    rv = omero.cli.argv()
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1706, in argv
    cli.invoke(args[1:])
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1159, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1236, in onecmd
    self.execute(line, previous_args)
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1318, in execute
    args.func(args)
  File "/home/omero/OMERO.server/lib/python/omero/install/windows_warning.py", line 26, in wrapper
    return func(self, *args, **kwargs)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/prefs.py", line 74, in open_and_close_config
    return func(*args, **kwargs)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/admin.py", line 1423, in diagnostics
    p = self.ctx.popen(self._cmd("-e", "application list"))  # popen
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1409, in popen
    stdout=stdout, stderr=stderr)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
mmongy@ubuntu-server-test-2:/home/omero$ python --version
Python 2.7.15+
mmongy@ubuntu-server-test-2:/home/omero$ OMERO.server/bin/omero admin diagnostics

================================================================================
OMERO Diagnostics (admin) 5.5.1-ice36-b122
================================================================================
        
Commands:   java -version                  1.8.0     (/usr/bin/java)
Commands:   python -V                      2.7.15    (/usr/bin/python)
Commands:   icegridnode --version          not found
Commands:   icegridadmin --version         not found
Commands:   psql --version                 10.10     (/usr/bin/psql)
Commands:   openssl version                1.1.111   (/usr/bin/openssl)

No icegridadmin available: Cannot check server list

Log dir:    /home/omero/OMERO.server/var/log 
No logs available

Environment:OMERO_HOME=(unset)             
Environment:OMERO_NODE=(unset)             
Environment:OMERO_MASTER=(unset)           
Environment:OMERO_USERDIR=(unset)          
Environment:OMERO_TMPDIR=(unset)           
Environment:PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin 
Environment:PYTHONPATH=(unset)             
Environment:ICE_HOME=(unset)               
Environment:LD_LIBRARY_PATH=(unset)        
Environment:DYLD_LIBRARY_PATH=(unset)      

Traceback (most recent call last):
  File "OMERO.server/bin/omero", line 130, in <module>
    rv = omero.cli.argv()
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1706, in argv
    cli.invoke(args[1:])
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1159, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1236, in onecmd
    self.execute(line, previous_args)
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1318, in execute
    args.func(args)
  File "/home/omero/OMERO.server/lib/python/omero/install/windows_warning.py", line 26, in wrapper
    return func(self, *args, **kwargs)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/prefs.py", line 74, in open_and_close_config
    return func(*args, **kwargs)
  File "/home/omero/OMERO.server/lib/python/omero/plugins/admin.py", line 1423, in diagnostics
    p = self.ctx.popen(self._cmd("-e", "application list"))  # popen
  File "/home/omero/OMERO.server/lib/python/omero/cli.py", line 1409, in popen
    stdout=stdout, stderr=stderr)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

is the issue then. I would assume that these utilities would have been installed during this step:

https://docs.openmicroscopy.org/omero/5.5.1/sysadmins/unix/server-ubuntu1804-ice36.html#installing-prerequisites

Can you check if they’ve been installed and if your $PATH is correctly set?
~Josh

Hi, Josh.

I retried every step of the Ice 3.6.4 installation, and i checked if every step worked properly.
I noticed the steps:

echo /opt/Ice-3.6.4/lib/x86_64-linux-gnu > /etc/ld.so.conf.d/ice-x86_64.conf

and

cat omero-ice36.env >> /etc/profile

didn’t worked properly. I had to manually create the files with ubuntu’s GUI. In the installation tutorial, it was indicated for the steps to be run as root, so i used the “sudo” command, for the first time, but it didn’t worked as expected.

So, the “OSError” problem is solved, but now, i am stuck with the server waiting on startup.

Anyway, thanks for the advice.

Here’s the output of the server starting command line and the diagnostics:

mmongy@ubuntu-server-test-2:/home/omero$ OMERO.server/bin/omero admin start
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
.............................
Failed to startup some components after 300 seconds
Calling "stop" on remaining components
Waiting on shutdown. Use CTRL-C to exit
....mmongy@ubuntu-server-test-2:/home/omero$ OMERO.server/bin/omero admin diagnostics

================================================================================
OMERO Diagnostics (admin) 5.5.1-ice36-b122
================================================================================
        
Commands:   java -version                  1.8.0     (/usr/bin/java)
Commands:   python -V                      2.7.15    (/usr/bin/python)
Commands:   icegridnode --version          3.6.4     (/opt/Ice-3.6.4/bin/icegridnode)
Commands:   icegridadmin --version         3.6.4     (/opt/Ice-3.6.4/bin/icegridadmin)
Commands:   psql --version                 10.10     (/usr/bin/psql)
Commands:   openssl version                1.1.111   (/usr/bin/openssl)

Server:     icegridnode                    not started

Log dir:    /home/omero/OMERO.server/var/log exists
Log files:  Blitz-0.log                    438.0 KB      errors=16   warnings=48  
Log files:  DropBox.log                    18.0 KB       errors=24   warnings=12  
Log files:  FileServer.log                 682 B        
Log files:  Indexer-0.log                  348.0 KB      errors=16   warnings=16  
Log files:  MonitorServer.log              694 B        
Log files:  PixelData-0.log                348.0 KB      errors=16   warnings=16  
Log files:  Processor-0.log                30.0 KB       errors=9    warnings=18  
Log files:  Tables-0.log                   33.0 KB       errors=9    warnings=18  
Log files:  TestDropBox.log                n/a
Log files:  master.err                     491.0 KB      errors=13   warnings=6   
Log files:  master.out                     empty
Log files:  Total size                     1.71 MB


Environment:OMERO_HOME=(unset)             
Environment:OMERO_NODE=(unset)             
Environment:OMERO_MASTER=(unset)           
Environment:OMERO_USERDIR=(unset)          
Environment:OMERO_TMPDIR=(unset)           
Environment:PATH=/opt/Ice-3.6.4/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin 
Environment:PYTHONPATH=(unset)             
Environment:ICE_HOME=/opt/Ice-3.6.4        
Environment:LD_LIBRARY_PATH=(unset)        
Environment:DYLD_LIBRARY_PATH=(unset)      

Cannot list deployed applications.
OMERO data dir:'/OMERO'                       Exists? True	Is writable? False
OMERO temp dir:'/home/mmongy/omero/tmp'       Exists? True	Is writable? True   (Size: 0)

JVM settings: Blitz-${index}                -Xmx620m -XX:MaxPermSize=512m -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer-${index}              -Xmx413m -XX:MaxPermSize=512m -XX:+IgnoreUnrecognizedVMOptions
JVM settings: PixelData-${index}            -Xmx620m -XX:MaxPermSize=512m -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository-${index}           -Xmx413m -XX:MaxPermSize=512m -XX:+IgnoreUnrecognizedVMOptions

You can check in var/log/Blitz-0.log:

but I assume the issue is that your repository is not writeable:

~Josh


P.S. I realized a bit late that I should have also directed you to this post – OMERO / Python 3 - 5.6 milestone for upgrade testing – in case the impending upgrade wasn’t yet on your radar. Sorry about that.

I used “sudo chmod 777 OMERO” to modify the rights, and this repository is now writable, but the server still fails to startup some components.

Shall i upload you the “Blitz-0.log” file? I started a new one to only have the matchings for a new server start, but after examining it, i’m still lost.

Sure. That’s a good place to start.
~Josh

Blitz-0.log.zip (3.5 KB)

Here it is. I had to zip it. Have a nice day.

Thanks, @mmongy . Oddly enough it doesn’t help. Something happened even before the Blitz-0 service. Can you also upload master.err?

~Josh

P.S. before I forget, you’ll eventually want to adjust the permissions on /OMERO to not be writeable by everyone.

master.err.zip (21.9 KB)

Here’s the “master.err” file. Thanks for the advice about the permissions. However, for now, i’m just performing installation tests (and hopefully, execution tests) of OMERO on virtual machine (Ubuntu Server 18.04, on VirtualBox), so it will be useful on the real server.

1 Like

Hmmm… I’m not seeing anything in master.err either. Can you share your configuration?

bin/omero config get

~Josh

Here it is:

mmongy@ubuntu-server-test-2:/home/omero/OMERO.server-5.5.1-ice36-b122/bin$ ./omero config get
omero.data.dir=/OMERO
omero.db.name=omero_database
omero.db.pass=********
omero.db.user=db_user
omero.glacier2.IceSSL.Ciphers=HIGH:ADH:@SECLEVEL=0
omero.web.application_server=wsgi-tcp

Your installation looks a bit weird with respect to the system users. It looks like you created the ‘omero’ user and installed the server under /home/omero. But your OMERO temp dir is ‘/home/mmongy/omero/tmp’ and you try to run the server as user mmongy. Who owns the ‘/home/omero/OMERO.server-5.5.1-ice36-b122’ directory? I wonder if there are just some permission issues because of that mix up. What happens if you switch to the omero user and try to start the server? You probably have to sudo su first, then su omero, then run bash --init-file /etc/profile.

Regards,
Dominik

Hi, Dominik.

After switching to the omero user and trying to start the server, i get that:

omero@ubuntu-server-test-2:~/OMERO.server-5.5.1-ice36-b122/bin$ ./omero admin start
/home/omero/OMERO.server-5.5.1-ice36-b122/etc/grid/config.xml read-only
No descriptor given. Using etc/grid/default.xml
!! 12/19/19 14:39:39.200 icegridnode: error: service caught unhandled std::exception:
Instance.cpp:1098: Ice::FileException:
file exception: Permission denied
path: var/log/master.out
icegridnode: failure occurred in daemon:
service caught unhandled std::exception:
Instance.cpp:1098: Ice::FileException:
file exception: Permission denied
path: var/log/master.out

About the “weird” installation, i just followed the guidelines of:
https://docs.openmicroscopy.org/omero/5.5.1/sysadmins/unix/server-ubuntu1804-ice36.html

Thanks anyway, Marc.

My suspicion now is that you installed the OMERO.server as another user than ‘omero’ under /home/omero. Could you try to change the owner of /home/omeroOMERO.server to the user ‘omero’? So as root run chown -R omero:omero /home/omero/OMERO.server*

Regards,
Dominik

After changing the owner of /home/omero/OMERO.server*, i tried to start the server with both users: “mmongy”, and “omero”

mmongy@ubuntu-server-test-2:/home/omero/OMERO.server-5.5.1-ice36-b122/bin$ ./omero admin start
/home/omero/OMERO.server-5.5.1-ice36-b122/etc/grid/config.xml read-only
No descriptor given. Using etc/grid/default.xml
!! 12/19/19 15:22:29.898 icegridnode: error: service caught unhandled std::exception:
   Instance.cpp:1098: Ice::FileException:
   file exception: Permission denied
   path: var/log/master.out
icegridnode: failure occurred in daemon:
service caught unhandled std::exception:
Instance.cpp:1098: Ice::FileException:
file exception: Permission denied
path: var/log/master.out
mmongy@ubuntu-server-test-2:/home/omero/OMERO.server-5.5.1-ice36-b122/bin$ sudo su
[sudo] password for mmongy: 
root@ubuntu-server-test-2:/home/omero/OMERO.server-5.5.1-ice36-b122/bin# su omero
$ ls
omero  omero.bat  setpythonpath.bat  winconfig.bat
$ omero admin start   
sh: 2: omero: not found
$ bash --init-file /etc/profile

omero@ubuntu-server-test-2:~/OMERO.server-5.5.1-ice36-b122/bin$ ./omero admin start
/home/omero/OMERO.server-5.5.1-ice36-b122/etc/grid/config.xml read-only
No descriptor given. Using etc/grid/default.xml
Waiting on startup. Use CTRL-C to exit
.............................
Failed to startup some components after 300 seconds
Calling "stop" on remaining components
Waiting on shutdown. Use CTRL-C to exit
......omero@ubuntu-server-test-2:~/OMERO.server-5.5.1-ice36-b122/bin$

I’m running out of ideas, to be honest. I think the best would probably be to start again from scratch, paying particular attention to which user (root or omero) has to be used for each step. Or as Josh already mentioned go straigt for the 5.6 installation which will be rolled out very soon anyway. You can find the docs for that here: https://docs.openmicroscopy.org/omero/5.6
Regards,
Dominik

Hi, Dominik.

In fact, i already tried the Omero Server 5.6 installation on Ubuntu Server 18.04, on VirtualBox. I have been careful about the users for each step, but i’m still stuck in the same problems when i’m trying to start the server (still fails to startup after 300 secs)

So i tried to figure if there was any other problems with my VirtualBox machine. After some research, i found this:

https://docs.openmicroscopy.org/omero/5.2.8/users/virtual-appliance.html

I configured the advanced network settings as mentioned, but it didn’t solved anything.

Finally, I tried a pre-made, “out-of-box” VirtualBox machine of Omero Server (it’s an old version, but i wanted to try it at least to figure how Omero server is supposed to work, and eventually to compare the installation with mine to figure where i made errors):

https://downloads.openmicroscopy.org/omero-virtual-appliance/5.2.1/

There were no user problems, but it still failed at starting:

VirtualBox_OMERO.va-5.2.1-b9_20_12_2019_15_37_05