How to install OMERO.web add-on apps with Docker

I’ve followed the instructions here to install OMERO.server and OMERO.web with Docker, but am now realizing that some functionality is missing (e.g., figure export). Is it possible to amend my Docker files to include the web add-ons, like as was done here with Ansible, or will I have to install them manually with Pip? Are either Docker or Ansible the preferred “easy” way to install and manage OMERO, or are both relatively equivalent in that regard? I have minimal experience with Docker and none with Ansible.

Hi Jonathan - this discussion is related:

In particlular see @manics answer and pointers to the doc:

1 Like

Sorry, we’ve gone back and forth on including figure in the base containers (e.g. Auto import, figure, wheel, etc. by joshmoore · Pull Request #21 · ome/omero-server-docker · GitHub) versus making the dockers extendable (e.g. Add infrastructure for a server extras image by manics · Pull Request #39 · ome/omero-server-docker · GitHub). The more we add to the dockers the more they have to be updated, etc. Feedback is certainly welcome. Note that here the export functionality is needed in the server docker rather than the web docker making this all the weirder.

Yes. omero-server-docker also uses ansible internally, so you could add another playbook to your server/Dockerfile and then run it with ansible-playbook.

Installing them manually in the server/Dockerfile is certainly also an option.

Not sure how to gauge which is “easiest”. That likely depends on your familiarity, and perhaps also with what infrastructure you have. E.g. if you have bare metal then likely using ansible directly is easiest. If someone has given you a managed docker host, then docker will likely be easier.

We use ansible for most of our production systems, but we are looking to move some of them to docker.

We certainly try to keep them functionally equivalent.
~Josh

Thanks, Guillaume! I appreciate the help.

Thanks for the detailed response, Josh. I can understand with such a big project that it would be difficult to keep things flexible and lightweight while maintaining full functionality. I would love to see a “batteries included” Docker/Ansible setup that includes the figure add-ons. While the OMERO documentation is extensive, it can be a bit daunting, at times, and figuring out the best or easiest way to do something is not always obvious. Along those same lines, it would be fantastic to have some “best practices” for things like setting up user groups and data annotation, particularly with some examples of standard groups/tags/key value pairs that many will find useful.

1 Like

Understood. Until there’s a full product, you could take a look at our production playbooks, e.g. prod-playbooks/playbook.yml at 79ea70cda08e3187803606601b28964648dda2a2 · ome/prod-playbooks · GitHub, where figure is installed.

~Josh

Thanks, I didn’t realize that there were production playbooks available. I think I will redo my test server with Ansible, since it seems like the most detailed examples are done that way.

In trying to use the ansible-addons configuration, since it seems to at least have the figure export enabled, but I seem to be getting an error.

[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
ERROR! couldn't resolve module/action 'postgresql_user'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/ducatlab/.ansible/roles/ome.postgresql/tasks/databases.yml': line 22, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: postgres | create users
  ^ here

I set /etc/ansible/hosts with

[omero-addons]
127.0.0.1

and installed ome-molecule via pip.

The only change I made to the playbook file was to change the default password. Is there some prerequisite step I missed?

What version of Ansible are you using (ansible --version)?

2.6.18. There seems to be something wrong with my installation, I think.

ansible 2.6.18
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/ducatlab/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ducatlab/.local/lib/python3.6/site-packages/ansible
  executable location = /home/ducatlab/.local/bin/ansible
  python version = 3.6.9 (default, Oct  8 2020, 12:12:24) [GCC 8.4.0]
Traceback (most recent call last):
  File "/home/ducatlab/.local/bin/ansible", line 117, in <module>
    cli.parse()
  File "/home/ducatlab/.local/lib/python3.6/site-packages/ansible/cli/adhoc.py", line 76, in parse
    super(AdHocCLI, self).parse()
  File "/home/ducatlab/.local/lib/python3.6/site-packages/ansible/cli/__init__.py", line 585, in parse
    self.options, self.args = self.parser.parse_args(self.args[1:])
  File "/usr/lib/python3.6/optparse.py", line 1387, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/usr/lib/python3.6/optparse.py", line 1427, in _process_args
    self._process_long_opt(rargs, values)
  File "/usr/lib/python3.6/optparse.py", line 1501, in _process_long_opt
    option.process(opt, value, values, self)
  File "/usr/lib/python3.6/optparse.py", line 785, in process
    self.action, self.dest, opt, value, values, parser)
  File "/usr/lib/python3.6/optparse.py", line 811, in take_action
    parser.exit()
  File "/usr/lib/python3.6/optparse.py", line 1559, in exit
    sys.exit(status)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ducatlab/.local/bin/ansible", line 161, in <module>
    shutil.rmtree(C.DEFAULT_LOCAL_TMP, True)
AttributeError: module 'ansible.constants' has no attribute 'DEFAULT_LOCAL_TMP'

This is on a bare metal system with Ubuntu 18.04.

Ansible 2.6 should be fine, that’s the version used for the repository CI tests that validate the addons playbook works.

Maybe try creating a new virtualenv just for Ansible?

Simon,

I created a virtual environment, reinstalled ansible and ome-ansible-molecule. I get no errors after running the ansible-galaxy install -r requirements.yml command.

I edited inventory.yml with either 127.0.0.1 or the IP of the host, but I get the same warning with both:

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

Is there another configuration step I missed?

Edit: I was able to run the playbook by editing /etc/ansible/ansible.cfg with

[defaults]
transport = local

Could you show us the full ansible-playbook command that you’re running?

What does

ansible-playbook -i inventory.yml --list-hosts <playbook>.yml

show?

Simon,

I think something is wrong with my Ansible installation. I had previously installed Ansible with apt, but uninstalled and then reinstalled in a virtual environment as you suggested. Below is the output of the playbook command.

(ome) root@PRL-DUCAT-S1:/mnt/raid1/ansible-example-omero-addons# ansible-playbook -i inventory.yml --list-hosts playbook.yml
/opt/ansible/ome/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography.exceptions import InvalidSignature

playbook: playbook.yml

  play #1 (omero_addons): omero_addons  TAGS: []
    pattern: [u'omero_addons']
    hosts (1):
      omero_addons

But now if I try running the playbook, it fails

PLAYBOOK: playbook.yml **********************************************************************************************************************************************************************
1 plays in playbook.yml

PLAY [omero_addons] *************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************
task path: /mnt/raid1/ansible-example-omero-addons/playbook.yml:5
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1610649699.45-186063794343094 `" && echo ansible-tmp-1610649699.45-186063794343094="` echo /root/.ansible/tmp/ansible-tmp-1610649699.45-186063794343094 `" ) && sleep 0'
Using module file /opt/ansible/ome/lib/python2.7/site-packages/ansible/modules/setup.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-3891D8TwPt/tmpIIXsl8 TO /root/.ansible/tmp/ansible-tmp-1610649699.45-186063794343094/setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1610649699.45-186063794343094/ /root/.ansible/tmp/ansible-tmp-1610649699.45-186063794343094/setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1610649699.45-186063794343094/setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1610649699.45-186063794343094/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_boDAj_/ansible_module_setup.py", line 130, in <module>
    from ..module_utils.basic import AnsibleModule
ValueError: Attempted relative import in non-package

fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_boDAj_/ansible_module_setup.py\", line 130, in <module>\n    from ..module_utils.basic import AnsibleModule\nValueError: Attempted relative import in non-package\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}
        to retry, use: --limit @/mnt/raid1/ansible-example-omero-addons/playbook.retry

PLAY RECAP **********************************************************************************************************************************************************************************
127.0.0.1                  : ok=0    changed=0    unreachable=0    failed=1

I’m not very familiar with Ansible, so I apologize if I’ve missed something obvious.

Ah, ok. This is looking very much like Python 2 troubles. Do you have any options for upgrading to Python 3?

~Josh

Josh,

I think you were right about Python 2 being an issue, but it doesn’t seem to solve my problem.

TASK [Gathering Facts] ******************************************************************************************************************************************************************
task path: /mnt/raid1/ansible-example-omero-addons/playbook.yml:5
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: ducatlab
<localhost> EXEC /bin/sh -c 'echo ~ducatlab && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ducatlab/.ansible/tmp/ansible-tmp-1610738482.7650425-69105106433365 `" && echo ansible-tmp-1610738482.7650425-69105106433365="` echo /home/ducatlab/.ansible/tmp/ansible-tmp-1610738482.7650425-69105106433365 `" ) && sleep 0'
Using module file /home/ducatlab/ansible/omero/lib/python3.6/site-packages/ansible/modules/setup.py
<localhost> PUT /home/ducatlab/.ansible/tmp/ansible-local-29678rxvpepb0/tmp67bm54qp TO /home/ducatlab/.ansible/tmp/ansible-tmp-1610738482.7650425-69105106433365/setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/ducatlab/.ansible/tmp/ansible-tmp-1610738482.7650425-69105106433365/ /home/ducatlab/.ansible/tmp/ansible-tmp-1610738482.7650425-69105106433365/setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/ducatlab/.ansible/tmp/ansible-tmp-1610738482.7650425-69105106433365/setup.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/ducatlab/.ansible/tmp/ansible-tmp-1610738482.7650425-69105106433365/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_7xIn1T/ansible_module_setup.py", line 130, in <module>
    from ..module_utils.basic import AnsibleModule
ValueError: Attempted relative import in non-package

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_7xIn1T/ansible_module_setup.py\", line 130, in <module>\n    from ..module_utils.basic import AnsibleModule\nValueError: Attempted relative import in non-package\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}

It still fails at ansible_module setup.py. I was previously running everything as root, but I get the same result as a user. I’ve tried uninstalling and reinstalling the ansible galaxy requirements, ansible, ome-ansible-molecule, but the error is the same.

Hey Jonathan,

Sorry for the delay. None of us have seen this error before and we’re at something of a loss. Best guess at the moment is that some combination of your attempts to get things running may have left conflicting libraries. Did you reload the requirements after moving to Python 3? (I wouldn’t have thought that was necessary.)

~Josh

Josh,

I did reload the requirements after switching to Python 3, but that didn’t seem to help. I’m going to try again on a fresh copy of Ubuntu 18.04. Is it still recommended to install ansible and ome-ansible-molecule via pip? I noticed that the instructions here make no mention of ome-ansible-molecule.

Jonathan

@jsakkos ome-ansible-molecule is a meta-package that installs Ansible as well as the Molecule test dependencies for deploying and testing OME Ansible roles and playbooks as well as Docker compose.

If you are only interested in deployment via Ansible, either installing Ansible via pip or

pip install ome-ansible-molecule

should work. The only downside of the latter is that it will bring a few additional dependencies to your Python environment.

Another suggestion is to install your galaxy roles to a local ./roles directory instead of the default shared directory (/home/ducatlab/.ansible/roles from your earlier logs):

ansible-galaxy install -p roles -r requirements.yml

This makes it easier to keep track of the role versions.

Thanks for the help @s.besson and @manics. I noticed that installing ansible via pip required adding its directory to PATH for it to be used. Also, the pip install of Ansible seems to use some Python 2. I removed and reinstalled ansible via apt and molecule via pip python3 -m pip install "molecule[ansible]".

It seemed to install (ansible-example-omero-addons) without errors and I can see omero and postgres running with top, but I can’t seem to access via the webclient.

sudo lsof -i -P -n

COMMAND     PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

postgres  18877        postgres    5u  IPv4 227081      0t0  TCP 127.0.0.1:5432 (LISTEN)
postgres  18877        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  18881        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  18882        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  18883        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  18884        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  18885        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  18886        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
icegridno 19828    omero-server   16u  IPv4 231927      0t0  TCP 127.0.0.1:33781 (LISTEN)
icegridno 19828    omero-server   29u  IPv4 228300      0t0  TCP 127.0.0.1:42979 (LISTEN)
icegridno 19828    omero-server   33u  IPv4 228302      0t0  TCP 127.0.0.1:4061 (LISTEN)
icegridno 19828    omero-server   34u  IPv4 228303      0t0  UDP 239.255.0.1:4061
icegridno 19828    omero-server   41u  IPv4 228306      0t0  TCP 127.0.0.1:35867 (LISTEN)
java      19862    omero-server  222u  IPv6 228333      0t0  TCP 35.8.197.106:40311->35.8.197.106:45120 (ESTABLISHED)
java      19862    omero-server  226u  IPv6 233593      0t0  TCP 127.0.0.1:42043 (LISTEN)
java      19862    omero-server  232u  IPv6 233600      0t0  TCP *:40311 (LISTEN)
java      19862    omero-server  236u  IPv6 230176      0t0  TCP 127.0.0.1:42988->127.0.0.1:5432 (ESTABLISHED)
java      19862    omero-server  249u  IPv6 234591      0t0  TCP 35.8.197.106:40311->35.8.197.106:45122 (ESTABLISHED)
python    19890    omero-server    7u  IPv4 225114      0t0  TCP 127.0.0.1:42471 (LISTEN)
python    19890    omero-server   22u  IPv6 226293      0t0  TCP *:46453 (LISTEN)
python    19894    omero-server    6u  IPv4 225107      0t0  TCP 127.0.0.1:44349 (LISTEN)
python    19894    omero-server   14u  IPv6 223992      0t0  TCP *:33125 (LISTEN)
java      19895    omero-server  216u  IPv6 230986      0t0  TCP 127.0.0.1:43020->127.0.0.1:5432 (ESTABLISHED)
java      19895    omero-server  224u  IPv6 232902      0t0  TCP 127.0.0.1:42819 (LISTEN)
java      19895    omero-server  231u  IPv6 232044      0t0  TCP *:36995 (LISTEN)
python    19899    omero-server    6u  IPv4 231949      0t0  TCP 127.0.0.1:34449 (LISTEN)
python    19899    omero-server   14u  IPv6 231953      0t0  TCP *:38809 (LISTEN)
glacier2r 19900    omero-server    8u  IPv4 229080      0t0  TCP 127.0.0.1:40085 (LISTEN)
glacier2r 19900    omero-server   15u  IPv6 229084      0t0  TCP *:4064 (LISTEN)
glacier2r 19900    omero-server   16u  IPv6 229085      0t0  TCP *:4063 (LISTEN)
glacier2r 19900    omero-server   17u  IPv4 229088      0t0  TCP 127.0.0.1:45935 (LISTEN)
java      19901    omero-server  216u  IPv6 232039      0t0  TCP 127.0.0.1:43022->127.0.0.1:5432 (ESTABLISHED)
java      19901    omero-server  224u  IPv6 230989      0t0  TCP 127.0.0.1:33941 (LISTEN)
java      19901    omero-server  230u  IPv6 230996      0t0  TCP *:42791 (LISTEN)
icebox    19903    omero-server    9u  IPv6 226268      0t0  TCP *:44559 (LISTEN)
icebox    19903    omero-server   13u  IPv6 226271      0t0  TCP *:35003 (LISTEN)
icebox    19903    omero-server   22u  IPv4 230131      0t0  TCP 127.0.0.1:38021 (LISTEN)
python    19904    omero-server    7u  IPv4 232724      0t0  TCP 127.0.0.1:34929 (LISTEN)
python    19904    omero-server   15u  IPv4 232036      0t0  TCP 35.8.197.106:45120->35.8.197.106:40311 (ESTABLISHED)
python    19904    omero-server   16u  IPv6 232894      0t0  TCP *:36715 (LISTEN)
python    19906    omero-server    6u  IPv4 229097      0t0  TCP 127.0.0.1:46411 (LISTEN)
python    19906    omero-server   14u  IPv4 229176      0t0  TCP 35.8.197.106:45122->35.8.197.106:40311 (ESTABLISHED)
python    19906    omero-server   15u  IPv6 232896      0t0  TCP *:43897 (LISTEN)
postgres  20274        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  20274        postgres   11u  IPv4 229146      0t0  TCP 127.0.0.1:5432->127.0.0.1:42988 (ESTABLISHED)
postgres  20497        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  20497        postgres   11u  IPv4 233696      0t0  TCP 127.0.0.1:5432->127.0.0.1:43020 (ESTABLISHED)
postgres  20500        postgres   10u  IPv4 227088      0t0  UDP 127.0.0.1:36144->127.0.0.1:36144
postgres  20500        postgres   11u  IPv4 233700      0t0  TCP 127.0.0.1:5432->127.0.0.1:43022 (ESTABLISHED)
gunicorn  23839       omero-web    5u  IPv4 244583      0t0  TCP 127.0.0.1:4080 (LISTEN)
gunicorn  23842       omero-web    5u  IPv4 244583      0t0  TCP 127.0.0.1:4080 (LISTEN)
gunicorn  23872       omero-web    5u  IPv4 244583      0t0  TCP 127.0.0.1:4080 (LISTEN)
gunicorn  23874       omero-web    5u  IPv4 244583      0t0  TCP 127.0.0.1:4080 (LISTEN)
gunicorn  23881       omero-web    5u  IPv4 244583      0t0  TCP 127.0.0.1:4080 (LISTEN)
gunicorn  23882       omero-web    5u  IPv4 244583      0t0  TCP 127.0.0.1:4080 (LISTEN)
redis-ser 24172           redis    6u  IPv4 491373      0t0  TCP 127.0.0.1:6379 (LISTEN)
redis-ser 24172           redis    7u  IPv6 491374      0t0  TCP [::1]:6379 (LISTEN)
nginx     24316            root    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24317        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24318        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24319        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24320        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24321        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24322        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24323        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24324        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24325        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24326        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24327        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
nginx     24328        www-data    6u  IPv4 253265      0t0  TCP *:80 (LISTEN)
cupsd     30278            root    6u  IPv6 293144      0t0  TCP [::1]:631 (LISTEN)
cupsd     30278            root    7u  IPv4 293145      0t0  TCP 127.0.0.1:631 (LISTEN)
cups-brow 30279            root    7u  IPv4 294236      0t0  UDP *:631

UFW is off and iptables had no rules setup by default, so I don’t think it’s a firewall issue, but it’s possible. I am able to access via OMERO.insight.