Setting up an environment to run local scripts

Hi,
I’m trying to set up an environment to work on some OMERO scripts now that we have a lot of time and I struggle to get things working. I’d need some help here.

I believe it must be an issue with the way I setup my environment. In particular, I’m confused about the need to have a copy of the OMERO server and omero-py…
So here is what I have:

  • MacOS Catalina
  • venv running python 3.7 with (for what I think is relevant): omero-py=5.6.2, omero-rois=0.3.0 and zeroc-ice=3.6.5
  • In my working directory a copy of the server: OMERO.server-5.6.0-ice36-b136

So I tried to use the demo.openmicroscopy.org server and the ping script from https://docs.openmicroscopy.org/omero/5.6.0/developers/scripts/user-guide.html.

Here is what I did and the traceback

Julios-iMac:OMERO.metrics julio$ source venv/bin/activate
(venv) Julios-iMac:OMERO.metrics julio$ omero -s demo.openmicroscopy.org -u jmateos script serve user
Password:
Created session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
2020-03-24 10:23:16,470 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-03-24 10:23:16,470 INFO  [              omero.processor.ProcessorI] (MainThread) Registering processor ",3tnr3D5!CIg7:K&|qti/omero.scripts.serve-0724d8c1-6553-45d4-9d23-7635928d2d81" -t -e 1.1:tcp -h 127.0.0.1 -p 35059 -t 60000
Press any key to exit...

on another terminal window:

Julios-iMac:OMERO.metrics julio$ source venv/bin/activate
(venv) Julios-iMac:OMERO.metrics julio$ cd OMERO.server-5.6.0-ice36-b136/
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ ls # You can see the ping script copy paste from the site
LICENSE.txt	history.rst	ping.py		sql
etc		lib		share		var
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script list user
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
 id | Scripts for user 
----+------------------
(0 rows)
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script upload ping.py 
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
Uploaded script
OriginalFile:470104
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script list user
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
 id     | Scripts for user 
--------+------------------
 470104 | ping.py          
(1 row)
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script launch 470104
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
Traceback (most recent call last):
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/bin/omero", line 118, in <module>
    rv = omero.cli.argv()
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1754, in argv
    cli.invoke(args[1:])
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1187, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1264, in onecmd
    self.execute(line, previous_args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1346, in execute
    args.func(args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/plugins/script.py", line 411, in launch
    params = svc.getParams(script_id)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero_api_IScript_ice.py", line 842, in getParams
    return _M_omero.api.IScript._op_getParams.invoke(self, ((scriptID, ), _ctx))
omero.InternalException: exception ::omero::InternalException
{
    serverStackTrace = Traceback (most recent call last):
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/util/decorators.py", line 69, in exc_handler
    rv = func(*args, **kwargs)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/processor.py", line 917, in parseJob
    client = self.user_client("OMERO.parseJob")
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/processor.py", line 800, in user_client
    client = omero.client(args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/__init__.py", line 67, in client
    return omero.clients.BaseClient(*args, **kwargs)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/clients.py", line 155, in __init__
    id.properties = Ice.createProperties(args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/Ice.py", line 942, in createProperties
    properties = IcePy.createProperties(args, defaults)
Ice.FileException: exception ::Ice::FileException
{
    error = 2
    path = /Users/julio/PycharmProjects/OMERO.metrics/venv/lib/etc/ice.config
}

    serverExceptionClass = 
    message = Internal exception
}

Hi @juliomateos,

can you try adding one step to set OMERODIR in the second terminal?

source venv/bin/activate
cd OMERO.server-5.6.0-ice36-b136/
export OMERODIR=$PWD  # Add this
... continue ...

~Josh

Same thing. I also reconnected in the first terminal using that variable. This is the output of the first terminal:

(venv) Julios-iMac:OMERO.metrics julio$ export OMERODIR=$PWD
(venv) Julios-iMac:OMERO.metrics julio$ omero -s demo.openmicroscopy.org -u jmateos script serve user
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
2020-03-24 12:43:45,737 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-03-24 12:43:45,737 INFO  [              omero.processor.ProcessorI] (MainThread) Registering processor 94oj3yy$4~R\/9%JksGDQ/omero.scripts.serve-a1f0fb17-1354-4103-8759-4e20ecf092ec -t -e 1.1:tcp -h 127.0.0.1 -p 35059 -t 60000
Press any key to exit...
2020-03-24 12:44:13,816 INFO  [                            omero.remote] (Dummy-3   )  Meth: ProcessorI.willAccept
2020-03-24 12:44:14,453 INFO  [                            omero.remote] (Dummy-3   )  Rslt: True
2020-03-24 12:44:14,568 INFO  [                            omero.remote] (Dummy-4   )  Meth: ProcessorI.parseJob
2020-03-24 12:44:14,568 INFO  [              omero.processor.ProcessorI] (Dummy-4   ) parseJob: Session = de696b6f-877f-45a9-a2a6-0d03190cccee, JobId = 437338
2020-03-24 12:44:14,569 INFO  [                            omero.remote] (Dummy-4   )  Excp: exception ::Ice::FileException
{
    error = 2
    path = /Users/julio/PycharmProjects/OMERO.metrics/etc/ice.config
}
2020-03-24 12:44:14,569 WARNI [                            omero.remote] (Dummy-4   ) <function ProcessorI.parseJob at 0x1107d7320> raised a non-ServerError (<class 'Ice.FileException'>): exception ::Ice::FileException
{
    error = 2
    path = /Users/julio/PycharmProjects/OMERO.metrics/etc/ice.config
}

and the second. I deleted an rey-loaded the script

(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script delete 470104
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script upload ping.py 
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
Uploaded script
OriginalFile:470123
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script list user
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
 id     | Scripts for user 
--------+------------------
 470123 | ping.py          
(1 row)
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script launch 470123
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
Traceback (most recent call last):
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/bin/omero", line 118, in <module>
    rv = omero.cli.argv()
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1754, in argv
    cli.invoke(args[1:])
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1187, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1264, in onecmd
    self.execute(line, previous_args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/cli.py", line 1346, in execute
    args.func(args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/plugins/script.py", line 411, in launch
    params = svc.getParams(script_id)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero_api_IScript_ice.py", line 842, in getParams
    return _M_omero.api.IScript._op_getParams.invoke(self, ((scriptID, ), _ctx))
omero.InternalException: exception ::omero::InternalException
{
    serverStackTrace = Traceback (most recent call last):
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/util/decorators.py", line 69, in exc_handler
    rv = func(*args, **kwargs)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/processor.py", line 917, in parseJob
    client = self.user_client("OMERO.parseJob")
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/processor.py", line 800, in user_client
    client = omero.client(args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/__init__.py", line 67, in client
    return omero.clients.BaseClient(*args, **kwargs)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/omero/clients.py", line 155, in __init__
    id.properties = Ice.createProperties(args)
  File "/Users/julio/PycharmProjects/OMERO.metrics/venv/lib/python3.7/site-packages/Ice.py", line 942, in createProperties
    properties = IcePy.createProperties(args, defaults)
Ice.FileException: exception ::Ice::FileException
{
    error = 2
    path = /Users/julio/PycharmProjects/OMERO.metrics/venv/lib/etc/ice.config
}

    serverExceptionClass = 
    message = Internal exception
}
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ echo $OMERODIR 
/Users/julio/PycharmProjects/OMERO.metrics/OMERO.server-5.6.0-ice36-b136
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ 

Hi Julio. Sorry, being within the OMERO.server-5.6.0-ice36-b136/ directory was an important part of what I was suggesting. In the end OMERODIR should equal:

/Users/julio/PycharmProjects/OMERO.metrics/OMERO.server-5.6.0-ice36-b136

~Josh

Ah yes! obviously. It’s me apologising. Thank you.

Now it’s running! The output is not what I expect, though. It does not seem to find the ‘keys’ And it throws some kind of communication not destroyed error

Julios-iMac:OMERO.metrics julio$ source venv/bin/activate
(venv) Julios-iMac:OMERO.metrics julio$ cd OMERO.server-5.6.0-ice36-b136/
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ export OMERODIR=$PWD
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script list user
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
 id     | Scripts for user 
--------+------------------
 470123 | ping.py          
(1 row)
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script launch 470123
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
Job 437353 ready
Waiting....
Callback received: FINISHED

	*** start stdout (id=470147)***
	* b'Keys found:\n[]\n'
	*** end stdout ***


	*** start stderr (id=470148)***
	* b'!! 03/24/20 20:46:02.107 error: communicator not destroyed during global destruction.'
	*** end stderr ***


	*** out parameters ***
	***  done ***

The log on the first terminal follows here:

Julios-iMac:OMERO.metrics julio$ source venv/bin/activate
(venv) Julios-iMac:OMERO.metrics julio$ cd OMERO.server-5.6.0-ice36-b136/
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ export OMERODIR=$PWD
(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero -s demo.openmicroscopy.org -u jmateos script serve user
Using session for jmateos@demo.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: 2018-10
2020-03-24 20:44:08,281 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2020-03-24 20:44:08,282 INFO  [              omero.processor.ProcessorI] (MainThread) Registering processor "+86nlGn1\"rSqYEHk=Q):/omero.scripts.serve-05f78e27-1c9c-439e-9383-24f61563c954" -t -e 1.1:tcp -h 127.0.0.1 -p 35059 -t 60000
Press any key to exit...
2020-03-24 20:45:58,099 INFO  [                            omero.remote] (Dummy-3   )  Meth: ProcessorI.willAccept
2020-03-24 20:45:58,866 INFO  [                            omero.remote] (Dummy-3   )  Rslt: True
2020-03-24 20:45:58,979 INFO  [                            omero.remote] (Dummy-4   )  Meth: ProcessorI.processJob
2020-03-24 20:45:58,979 INFO  [              omero.processor.ProcessorI] (Dummy-4   ) processJob: Session = 77e04fa6-a2f8-498a-80e7-215c814e0ca4, JobId = 437353
2020-03-24 20:46:00,389 INFO  [              omero.processor.ProcessorI] (Dummy-4   ) Using launcher: /Users/julio/PycharmProjects/OMERO.metrics/venv/bin/python
2020-03-24 20:46:00,389 INFO  [              omero.processor.ProcessorI] (Dummy-4   ) Using process: 
2020-03-24 20:46:00,402 INFO  [                omero.processor.ProcessI] (Dummy-4   ) Created 77e04fa6-a2f8-498a-80e7-215c814e0ca4 in /Users/julio/omero/tmp/omero_julio/69239/processkfvlhh4n.dir
2020-03-24 20:46:00,550 INFO  [              omero.processor.ProcessorI] (Dummy-4   ) Downloaded file: 470123
2020-03-24 20:46:00,555 INFO  [                omero.processor.ProcessI] (Dummy-4   ) <proc:69272,rc=-,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Activated
2020-03-24 20:46:00,837 INFO  [                            omero.remote] (Dummy-4   )  Rslt: "+86nlGn1\"rSqYEHk=Q):/Process-8248f109-0031-4763-8ac9-8be29402d5c6" -t -e 1.1:tcp -h 127.0.0.1 -p 35059 -t 60000
2020-03-24 20:46:00,904 INFO  [                            omero.remote] (Dummy-3   )  Meth: ProcessI.registerCallback
2020-03-24 20:46:00,904 INFO  [                            omero.remote] (Dummy-3   )  Rslt: None
2020-03-24 20:46:01,254 INFO  [                            omero.remote] (Dummy-5   )  Meth: ProcessI.registerCallback
2020-03-24 20:46:01,254 INFO  [                            omero.remote] (Dummy-5   )  Rslt: None
2020-03-24 20:46:01,395 INFO  [                            omero.remote] (Dummy-4   )  Meth: ProcessI.poll
2020-03-24 20:46:01,395 INFO  [                omero.processor.ProcessI] (Dummy-4   ) <proc:69272,rc=-,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Polling
2020-03-24 20:46:01,395 INFO  [                            omero.remote] (Dummy-4   )  Rslt: None
2020-03-24 20:46:01,531 INFO  [                            omero.remote] (Dummy-3   )  Meth: ProcessI.poll
2020-03-24 20:46:01,531 INFO  [                omero.processor.ProcessI] (Dummy-3   ) <proc:69272,rc=-,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Polling
2020-03-24 20:46:01,531 INFO  [                            omero.remote] (Dummy-3   )  Rslt: None
2020-03-24 20:46:02,670 INFO  [                            omero.remote] (Dummy-5   )  Meth: ProcessI.poll
2020-03-24 20:46:02,670 INFO  [                omero.processor.ProcessI] (Dummy-5   ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Polling
2020-03-24 20:46:02,670 INFO  [                omero.processor.ProcessI] (Dummy-5   ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Deactivating
2020-03-24 20:46:02,670 INFO  [                            omero.remote] (Dummy-5   )  Meth: ProcessI.shutdown
2020-03-24 20:46:02,670 INFO  [                            omero.remote] (Dummy-5   )  Rslt: None
2020-03-24 20:46:04,022 INFO  [                omero.processor.ProcessI] (Dummy-5   ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Changed job status from Running to Finished
2020-03-24 20:46:05,013 INFO  [                omero.processor.ProcessI] (Dummy-5   ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Uploaded 15 bytes of /Users/julio/omero/tmp/omero_julio/69239/processkfvlhh4n.dir/out to 470147
2020-03-24 20:46:05,804 INFO  [                omero.processor.ProcessI] (Dummy-5   ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Uploaded 85 bytes of /Users/julio/omero/tmp/omero_julio/69239/processkfvlhh4n.dir/err to 470148
2020-03-24 20:46:05,954 INFO  [                omero.processor.ProcessI] (Dummy-5   ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Lived 2s. Deactivation took 3s.
2020-03-24 20:46:05,954 INFO  [                omero.processor.ProcessI] (Dummy-5   ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Callback processFinished
2020-03-24 20:46:06,093 INFO  [                            omero.remote] (Dummy-5   )  Rslt: object #0 (::omero::RInt)
{
    _val = 0
}
2020-03-24 20:46:06,419 INFO  [                            omero.remote] (Dummy-4   )  Meth: ProcessI.poll
2020-03-24 20:46:06,419 INFO  [                            omero.remote] (Dummy-4   )  Rslt: object #0 (::omero::RInt)
{
    _val = 0
}
2020-03-24 20:46:06,563 INFO  [                            omero.remote] (Dummy-3   )  Meth: ProcessI.poll
2020-03-24 20:46:06,563 INFO  [                            omero.remote] (Dummy-3   )  Rslt: object #0 (::omero::RInt)
{
    _val = 0
}
2020-03-24 20:46:06,639 INFO  [                            omero.remote] (Dummy-5   )  Meth: ProcessI.poll
2020-03-24 20:46:06,639 INFO  [                            omero.remote] (Dummy-5   )  Rslt: object #0 (::omero::RInt)
{
    _val = 0
}
2020-03-24 20:46:08,356 INFO  [                            omero.remote] (Thread-2  )  Meth: ProcessI.poll
2020-03-24 20:46:08,356 INFO  [                            omero.remote] (Thread-2  )  Rslt: object #0 (::omero::RInt)
{
    _val = 0
}
2020-03-24 20:46:08,357 INFO  [                omero.processor.ProcessI] (Thread-2  ) <proc:69272,rc=0,uuid=77e04fa6-a2f8-498a-80e7-215c814e0ca4> : Keep alive failed

For the record this is the script:

import omero.scripts as scripts
client = scripts.client("ping.py", "simple ping script",
                        scripts.Long("a"), scripts.String("b"))

keys = client.getInputKeys()
print("Keys found:")
print(keys)
for key in keys:
    client.setOutput(key, client.getInput(key))

Hi, just figured out that (1) the parameters are optional and therefore not requested after running the script and (2) that the script is not closing the session (maybe to update in the doc). Now it runs beautifully.

Thanks!!

(venv) Julios-iMac:OMERO.server-5.6.0-ice36-b136 julio$ omero script launch 453 a=666 b=Hello
Using session for facility_staff_1@192.168.0.21:4064. Idle timeout: 10 min. Current group: metrics
Job 675 ready
Waiting....
Callback received: FINISHED

	*** start stdout (id=507)***
	* b"Keys found:\n['a', 'b']\n"
	*** end stdout ***


	*** out parameters ***
	* a=666
	* b=Hello
	***  done ***
1 Like

Glad it’s working. Thanks for letting us know.

Agreed. See https://github.com/ome/ome-documentation/issues/2113 but do note that that’s a warning from Ice itself and doesn’t have any impact on your script.

All the best,
~J.