Automated import

Dear @OMETeam helpdesk

Do you have full documentation how bulk config works and which data can be handled over import using yaml tempalte?

From when I remember it was done using following config https://github.com/IDR/idr-metadata/blob/0714af1cb7f144cab78744d44a0f0a79deffe50b/idr0013-neumann-mitocheck/screenB/idr0013-screenB-renderdef.yml
Now looks like things has changed https://github.com/IDR/idr-metadata/commit/8e787ac1d8de86b909daf5a4c080fe92cdbb1ca3#diff-2c7f9cbad041c030f19b94334ad18c6c

Could you point to the doc as https://docs.openmicroscopy.org/omero/5.6.2/users/cli/import-bulk.htm is brief or help me to build config ?

I really appreciate if I get help.

Ola

@olatarkowska the two commits you are linking to are identical and seem to point at rendering files rather than bulk import files? Is your question related to the specification of the bulk import YML files or the rendering YML files (or both)?

For the bulk import, in addition to the resource you mentioned, work has been put into a bulk import guide.

Hi @s.besson and thank you for your quick reply.

I have updated the link :slight_smile:

I am mainly interested in applying custom histogram, min max and channel renaming. Only found these examples. Could you please help me?

Ola

@olatarkowska Additionally to what @s.besson said, please have a look also at https://omero-guides.readthedocs.io/en/latest/upload/docs/index.html - the application of rendering settings via CLI rendering plugin (also with channel names) is covered there.

Please let us know if this is enough of information. Good luck. All the best

Petr

Hi @pwalczysko and thanks for your quick reply. I would prefer to have single yml for reproducability. I thought thats why IDR has everything in the config files :slight_smile: Doing thigns right in the first place is the way to go… It means I cannot do everything at once over import?

CLI has one disadvantage, parsing console output for Image ID. What is the best way to return Image ID then ? I also noticed that import errors are not going to sys.err and pipeline is not failing correctly, but thats another issue

Ola

I am mainly unterested in applying custom histogram, min max and channel renaming. Only found these examples. Could you please help me?

I think the initial confusion comes from the fact the recent versions of the render plugin/YML format have allowed most of the channels attributes to be optionally specified. All the recent study repositories are maintained separately - see https://github.com/IDR/idr0077-valuchova-flowerlightsheet/tree/master/experimentA/rendering_settings/ for a recent set of examples YML file setting the channel names as well as the rendering window limits e.g.

---
channels:
    1:
        active: true
        end: 500.0
        start: 200.0
    2:
        active: true
        end: 7000.0
        start: 150.0
    3:
        active: true
        end: 1000.0
        start: 200.0
    4:
        active: false
greyscale: false
version: 2

Note in version 2 of the specification, the attributes setting the rendering window have been renamed from min/max to start/end in order to match the semantics of the OMERO model.

1 Like

Thanks Seb, and that is available from OMERO version 5.5+ ?

Additional question, where is the src of the plugin/parser/etc? it will be easier for me to understand how it works as well

Definitely compatible with OMERO.server 5.5+ although the plugin will pull the latest omero-py dependency client side.

The plugin is available from https://pypi.org/project/omero-cli-render/ and the source code is at https://github.com/ome/omero-cli-render/.

1 Like

Hi @s.besson, thanks for that, what about names? Coudl you give a complete example for labeling channels? Thanks Ola

Hi Ola,

channel names can be specified using the optional label key.
Immediate real-world examples coming to mind would either be the YML files linked from https://omero-guides.readthedocs.io/en/latest/upload/docs/render.html mentioned earlier by @pwalczysko or https://github.com/IDR/idr0085-walsh-mfhrem/blob/adb35a400c2bd6bf3396803a84e4b20943c970e1/experimentA/rendering_settings/Brain_CMDiI_Lectin_z172.yml for the rendering settings associated to a recent IDR publication.

Sebastien

1 Like

Script that you use would be useful. Where is it?

Are you talking about a script to apply the rendering settings from a configuration file to a target image/plate/screen/dataset/project? If so, at least internally, we typically use the omero render set command directly.

1 Like

Yes, I am looking for a strategy that allows me to import image and return image id so I can apply these settings. These are 2 lines in bash so ImageID has to be pass. What is the recomanded way to do it? GREP? Is a shame that channel information cannot be pass as a param over improt step. Could you write that for me please?

Ah I understand the workflow you are looking after now (import + render). As it stands, the standard output of the import command will return the objects using the <Object>:<Id> semantics that can be piped to the omero render command.

The minimal example below should show conceptually how to chain the commands to import two test images and apply rendering settings to the created image:

$ mkdir test && touch test/test1.fake && touch test/test2.fake
$ omero import test --file import.out
$ for i in $(cat /tmp/import.out); do omero render set $i render.yml; done
1 Like

Thanks Seb, thats very useful

Hi @s.besson

I am affraid it dont see that works in 5.5.1

/nfs/users/nfs_o/omero/OMERO.server render set: error: argument object: invalid proxy value:

What needs to be installed for 5.5.1? Can you provide me with the full installation instruction?

I found what is wrong, I am not getting ImageID but Plate:1645, does it mean I cannot apply rdef to entire plate? Can you provide me with the script that will work around that?

In theory https://github.com/ome/omero-cli-render/blob/67cb072f83ab0e61410c89337c98e3ee274524eb/src/omero_cli_render.py#L444 should work, am I doing something wrong?

As you pointed out, the plugin definitely has logic to deal not only with individual Image as well as Project/Dataset/Screen/Plate. I successfully tested a minimal example with a single-image plate:

$ touch "test&plates=1.fake"
$ omero import test\&plates\=1.fake --file plate.out
$ for i in $(cat /tmp/plate.out); do omero render set $i render.yml; done

Can you try the above in your environment and paste the error if it fails?

Here you go

2020-09-16 09:49:47,514 4528       [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /nfs/assembled_images/datasets/JSP_HSS/JSP_HSS/JSP
Other imported objects:
Fileset:2806
2020-09-16 09:49:47,519 4533       [l.Client-0] INFO      ome.formats.importer.cli.ErrorHandler -/path/to/test Number of errors: 0

==> Summary
1 file uploaded, 1 fileset, 1 plate created, 1 image imported, 0 errors in 0:00:02.478
+ /nfs/users/nfs_o/omero/OMERO.server-5.5.1-ice36-b122/bin/omero render set 
/path/to/test.ome.tif.idlog
/path/to/test.ome.tif.render.yml
usage: /nfs/users/nfs_o/omero/OMERO.server render set [-h] [--skipthumbs]
                                                      [--disable]
                                                      [--ignore-errors]
                                                      object [object ...]
                                                      channels
/nfs/users/nfs_o/omero/OMERO.server render set: error: argument object: invalid 
proxy value:
'/path/to/test.ome.tif.idlog'

both files /path/to/test.ome.tif.idlog /path/to/test.ome.tif.render.yml exist

$ cat /path/to/test.ome.tif.idlog
Plate:1648
$ cat /path/to/test.ome.tif.render.yml 
channels:
    1:
        active: true
        color: FF00FF
        label: CH1
        end: 40000.0
        start: 491.0
    2:
        active: true
        color: FFFF00
        label: CH2
        end: 18000.0
        start: 1003.0
greyscale: false
version: 2