Automated uploader to OMERO in python

Hi
Our FIBICS NPVE pattern generator that we use with our HIM or Ga-FIB creates tiff images during the mill process. From time to time it is useful to have those together with the other metadata that is recorded in txt files for the mill process. The images are nicely stored in a directory tree

project - sample - mill shape - tiff and data files

The sample directory might contain additional not automatically recorded images.

I’d like to import these files into omero mapping sample to project and mill shape to dataset. I tried to do this in python from a jupyter notebook but failed to get the code working for filesets. I can use the CLI from my python script but these seems to be clumsy and surprisingly slow. I’ve got the image upload working and now I am trying to get the txt files from each shape uploaded and linked to the datasets.

My question essentially boils down to
How can I do

omero import /path/to/my-dir -T Project:name:MyProjectName/Dataset:name:MyDatasetName --parallel-upload=8

from python so I can easily traverse sub-directories and construct the Dataset and Project names from other input available in those directories.

I can provide data and also the code that I already have but what I think I need is a good example on how to upload filesets from python.

thanks

Gregor

Have a look at https://gitlab.com/openmicroscopy/incubator/omero-python-importer/-/tree/master

Hopefully that should do most of what you want.
Let us know if you have any questions.

Will

Hi will

thanks for the link. It works but I had to change the way you open the file in the ‘upload_files’ function.
I got an Unicode error, probably because images are binary files.

UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x80 in position 8: invalid start byte

I changed line 118 in ‘upload_files’ to

with open(fobj, ‘rb’) as f:

Gregor

Ah, that sounds like it’s because that code sample was written for Python 2 and we’ve not updated to use with Python 3.
Thanks for the feedback, glad to hear it’s working,

Cheers,
Will.

Hi,

I’m starting to work on a way to automatically import images from the users hard drive and was wondering if the best route to take was to start from the pure python importer referred above, or use the java wrapper through things like:

suprocess.run("omero import /path/to/my-dir -T", shell=True)

Thanks!

Guillaume

Hi Guillaume,

I think it would depend on whether you want to write your importer in Python, Java or bash scripts?

Simplest would probably just be a bash script.
For example, we use a bash script to populate a DB with users and to import images for testing setup:

But if you want much more complex logic or add a UI then you may wish to use Python or Java?

You might want to continue the discussion on a new topic since this one is marked as “solved”.

Cheers,
Will.