How to use OMERO.fs

Hello,

I try to import / copy data to OMERO server, which includes images, text and nested folders .
They have a little bit complicated folder structure to import as it is.
So I broke them down so as to import them to OMERO server using “OMERO.insight”
or series of commands. It is a sure process. But it is laborious.

I found the following description:

" OMERO.fs server stores the original files in the ManagedRepository , preserving file names and any nested directory structure."

in "OMERO 5.6.0 documentation. So I would like to try to use “OMERO.fs”.

There are documents that explain “OMERO.fs”. But I could not find samples.
Would someone show me a sample code or the like for my reference?

Thank you.

Best regards,
H. Fujimoto

All methods of import are already built on top of OMERO.fs, and store the files in the ManagedRepository just as the quote from https://docs.openmicroscopy.org/omero/5.6.0/developers/Server/FS.html describes. The methods to import are:

  • OMERO.insight, client-side, pure Java
  • OMERO.cli, client-side, mix of Java and Python
  • via the API in Python or Java
  • OMERO.dropbox, server-side

I assume you haven’t tried OMERO.dropbox yet, but perhaps it could help you workaround the current Python/Windows issue that is blocking you. The basic idea is that if the files are placed correctly on the server, then they will be detected and automatically imported. The primary caveat is that it requires filesystem-level notifications (e.g. inotify) which aren’t available for remote systems like NFs.

Let us know if that sounds like an option for you. The https://docs.openmicroscopy.org/omero/5.6.0/sysadmins/dropbox.html page has more information on configuration.

Cheers,
~Josh

Dear Josh-san,

Thank you for your suggestion and remembering my subjects.
I am sorry for my delayed reply.
Your assumption is correct. I have never tried OMERO.dropbox.
So I set up OMERO.dropbox and tried to use it.

I copied my data folder that includes nested folders and various files to “/OMERO/DropBox/DS” folder last night.
I have found lots of images copied to my designated user’s this morning.

I have questions:

  1. Is there a way to keep a directory structure?
    The directory hierarchy was flattened. All the image files went into one container “Orphaned Images”.
  2. Is there a way to copy metadata relating images together?
    I found only images copied to the container.

I tried to change settings by modifying “templates.xml”. The below lines with " * " are my modified lines.

<property name="omero.fs.host" value="localhost" />
<property name="omero.fs.port" value="${ROUTERPORT}" />
<property name="omero.fs.maxRetries" value="5" />
<property name="omero.fs.retryInterval" value="3" />
<property name="omero.fs.defaultDropBoxDir" value="DropBox" />
<property name="omero.fs.platformCheck" value="True" />
* <property name="omero.fs.importUsers" value="DS" />
* <property name="omero.fs.watchDir" value="/home/omero/Data/DS" />
<property name="omero.fs.eventTypes" value="Creation,Modification" />
<property name="omero.fs.pathMode" value="Follow" />
<property name="omero.fs.whitelist" value="" />
<property name="omero.fs.blacklist" value="" />
<property name="omero.fs.timeout" value="0.0" />
<property name="omero.fs.timeToLive" value="0" />
<property name="omero.fs.timeToIdle" value="600" />
<property name="omero.fs.blockSize" value="0" />
<property name="omero.fs.ignoreSysFiles" value="True" />

* <property name="omero.fs.ignoreDirEvents" value="False" />

<property name="omero.fs.dirImportWait" value="60" />
<property name="omero.fs.fileBatch" value="10" />
<property name="omero.fs.throttleImport" value="10" />
<property name="omero.fs.readers" value="" />
<property name="omero.fs.importArgs" value="" />
<property name="omero.fs.fileServerIdString" value="FileServer" />
<property name="omero.fs.fileServerAdapterName" value="omerofs.FileServer" />
<property name="omero.fs.monitorServerIdString" value="MonitorServer" />
<property name="omero.fs.monitorServerAdapterName" value="omerofs.MonitorServer" />
<property name="omero.fs.serverIdString" value="MonitorServer" />
<property name="omero.fs.clientIdString" value="DropBox" />
<property name="omero.fs.clientAdapterName" value="omerofs.DropBox" />
<property name="omero.fs.serverIdString" value="MonitorServer" />
<property name="omero.fs.clientIdString" value="TestDropBox" />
<property name="omero.fs.clientAdapterName" value="omerofs.TestDropBox" />
<property name="omero.fstest.config" value="${OMERO_ETC}testdropbox.config" />

Strange to say, the modifications are not reflected.
After copying new “templates.xml”, I stop and restart OMERO.server.
Reading “DropBox.log”, I found the modifications were not reflected.
“templates.xml” was overwritten by the older one.
So I could not succeed in using OMERO.dropbox again today.

I would appreciate it, if I could have your suggestion.

Thank you .

Best regards,
Hiroki Fujimoto

Fujimoto-san,

Setting the importArgs option to: -T 'regex:/home/omero/Data/DS/(?<Container1>.*)' should do what you want. I tested via:

cd /tmp
mkdir -p a/b/c/d.fake
omero import -T "regex:/tmp/(?<Container1>.*)" a/

which imports the file d.fake into a Dataset a/b/c.

See https://docs.openmicroscopy.org/omero/5.6.0/users/cli/import-target.html for more information.

I’m not sure what you mean, but I don’t think so.

You will need to edit etc/templates/grid/templates.xml instead.

~Josh

Dear Josh-san,

Thank you for your advice.
I am sorry for my silence, as I was away from my office.

cd /tmp
mkdir -p a/b/c/d.fake
omero import -T "regex:/tmp/(?<Container1>.*)" a/

I tried your sample commands and realized it worked.

Setting the importArgs option to: -T 'regex:/home/omero/Data/DS/(?<Container1>.*)'

Following your advice, I added them to “importArgs” option as follows:

property name=“omero.fs.importArgs” value="-T ‘regex:/home/omero/Data/DS/(?<Container1>.*)’ "

A red marker appears on the “<” in <Container1>. Something is wrong.

As for “templates.xml”, your advice worked well. Thank you.

I’m not sure what you mean, but I don’t think so.

I am sorry for my ambiguous sentences.
When I use omero.DropBox, images are automatically imported but metadata attached to the images are not. So we have to import the metadata manually. Am I correct?

I would appreciate it, if I could have your suggestion.
Thank you .

Best regards,
Hiroki Fujimoto

Fujimoto-san,

:bow:

Ah, apologies. I didn’t consider that this needs to be embedded in XML. Can you try &lt; and &gt; to escape < and > respectively?

Thank you for the clarification. Is this metadata in a separate file that you have beside each image? (As opposed to metadata that is coming from the microscopy file format itself)

All the best,
~Josh

Dear Josh-san,

Thank you for your reply.
I found the describing rules of some symbols in XML, too.
I am sorry for my careless mistake. Your advice worked well.
The target data set name is the same as the path of the source file.

But I came across another situation, which I will report.
I have changed my “templates.xml” as follows:

    <property name="omero.fs.importUsers"  value="DS"/>
    <property name="omero.fs.watchDir"  value="/home/omero/Data/DS"/>
    <property name="omero.fs.eventTypes"  value="Creation,Modification"/>
    <property name="omero.fs.pathMode"  value="Follow"/>
    <property name="omero.fs.whitelist"  value=""/>
    <property name="omero.fs.blacklist"  value=""/>
    <property name="omero.fs.timeout"  value="0.0"/>
    <property name="omero.fs.timeToLive"  value="0"/>
    <property name="omero.fs.timeToIdle"  value="600"/>
    <property name="omero.fs.blockSize"  value="0"/>
    <property name="omero.fs.ignoreSysFiles"  value="True"/>
    <property name="omero.fs.ignoreDirEvents"  value="True"/>
    <property name="omero.fs.dirImportWait"  value="60"/>
    <property name="omero.fs.fileBatch"  value="10"/>
    <property name="omero.fs.throttleImport"  value="10"/>
    <property name="omero.fs.readers"  value=""/>
    <property name="omero.fs.importArgs"  value="-T 'regex:/home/omero/Data/DS/(?&lt;Container1&gt;.*)' "/>

I copied a folder, which has three nested folders, to my “watchDir”. As I expected, the importing
of the images started automatically.
The first folder has text files only. This folder was not imported. It is what I expected.
The second folder has both text files and images. Only images were imported. It is what I expected too.
The third folder has images only. Images were imported. But the importing did not stop. It repeated.
So I stopped the DropBox in the middle of the 3rd repetition.

I attach the file tree of the folder that I was going to import.
Folder.txt (9.8 KB)

Would you tell me what was wrong in my “templates.xml” or my folder?

As for the metadata importing, I will make “omero” commands to import images and text files
and issue the commands sequentially. These sequential commands will attach the text files
to the data set. I will make such a python script.
At now for me, I suppose that it is a sure way to realize my purpose.

Thanks in advance.

Best regards,
Hiroki Fujimoto

Fujimoto-san,

Bizarre. This is a behavior I’ve never seen before. Certainly nothing in your Folder.txt file would make me think that you would have each image three (or more) times. I would expect both the TIFF and the JPEG versions to be imported separately though. Could you upload your DropBox log and MonitorServer.log files from var/log?

Many thanks,
~Josh

Dear Josh-san,

Thank you for your reply.
I attach DropBox.log and MonitorServer.log to this post.

Logs.zip (142.8 KB)

In the attached log files, the date “2020-03-27” is the day when I saw the situation.

I would appreciate it, if you show me what was wrong in my operations.
Thank you.

Best regards,
Hiroki Fujimoto

Fujimoto-san,

thanks for the logs. There is definitely at least one issue that’s quite clear. See https://github.com/ome/omero-dropbox/issues/14

We’ll work on a release of DropBox (5.6.2) for you to try.

All the best,
~Josh

Dear Josh-san,

Thank you for investigating the logs and finding an issue.
I wait for your improvement.

Thank you.

Best regards,
Hiroki Fujimoto

Fujimoto-san,

I just released https://pypi.org/project/omero-dropbox/5.6.2/

Please use pip install -U omero-dropbox and restart your server to give it a try.

All the best,
~Josh.

Dear Josh-san,

Thank you for your quick improvement and its notification.
I have tried the new DropBox.

Just after restarting the server, the new DropBox worked well.
When importing from DropBox had terminated, I removed the folder from my “watchDir”.
After that, Dropbox never responds, until I restart the server.

An error was recorded in “MonitorServer.log”. Lines after “2020-04-03 13:33:35” are those.

I attach “DropBox.log” and “MonitorServer.log” to this post.
Logs.zip (22.8 KB)

I would appreciate it, if I could have your comments.

Thanks in advance.

Best regards,
Hiroki Fujimoto

Thanks for testing, Fujimoto-san. This looks like a deeper issue. I will get back to you as soon as I know more. ~Josh

Hi Fujimoto-san.

Just a quick update that I haven’t yet been able to reproduce your issue. Sorry for the long wait.

~Josh

Dear Josh-san,

Thank you for taking care of this.
Do not worry.
I am working from home now. My access to the system is limited.
So my OMERO server for the test is closed.

Any update would be appreciated.

Thank you.

Best regards,
Hiroki Fujimoto

Fujimoto-san,

:slight_smile: Understood.

Will do ASAP.

~Josh