Managing data with in-place import in remote server

Hello,

below details of the OMERO version i am using and the problem i am facing

OMERO version used:
OMERO 5.5.1 docker compose

I am setting an in-place import for all the images as i cannot afford to import the images directly into the docker compose. Simply a problem with space.

For the in place import in docker compose, i have set up the yml file so that there is a line under the OMERO server volume in which the remote folder (which in my case is on a NAS) with the images is mapped:

mountpoint_clientserver/linkedfolder_omerodocker:permissions

“mnt/OMEROdata:/external_data:ro”

As it stand i can easily import all the images that are located in the NAS under the folder OMEROdata.

this is the typical expression i use

bin/omero import --transfer=ln_s /external_data/IMG_7106.JPG --sudo root -u smsg_manager -g smsg -T Dataset:name:imports

However, the folder OMEROdata will soon be populated with images from various users and it will make the in-place import very difficult as I cannot see how i can easily import “x” images if there are a hundresds of them in a folder, without any structure.
So, the logic is that the users that can access OMEROdata start building new directories and let the sysadmin to do the in place import from specific directories.

I tried to add a file which was in a subdirectory of OMEROdata (e.g. /mnt/OMEROdata/newfolder/IMG009.jpg) but it didn’t work.
Essentially the error message is “UNREADABLE_FILE”

Any clue on how this issue can be overcome?
Do i need to modify the line in the yml file? Do i simply need to change the --transfer=ln_s with the path of the linked directory?

/mnt/OMEROdata/newfolder?

thanks a lot,
iLaria

Hi Ilaria,

Can you include more of this error?

You shouldn’t need to change the mount command (especially not after each new directory).

If I follow correctly, this should “just work”. A few questions:

  • What are the permissions on newfolder and IMG009.jpg?
  • If they aren’t set up permissively, does chmod fix the issue?
  • Do you know if you have selinux configured on this machine?
  • What version of docker is running on the server?
  • What type of filesystem/mount is /mnt/OMEROdata?

Cheers,
~Josh

hi Josh,

message error is below

inplace-import-ERROR3.txt (10.8 KB)

permissions are as well depicted in the file attached below. Note 1026:users refer to the person who uploaded the data in the NAS

I didn’t try chmod simply because if i upload the images that are simply in OMEROdata, there is no error (the inplace import runs OK), but the issue is with the images in any subfolder created in OMEROdata. The permissions as you can see looks all the same.

Selinux: nope this is not installed. When i tried getenforce it returned i need to install selinux-utils, hence i think there is no selinux on the server. Note, i am running Ubuntu 18.04.

Docker version: 19.03.1
API version 1:40

The mount is nfs.

Thanks for your help!

Hi Ilaria,

I’m now hoping this is easier than I thought:

java.io.FileNotFoundException: /external_data/LA0001_L.JPG

Are you still mounting via “mnt/OMEROdata:/external_data:ro”? If so, should your command not be:

bin/omero import --transfer=ln_s /external_data/dolphins/LA0001_L.JPG ...
                                                xxxxxxxx

~Josh

hi Josh,

i wish was that simple, but no joy. Here below the error message
inplace-import-ERROR4.txt (5.3 KB)

I had actually thought that the i could not change the /external_data/ as this is what we wrote in the yml file.
Maybe i am wrong and i don’t understand how the soft link works, thus my apologies in advance.

iLaria

Hi iLaria,

well, at least we have a new error:

ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /external_data/dolphins/LA0001_L.JPG
java.io.IOException: Input/output error
	at sun.nio.ch.FileDispatcherImpl.pread0(Native Method) ~[na:1.8.0_212]

I’d say let’s take OMERO out of the mix. You will need to be able to access that file from within the docker regardless. Can you use docker-compose exec omero bash or possibly docker-compose exec omeroserver bash (depending on what’s in your docker-compose.yml file) and then try to access that file:

cd /external_data/dolphins/
file LA0001_L.JPG
wc LA0001_L.JPG

By the way, not that it should be necessary, but have you tried restarting your docker containers since that directory was created?

~Josh

Hello Josh,

so yes, i did run the docker-compose down and yes! i can see the images, so at least they are in the external_data folder. I guess that it means the mount works.
So perhaps the error is in that in-place import?

images_in_OMERO

1 Like

Agreed, that looks encouraging. But to confirm: both the inplace import error and this screenshot took place after the restart?

I can’t think of anything particularly special about in-place import that would cause this exception. It might be worth trying a bog standard import (bin/omero import /external_data/dolphins/LA0001_L.JPG) which you can subsequently delete. I imagine this is more an issue of the OMERO user accessing files over NFS belonging to 1026:users, but I don’t yet have a good theory why.

~J.

yes, they took place after the restart.

"I imagine this is more an issue of the OMERO user accessing files over NFS belonging to 1026:users"

wait a second…i haven’t given access to the NAS to any of the OMERO users i created in docker compose as i didn’t have this step in my notes.
Is the OMERO user you are mentioning the one that does the in-place import?
in this case omero-server?

The only thing I have done was to set up the soft link in the yml file and make the folder read only

That’s it. How would you give access to the NAS folder to omero-server?

unfortunately by not adding any image to OMERO i cannot deploy the server in the Falklands and no one is still using it here.

hi Josh,

so perhaps we can see some light at the end of the tunnel

if i run

bin/omero import /external_data/dolphins/LA001_L.JPG --sudo root -u USERNAME -g GROUPNAME -T Dataset:name:DATASETNAME

i manage!

bin/omero import --transfer=ln_s is the bit that doens’t work. Perhaps Simon when is back can have a look at this mail exchange and then try to understand what is wrong with that command.

As things stand right now the problem has been “solved” by changing the command line.
i can now make OMERO available to the users in the Falklands.

Hi Ilaria

Glad to hear that the command started to work. Nevertheless, I would like to draw your attention to the fact that the removal of the --transfer=ln_s means, that the import is no more in-place.
So, you have to consider whether or not this is what you and your users need. The positives for not having the in-place import are that the data which were imported from the mounted drive is (after the import has been done) are duplicated in OMERO. But exactly that might be also a drawback - you will need some place on the machine where the OMERO is running, as more and more imports are being performed. Did you consider that ?

Congrats

All the best

Petr

hi Petr,

thanks a lot for pointing out this relevant point as i was not aware (shame on me as i should have read the docs!). So the answer is no…i need the in-place import so, i am back to square 1 and with the problem unsolved. At this point i hope that Simon can cast light on this issue as whatever i could do, i did it (i think).