ImageJ plugin and big 2D image: Ice.ConnectionLostException

Hi!

I’m currently trying to process an image stored on our server which is 18000 x 18000 pixels.
When I try to open it through the ImageJ plugin (whether by double-clicking it or through a script), I get an error message.
At first, I got error messages about Ice.MessageSizeMax, so I increased it a little bit on our server.
However, I now get this message: loci.formats.FormatException: Ice.ConnectionLostException

I guess, when scripting, it should be accessed by tiles, but it seems the API is a bit different in ImageJ Jython from the Python API. Or maybe I got confused and missed something?
Also, that means an end-user cannot open this image other than by downloading the file first?

I fear I’m just asking dumb questions… But I’d be very thankful if someone could enlighten me on this topic!

Hi Pierre

For large images, you will not be able to use directly the Bio-Formats unless you crop the image.
The Jython example script below directly uses the PixelsStore to retrieve the plane/region you wish to load.

This example script will only work for some images, not all image types are supported

Cheers

Jmarie

Thank you!

I tried something like that, but I still get this error message when I try to get the image through store.getPlane():

plane = store.getPlane(z, c, t)
reason = “requested 694079280 bytes, maximum allowed is 256000000 bytes (see Ice.MessageSizeMax)”

… although I already increased Ice.MessageSizeMax on my server…
omero config get does give me:

Ice.MessageSizeMax=1572864

I thought I should get the image through tiles, as shown in the Python API documentation, something like:

image = conn.getObject("Image", imageId)
size_z = image.getSizeZ()
size_c = image.getSizeC()
size_t = image.getSizeT()
pixels = image.getPrimaryPixels()
c, t = 0, 0
tile = (50, 50, 10, 10)
zct_list = [(iz, c, t, tile) for iz in range(size_z)]
planes = pixels.getTiles(zct_list)

But as I mentioned before, the API looks a bit different as the entry point is “Gateway” and not “BlitzGateway”. I suppose I could try using getCol()

Hi Pierre
You should try to use
getTile(int z, int c, int t, int x, int y, int w, int h) instead of getPlane

Jmarie

Ok. Thank you!
I finally got it to work with this (EDIT: although I think I still need to add metadata like pixel size):

def open_omero_image(gateway, ctx, image_id):
    SIZE_TILE = 16000
    browse = gateway.getFacility(BrowseFacility)
    image = browse.getImage(ctx, image_id)    
    pixels = image.getDefaultPixels()
    size_z = pixels.getSizeZ()
    size_t = pixels.getSizeT()
    size_c = pixels.getSizeC()
    size_x = pixels.getSizeX()
    size_y = pixels.getSizeY()
    pixtype = pixels.getPixelType()
    pixels_type = FormatTools.pixelTypeFromString(pixtype)
    bpp = FormatTools.getBytesPerPixel(pixels_type)
    SIZE_TILE = SIZE_TILE / bpp
    ntile_x = int(math.ceil(float(size_x) / float(SIZE_TILE)))
    ntile_y = int(math.ceil(float(size_y) / float(SIZE_TILE)))
    is_signed = FormatTools.isSigned(pixels_type)
    is_float = FormatTools.isFloatingPoint(pixels_type)
    is_little = False
    interleave = False

    # setup the pixelStore
    store = gateway.getPixelsStore(ctx)
    pixels_id = pixels.getId()
    store.setPixelsId(pixels_id, False)

    imp   = IJ.createHyperStack("tmp", size_x, size_y, size_c, size_z, size_t, bpp*8)
    stack = imp.getImageStack()

    niter = size_t * size_z * size_c * ntile_x * ntile_y
    
    for t in range(size_t):
        for z in range(size_z):
            for c in range(size_c):
                n = imp.getStackIndex(c+1,z+1,t+1)
                ip = stack.getProcessor(n)
                for x in range(ntile_x):
                    w = min(size_x - x*SIZE_TILE, SIZE_TILE)
                    for y in range(ntile_y):
                        h = min(size_y - y*SIZE_TILE, SIZE_TILE)
                        tmp_ip = ip.createProcessor(w, h)
                        tile = store.getTile(z, c, t, x*SIZE_TILE, y*SIZE_TILE, w, h)
                        ImageTools.splitChannels(tile, 0, 1, bpp, False, interleave)
                        pixels = DataTools.makeDataArray(tile, bpp, is_float, is_little)
                        tmp_ip.setPixels(pixels)
                        ip.copyBits(tmp_ip, x*SIZE_TILE, y*SIZE_TILE, Blitter.COPY)
                        progress = float(1 + y + x*ntile_y + c*ntile_x*ntile_y + z*size_c*ntile_x*ntile_y + t*size_z*size_c*ntile_x*ntile_y) / float(niter)
                        print(str(100*progress)+"%")
                stack.setProcessor(ip,n)
                
    imp.setStack(stack)
    store.close()

    imp.setOpenAsHyperStack(True)
    imp.setDisplayMode(IJ.COMPOSITE)

    return imp

Also, a bit unrelated but if I wanted users to run this macro from the command line (headless ImageJ), what would be the best (most secure) way to manage the authentication?

Hi Pierre

You could collect the arguments at the beginning of you script. Below is a simple groovy example tested using Fiji

#@ String(label="username") username
#@ String(label="password", style='password', persist=false) password

println username
println password

then run from the command line

./ImageJ-macosx --headless --run ~/Documents/test.groovy "username='test',password='foo'"

Cheers

Jmarie

Ok. That’s what I’m doing, but I wanted to avoid putting the password in clear.
I guess I’ll just have to wrap it in a shell script with a password prompt.
Also, I actually can’t run it headless anyway, at least for now, as my script relies on the ROI manager, which requires the UI.

Thank you for your help!

It’s been a few months now, but I now realize that even doing so would still leave the password for all to see using ps

Yes, and the same is true (though more involved) for environment variables. Reading from a protected file should be safe from all except superusers.

~Josh

1 Like