Server Error After Populating Metadata for Images Using Web Script

Today I was testing out the populate metadata script and I managed to add some metadata to a dataset. It seemed to work and the attributes were showing up for the images. However, a few seconds later an internal server error popped up, and continued to pop up when I looked at some of the other images. The strange thing is, sometimes when I initially click an image to bring up the sidebar view, the table will show up with the attributes blank, but when I click it again, they all show up correctly.
Here’s the log:

`{"stacktrace": "Traceback (most recent call last):
File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omeroweb/webgateway/views.py\", line 1346, in wrap
rv = f(request, *args, **kwargs)
File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omeroweb/webgateway/views.py\", line 2882, in object_table_query
tableData = _table_query(request, annotation['file'], conn, **kwargs)
File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omeroweb/webgateway/views.py\", line 2809, in _table_query
cols = t.getHeaders()
File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero_Tables_ice.py\", line 1052, in getHeaders
return _M_omero.grid.Table._op_getHeaders.invoke(self, ((), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = Traceback (most recent call last):
    File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/util/decorators.py\", line 68, in exc_handler
    rv = func(*args, **kwargs)
    File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/util/decorators.py\", line 28, in handler
    return func(*args, **kwargs)
    File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/tables.py\", line 201, in getHeaders
    rv = self.storage.cols(None, current)
    File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/util/decorators.py\", line 90, in with_lock
    return func(*args, **kwargs)
    File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/hdfstorageV2.py\", line 406, in cols
    names = self.__mea.colnames
    AttributeError: 'NoneType' object has no attribute 'colnames'
    
    serverExceptionClass = 
    message = Internal exception
    }
    ", "message": "exception ::omero::InternalException
    {
        serverStackTrace = Traceback (most recent call last):
        File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/util/decorators.py\", line 68, in exc_handler
        rv = func(*args, **kwargs)
        File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/util/decorators.py\", line 28, in handler
        return func(*args, **kwargs)
        File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/tables.py\", line 201, in getHeaders
        rv = self.storage.cols(None, current)
        File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/util/decorators.py\", line 90, in with_lock
        return func(*args, **kwargs)
        File \"/home/omero/OMERO.server-5.5.1-ice36-b122/lib/python/omero/hdfstorageV2.py\", line 406, in cols
        names = self.__mea.colnames
        AttributeError: 'NoneType' object has no attribute 'colnames'
        
        serverExceptionClass = 
        message = Internal exception
        }"}`

edit:
I did use the old populate-metadata script in the past and when the attributes are not working it actually shows the attribute fields from those older images. I checked those older images and the metadata is not showing up correctly for them. Should I try deleting those images?

Here was the csv I used:

# header s,s,s,s
Image Name, type, price, temperature
apple.jpg, apple pie, $5, hot
berry.jpg, berry pie, $4, hot
chocolate.jpg, chocolate pie, $5, cold
coconut.jpg, coconut cream pie, $8, cold
lemon.jpg, lemon meringue pie, $5, cold
peanut.jpg, peanut pie, 4$, cold
pumpkin.jpg, pumpkin pie, $4, cold

I deleted the older images and checked the option to delete the older annotations as well and restarted the server. However, the issue persists, and the old attributes still appear along with that same error.
Edit: I deleted the new images and reuploaded them and added annotations from scratch. Same error as occurring as before.

Here’s a gif of the error happening: https://i.imgur.com/qhKQAc8.gif

Hi @pmann,

usually when one sees AttributeError: 'NoneType' object has no attribute it manes that the service has stopped running, but that doesn’t seem to be the problem in your case. Could you attach the var/log/Tables-0.log file to see if there are more details server-side?

Thanks,
~Josh

Here’s the log file:Tables-0.log.txt (1.5 MB)
Thanks for the help,
Philip

Thanks, @pmann. I can see the error but it doesn’t really provide much more information. (Except that there are frequent though unrelated errors on getWhereList which might be worth looking into)

Can you confirm that this is only happening for the file created from the CSV above:

and that other tables are working correctly? If so, do you have an example of one of those?

~Josh

Other tables exhibit the same behavior.
I tried it again with fruits instead of pie. It looks like whenever I annotate a dataset, it initially works correctly without any problems. However, when I refresh the browser, reopen the dataset, and expand the table field, the error pops up and I get the same issue with the new dataset.
Here’s the csv:

Image Name,fruit,price,color
apples.jpg,apple,$1,red
banana.jpg,banana,$0.25,yellow
blackberry.jpg,blackberry,$1.50,black
raspberry.jpg,raspberry,$2.00,red

Hi Philip,
sorry for the late reply, it took me a while trying to replicate this issue.
The information you see in the ‘Tables’ tab when you select an image is actually not attached to the image. OMERO.web will look at the dataset and project level, check if there’s a ‘bulk_annotations’ attachment and then looks for an entry for this specific image in there. Unfortunately if there are more than one bulk_annotations attachments on either level it will pick a random one. This explains why you interchangingly see the correct values, old values or the error when repeatedly clicking on the same image. I suspect you have multiple ‘bulk_annotations’ on the dataset and/or project. Could you check that? And probably one of the bulk_annotations tables is somehow broken, hence the errors.
You should probably delete all of them, and then run ‘metadata populate’ again, to make sure there is only one correct bulk_annotations table.
Kind Regards,
Dominik

Hey Dominik,

Thanks for the help! Unfortunately the problem seems to persist.
I deleted the project and created a new one with a different name. A created a single dataset with a different name within it and then once again used the web script to populate that dataset with metadata. The dataset has a single csv and a single annotation attached (not the project, the dataset). I’m still getting internal server errors, although the empty attributes are no longer showing up. The actual metadata shows up completely fine, just the error continues to pop up very often.
Is there some way I could delete all of my annotations? I’m wondering if one of them got messed up somehow and is messing everything else up. Also, is it possible that it’s a problem with something that the web script is doing? I might try running it from the CLI to see if I still get the error.

Thanks,
Philip

Hi Philip, You’re talking to the OME team and they are the experts. For a different approach I annotate images/datasets with a csv and the CLI. If you look about half way down (May 27) this post: Setting up OMERO in a new lab - suggestions, stories and advice welcome

I outline a way to do that. I don’t go through the webclient or “attach” the annotations first. Everything goes thru the CLI. Good luck! John

Hi,
I reproduced this error exactly (internally at https://merge-ci.openmicroscopy.org/web/webclient/?show=dataset-4919 (user-3)) and I see exactly the same stack trace as above. And it happens intermittently as you reported, suggesting that there’s some race condition (I see there’s a with_lock in the stack trace). In fact, when I open the devtools in the browser and in the Network tab I see which URL is causing the error. When I open that URL on it’s own in a new tab, I never see a problem. e.g. https://merge-ci.openmicroscopy.org/web/webgateway/table/Dataset.imageLinks.child/64804/query/?query=image-64804 (this fails silently with "error": "Error executing query: (image==64804)" without causing an error in the browser).

Hope that helps others to dive deeper?

Will.

@will-moore,

thanks for the steps to reproduce. My best guess at this point is that what’s happening is:

Thread 1                 Thread 2
--------                 --------
open file
                         open file
query
                         ...
close                
                         query with exception

I’ll try to either have the service expanding its lock or retry on “already closed”.

~Josh

Almost

$ PYTHONPATH=lib/python python t2.py
exiting...
0	exception ::omero::InternalException
1	exception ::omero::InternalException
2	exception ::omero::InternalException
3	exception ::omero::InternalException
4	exception ::omero::InternalException
5	exception ::omero::InternalException
6	exception ::omero::InternalException
7	exception ::omero::InternalException
8	exception ::omero::InternalException
9	None
bash-4.2$ cat t2.py
import omero
import omero.cli
import threading

THREADS = 10
LOOPS = 20

class T(threading.Thread):
    def __init__(self, table, index):
        threading.Thread.__init__(self)
        self.table = table
        self.index = index
        self.exc = None
    def run(self, *args, **kwargs):
        try:
            if self.index < (THREADS-1):
                for x in range(LOOPS):
                    try:
                        self.table.getHeaders()
                    except Exception as e:
                        self.exc = e
                        break
            else:
                print "exiting..."
        finally:
            self.table.close()

with omero.cli.cli_login() as cli:
    client = cli.get_client()
    t = client.sf.sharedResources().newTable(1, "foo.h5")
    t.initialize([omero.grid.LongColumn("foo")])
    file = t.getOriginalFile()
    threads = []
    for x in range(THREADS):
        threads.append(T(t, x))
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()
    # Final thread closes

for idx, thread in enumerate(threads):
    print "%s\t%s" % (idx, str(thread.exc).split("\n")[0])

i.e. this is caused by a single service instance (“servant”) being used amongst many threads without coordination on when close() will be called. My first tests tried to reproduce with multiple services, but the internal reference counting “did the right thing”.

Created https://github.com/openmicroscopy/openmicroscopy/issues/6092

Thanks, all.