Executing Pipeline from Celery is causing issue


Attached is my program which i am calling from Celery worker. It is giving below error

[2014-07-01 12:59:12,410: ERROR/PoolWorker-5] Error detected during run of module LoadImages
Traceback (most recent call last):
  File "/home/ubuntu/dev/CellProfilerNew/cellprofiler/pipeline.py", line 1787, in run_with_yield
  File "/home/ubuntu/dev/CellProfilerNew/cellprofiler/modules/loadimages.py", line 2174, in run
    image = provider.provide_image(workspace.image_set)
  File "/home/ubuntu/dev/CellProfilerNew/cellprofiler/modules/loadimages.py", line 3199, in provide_image
    self.get_name(), url=self.get_url())
  File "/home/ubuntu/dev/CellProfilerNew/bioformats/formatreader.py", line 874, in get_image_reader
    rdr = ImageReader(path=path, url=url)
  File "/home/ubuntu/dev/CellProfilerNew/bioformats/formatreader.py", line 587, in __init__
  File "/home/ubuntu/dev/CellProfilerNew/cellprofiler/utilities/jutil.py", line 1566, in make_instance
    klass = get_env().find_class(class_name)
AttributeError: 'NoneType' object has no attribute 'find_class'

When i run this program from terminal it is working fine. is this issue related to threads where multiple requests are causing issue?
CellProfilerTest.py (4.73 KB)

I’m looking at the code, one perplexing thing is that you have a call to kill_vm() without importing it from cellprofiler.utilities.jutil. The behavior you’re getting looks to me like the code is running after the Java VM has been killed (you only can start the JVM once, so another call to start_vm won’t help). So maybe you called deconvolve_image twice?

I’d put the kill_vm call inside the main() function and then you can use deconvolve_image as many times as you want. Also, look at some of the tests - you can execute modules directly using something like

import cellprofiler.pipeline as cpp
import cellprofiler.workspace as cpw
import cellprofiler.measurements as cpm
import cellprofiler.cpimage as cpi
import cellprofiler.objects as cpo
from bioformats import load_using_bioformats
from bioformats.formatwriter import write_image
from cellprofiler.modules.unmixcolors import UnmixColors

pipeline = cpp.Pipeline()
module = UnmixColors()
module.module_num = 1
module.input_image_name.value = "i"
module.outputs[0].image_name.value = "o"
measurements = cpm.Measurements()
measurements.add("i", cpi.Image(load_using_bioformats(path_to_input_image))
workspace = cpw.Workspace(pipeline, module, measurements, cpo.ObjectSet(), measurements, None)
write_image(path_to_output_image, measurements.get_image("o").pixel_data, PT_UINT8)

This kind of approach is a bit more straightforward, uses less infrastructure and is more likely to work in future CP versions (otherwise, I’d have to change a LOT of tests!).