CP headless and JPEG2000

Hi there,

First of all, thank you very much for this fantastic piece of software! I’m struggling with CP in headless mode. We managed to have a (semi-) running installation of CP2.1 on a RedHat 6.4 Enterprise server. The system is running fine in headless mode for sets of JPG encoded images, I do, however, have issues with 16bit JPEG2000 encoded images. I don’t see much in the output; the process just starts and closes w/o an error message in DEBUG mode.

In general it seems possible to use this file format: the GUI version loads and runs them properly. It seems to me that these images (.jp2 file ending) are not correctly recognized by the loading images module in headless mode. Could this be possible?

I tried to attach some example image files. Seems like your forum doesn’t like jp2 files as well. :smile: Could you help me out here, please? Thank you very much! The pipeline is attached.

With my very best regards,
Matthias
jpeg2000pipeline.cppipe (33.6 KB)

Hi Matthias,

JPEG2000 should be supported fully by CP, because BioFormats supports it (openmicroscopy.org/site/sup … -2000.html). But that doesn’t mean there isn’t an issue somewhere on the software side!

We’d need a JPEG2000 image to debug, as you surmised. If you zip the file, the forum should allow files with ‘zip’ extensions. Or share via the cloud (e.g. Dropbox or Google Drive).

Thanks,
David

Hi David,

Thank you very much for your quick reply. I could have thought about that earlier; I attached the zipped jp2s.

Best regards,
Matthias
jp2_examples.zip (9.49 MB)

Hi David,

I was digging a little bit deeper and it seems I encounter a mix of issues: First, it looks like I’m running into the bug described in viewtopic.php?f=14&t=3696&p=10892&hilit=headless#p10892. If I understood correctly, this has already been fixed in one of the more recent dev-versions. Currently, I have installed the yum repos, which corresponds to the 2.1 release. I’ll try to create a build from the master and check whether this is solved for my case.

Regarding jp2: as I said, standard jpg files seemed to work, but I realized I used an old pipeline (v2.0) for the headless run. As the formerly attached jp2 pipeline was build in v2.1, I retested this using a v2.0 version. This pipeline works fine for jpg and tiff versions of the images but not for jp2. I attached the CP 2.0 version pipeline. So there still seems to be the issue with jp2 files. (using the GUI, all of them can be read)

Best regards,
Matthias
jpeg2000pipeline.cp (36.1 KB)

I just installed the nightly trunk build rpm. It seems like there is still an issue with the image loading during a headless call as described in the linked forum topic.

This is my command: cellprofiler -L DEBUG -c -r -i /home/wirthma1/cpjpeg2000_test_jpg -o /home/wirthma1/cpjpeg2000_test_jpg -p pipeline_general.cppipe

Thanks – I submitted this as a Github issue for us to look into:
github.com/CellProfiler/CellPro … ssues/1174

David

I just tried this out and successfully read the file, being sure to set java.awt.headless to ensure that the file could be read headless. One of the significant improvements in Bio-formats 5.0 over 4.x is its independence from AWT which requires access to a UI. So using the old code should fail and using the new code should succeed. There’s some chance that reinstalling did not remove the Bio-formats 4.x jars and these are being used - these are in the directory, “/usr/cellprofiler/src/CellProfiler/imagej”. Could you check to see if your install has anything extra? The latest RPM has the following jars:

/usr/cellprofiler/src/CellProfiler/imagej/jars/JWlz-1.4.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/asm-4.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/asm-analysis-4.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/asm-commons-4.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/asm-tree-4.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/asm-util-4.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/base64-2.3.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/bcmail-jdk14-138.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/bcprov-jdk14-138.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/blitz.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/bsh-2.0b4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/bytelist-1.0.11.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/c3p0-0.9.1.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/cellprofiler-dependencies-classpath.txt
/usr/cellprofiler/src/CellProfiler/imagej/jars/cellprofiler-java-dependencies-classpath.txt
/usr/cellprofiler/src/CellProfiler/imagej/jars/cellprofiler-java.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/clojure-1.3.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/common.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/commons-codec-1.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/commons-httpclient-3.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/commons-lang3-3.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ehcache-core-2.6.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/eventbus-1.4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/formats-api-5.0.0-rc4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/formats-bsd-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/formats-common-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/formats-gpl-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/forms-1.2.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ice-glacier2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ice-grid.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ice-storm.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ice.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-1.48s.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-app-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-commands-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-core-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-data-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-legacy-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-options-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-platforms-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-scripting-beanshell-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-scripting-clojure-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-scripting-java-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-scripting-javascript-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-scripting-jruby-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-scripting-jython-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-text-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-tools-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-ui-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-ui-common-awt-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-ui-swing-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-ui-swing-commands-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-ui-swing-script-editor-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-ui-swing-tools-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-ui-swing-updater-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-updater-core-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-updater-ssh-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-updater-webdav-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-util-awt-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij-util-swing-2.0.0-beta-7.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ij1-patcher-0.2.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/imglib2-2.0.0-beta-25.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/imglib2-algorithms-2.0.0-beta-25.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/imglib2-ij-2.0.0-beta-25.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/imglib2-meta-2.0.0-beta-25.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/imglib2-ops-2.0.0-beta-25.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/invokebinder-1.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/itext-2.1.5.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jai_imageio-1.0.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jai_imageio-5.0.0-rc4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jama-1.0.3.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/javassist-3.16.1-GA.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/javax.json-1.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jcip-annotations-1.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jcl-over-slf4j-1.7.5.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jcodings-1.0.10.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jcommander-1.27.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jcommon-1.0.17.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jdatepicker-1.3.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jdom2-2.0.4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/je-4.0.92.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jffi-1.2.7-native.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jffi-1.2.7.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jfreechart-1.0.14.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jhotdraw-7.6.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jna-3.0.9.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jnr-constants-0.8.5.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jnr-enxio-0.4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jnr-ffi-1.0.7.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jnr-netdb-1.1.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jnr-posix-3.0.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jnr-unixsocket-0.3.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jnr-x86asm-1.0.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/joda-time-2.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/joda-time-2.3.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/joni-2.1.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jruby-1.7.11.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jruby-complete-1.7.11.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jruby-core-1.7.11.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jruby-stdlib-1.7.11.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jsch-0.1.49.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/junit-4.8.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jython-2.5.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/jzlib-1.1.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/kryo-2.21-shaded.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/loci-legacy-5.0.0-rc2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/logback-classic-1.0.9.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/logback-core-1.0.9.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/lwf-stubs-5.0.0-rc4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/mapdb-0.9.7.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/markdownj-0.3.0-1.0.2b4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/mdbtools-java-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/metakit-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/miglayout-3.7.3.1-swing.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/mines-jtk-20100113.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/minimaven-1.0.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/model-psql.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/nailgun-server-0.9.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/native-lib-loader-2.0-SNAPSHOT.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/netcdf-4.3.19.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/object-inspector-0.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ome-io-5.0.0-rc2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ome-java-2.6.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ome-poi-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/ome-xml-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/opencsv-2.3.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/perf4j-0.9.13.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/protobuf-java-2.5.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/quartz-2.1.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/rhino-1.7R4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/rsyntaxtextarea-2.0.4.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/scifio-0.10.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/scifio-5.0.0-rc2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/scifio-devel-5.0.0-rc2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/scijava-common-2.11.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/serializer-2.7.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/slf4j-api-1.7.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/snakeyaml-1.13.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/snakeyaml-1.6.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/specification-5.0.0-rc4.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/testng-6.8.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/tools-1.4.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/turbojpeg-5.0.0.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/udunits-4.3.18.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/udunits-4.3.19.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/xalan-2.7.1.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/xercesImpl-2.6.2.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/xml-apis-1.3.04.jar
/usr/cellprofiler/src/CellProfiler/imagej/jars/yecht-1.0.jar

Dear Lee, David.

Thank you very much for your help regarding this. Meanwhile, I actually found the source of my issue. Pretty simple thing, actually, but it took some time. :smile: In loadimages.py the jp2 file ending is not accepted. Line 155 contains j2k, most probably your definition of jpeg2000 encoded images. When I renamed my image file endings to j2k, the pipeline runs. To my knowledge jp2 is the standardized file type for JPEG2000 files (see e.g. en.wikipedia.org/wiki/Jpeg2000). It would be really nice if you could include this ending in loadimages.py. If I understood correctly, the loadimages.py module corresponds to the CP2.0 legacy mode. When you tested it on your on installation, you probably created a pipeline running the “new” way of loading images and then extracted the CP pipeline including the image locations. I guess there, jp2 file extensions are accepted, which also explains why I could run them using the GUI.

The way we set up CP here is the following: we developed an automatic process to upload images from our imaging machines into a database, and then start feature calculation jobs on our cluster straightaway. Here, one feature calculation job corresponds to a set of images within one well. The CP call is using always the same prepared pipeline, but each job gets a different image folder as input. This seems not to work anymore with version 2.1, if I understood correctly, as headless runs require now that the exact location of the images needs to be annotated in the pipeline file. I saw it would be possible to create the corresponding image tags on-the-fly and add them to the CP pipeline file before running it, but it would be so much easier to keep just one pipeline file and direct the input image folder to CP with -i , as it is working with version 2.0. Is there a particular reason why you decided to change that?

I guess somewhere in your python code an exception is thrown in case sth is wrong within the pipeline (eg image loading in my case). Despite running the headless call with an -L DEBUG, an error message was not visible in my case. This is the respective output:

Created temporary file /tmp/Cpmeasurements_n4Eg_.hdf5 /usr/cellprofiler/src/CellProfiler/CellProfiler.py: (844 <module>): main(sys.argv) /usr/cellprofiler/src/CellProfiler/CellProfiler.py: (251 main): run_pipeline_headless(options, args) /usr/cellprofiler/src/CellProfiler/CellProfiler.py: (828 run_pipeline_headless): initial_measurements = initial_measurements) /usr/cellprofiler/src/CellProfiler/cellprofiler/pipeline.py: (1614 run): copy = initial_measurements) /usr/cellprofiler/src/CellProfiler/cellprofiler/measurements.py: (270 __init__): for frame in traceback.extract_stack(): HDF5Dict.__init__(): /tmp/Cpmeasurements_n4Eg_.hdf5, temporary=True, copy=None, mode=w HDF5Dict.flush(): /tmp/Cpmeasurements_n4Eg_.hdf5, temporary=True HDF5Dict.flush(): /tmp/Cpmeasurements_n4Eg_.hdf5, temporary=True HDF5Dict.flush(): /tmp/Cpmeasurements_n4Eg_.hdf5, temporary=True HDF5Dict.__del__(): /tmp/Cpmeasurements_n4Eg_.hdf5, temporary=True stopping worker thread 0 stopping worker thread 1 Exiting the JVM monitor thread

I would suggest, if I may, to forward exceptions to the DEBUG log, to help with the identification of issues within modules or a pipeline.

On a sidenote, just to answer Lee’s question, my imageJ jars folder contains exactly the mentioned jars files. So the upgrade seems not to keep old files.

Thank you very much and keep up the good work! I really like this project very much!

Best regards,

Matthias

Thanks Matthias,
I’ll check today with LoadImages - probably this is a regression, not an intended change in behavior. I think the way you’re approaching things is reasonable and should be supported, but if it’s possible to use LoadData instead of LoadImages, that’s preferred and will always have first-class support.

–Lee

Ouch - nice catch, you’re absolutely right, the extension list was the problem and you solved it. I’ll make that change momentarily. We’re releasing today - it’s a bit of extra work to retag the release, build and test. Is it ok for you to get this change off of the trunk builds page or similar?

–Lee

I guess it will be included in the next tag (2.1.1.rc5), right? So yes, no worries, I’ll manage. It was more important to understand and solve this issue in general, rather than having it solved and fixed within the same day. :smile:

Thanks again,
Matthias

This issue has been addressed with the 2.1.1 release.
-MArk