JavaExceptions with cellprofiler-3.1.8 and batch processing

I’m testing out batch processing on my local Mac before I move on to needlessly clogging CPUs on a larger server.
I’ve loaded the entire test data (76 image sets) into CP, added the CreateBatchFiles module, and generated the Batch_data.h5 file.

I then quit CP, and ran the following command:

/Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp \
-p /Users/nieder/Documents/CellProfiler/Batch_data.h5  \
-o /Users/nieder/Documents/UTSW/8KTestSubset/CellProfiler/original_CP  \
-c -r  -f 1 -l 76

Headless CellProfiler starts up and then throws exceptions and errors:

nieder $ /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp -p /Users/nieder/Documents/CellProfiler/Batch_data.h5  -o /Users/nieder/Documents/UTSW/8KTestSubset/CellProfiler/original_CP  -c -r  -f 1 -l 76
[5803] PyInstaller Bootloader 3.x
[5803] LOADER: executable is /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp
[5803] LOADER: homepath is /Applications/CellProfiler-3.1.8.app/Contents/MacOS
[5803] LOADER: _MEIPASS2 is NULL
[5803] LOADER: archivename is /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp
[5803] LOADER: /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp contains a digital signature
[5803] LOADER: No need to extract files to run; setting extractionpath to homepath
[5803] LOADER: Already in the child - running user's code.
[5803] LOADER: Python library: /Applications/CellProfiler-3.1.8.app/Contents/MacOS/Python
[5803] LOADER: Loaded functions from Python library.
[5803] LOADER: Manipulating environment (sys.path, sys.prefix)
[5803] LOADER: sys.prefix is /Applications/CellProfiler-3.1.8.app/Contents/MacOS
[5803] LOADER: Setting runtime options
[5803] LOADER: Initializing python
[5803] LOADER: Overriding Python's sys.path
[5803] LOADER: Post-init sys.path is /Applications/CellProfiler-3.1.8.app/Contents/MacOS
[5803] LOADER: Setting sys.argv
[5803] LOADER: setting sys._MEIPASS
[5803] LOADER: importing modules from CArchive
[5803] LOADER: extracted struct
[5803] LOADER: callfunction returned...
[5803] LOADER: extracted pyimod01_os_path
[5803] LOADER: callfunction returned...
[5803] LOADER: extracted pyimod02_archive
[5803] LOADER: callfunction returned...
[5803] LOADER: extracted pyimod03_importers
[5803] LOADER: callfunction returned...
[5803] LOADER: Installing PYZ archive with Python modules.
[5803] LOADER: PYZ archive: out00-PYZ.pyz
[5803] LOADER: Running pyiboot01_bootstrap.py
[5803] LOADER: Running pyi_rth_traitlets.py
[5803] LOADER: Running pyi_rth_mplconfig.py
[5803] LOADER: Running pyi_rth_mpldata.py
[5803] LOADER: Running pyi_rth_pkgres.py
[5803] LOADER: Running pyi_rth_multiprocessing.py
[5803] LOADER: Running CellProfiler.py
/Applications/CellProfiler-3.1.8.app/Contents/MacOS/cellprofiler/utilities/hdf5_dict.py:539: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
/Applications/CellProfiler-3.1.8.app/Contents/MacOS/cellprofiler/utilities/hdf5_dict.py:541: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
font search path ['/Applications/CellProfiler-3.1.8.app/Contents/MacOS/mpl-data/fonts/ttf', '/Applications/CellProfiler-3.1.8.app/Contents/MacOS/mpl-data/fonts/afm', '/Applications/CellProfiler-3.1.8.app/Contents/MacOS/mpl-data/fonts/pdfcorefonts']
Could not open font file /Library/Fonts/NISC18030.ttf
generated new fontManager
Times reported are CPU and Wall-clock times for each module
Fri May  1 20:35:02 2020: Image # 1, module Images # 1: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Fri May  1 20:35:02 2020: Image # 1, module Metadata # 2: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Exception in thread "Thread-1" org.mozilla.javascript.WrappedException: Wrapped javax.xml.transform.TransformerException: org.xml.sax.SAXException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces. (<java-python-bridge>#2)
	at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1754)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:148)
	at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)
	at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
	at org.mozilla.javascript.gen._java_python_bridge__3._c_script_0(<java-python-bridge>:2)
	at org.mozilla.javascript.gen._java_python_bridge__3.call(<java-python-bridge>)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
	at org.mozilla.javascript.gen._java_python_bridge__3.call(<java-python-bridge>)
	at org.mozilla.javascript.gen._java_python_bridge__3.exec(<java-python-bridge>)
	at org.mozilla.javascript.Context.evaluateString(Context.java:1079)
Caused by: javax.xml.transform.TransformerException: org.xml.sax.SAXException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
	at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:469)
	at org.cellprofiler.imageset.ImageSet.decompress(ImageSet.java:177)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	... 9 more
Caused by: org.xml.sax.SAXException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
	at org.apache.xml.utils.DOMBuilder.startElement(DOMBuilder.java:351)
	at org.apache.xalan.transformer.TransformerIdentityImpl.startElement(TransformerIdentityImpl.java:1020)
	at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
	at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
	at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
	at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:452)
	... 15 more
Caused by: org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
	at org.apache.xerces.dom.AttrNSImpl.setName(Unknown Source)
	at org.apache.xerces.dom.AttrNSImpl.<init>(Unknown Source)
	at org.apache.xerces.dom.CoreDocumentImpl.createAttributeNS(Unknown Source)
	at org.apache.xerces.dom.ElementImpl.setAttributeNS(Unknown Source)
	at org.apache.xml.utils.DOMBuilder.startElement(DOMBuilder.java:336)
	... 26 more
Error detected during run of module NamesAndTypes
Traceback (most recent call last):
  File "cellprofiler/pipeline.py", line 1782, in run_with_yield
  File "cellprofiler/pipeline.py", line 2034, in run_module
  File "cellprofiler/modules/namesandtypes.py", line 1729, in run
  File "cellprofiler/modules/namesandtypes.py", line 1677, in get_imageset
  File "javabridge/jutil.py", line 394, in run_script
JavaException: org.xml.sax.SAXException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces.
Fri May  1 20:35:02 2020: Image # 1, module NamesAndTypes # 3: CPU_time = 0.33 secs, Wall_time = 0.21 secs
[5803] LOADER: OK.
[5803] LOADER: Cleaning up Python interpreter.

Thoughts on what’s broken and how to fix it? The pipeline runs great in GUI mode.

Hmmm, I wonder if CellProfiler isn’t properly accessing the copy of Java that’s inside its build environment.

Do you have Java otherwise installed on your machine? What is the output of java -version in your terminal? What about echo $JAVA_HOME

If JAVA_HOME comes back blank, can you try temporarily setting JAVA_HOME with export JAVA_HOME=/Applications/CellProfiler-3.1.8.app/Contents/Resources/Home on that machine, confirm that java -version comes back then looking like the below, and try again your command (or another I’ve provided below), in that same terminal window? Otherwise, can you tell us what JAVA_HOME on your machine comes back as?

In any case, as long as JAVA_HOME is set properly on your server, you’re unlikely to run into these issues there, regardless.

openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

/Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp -c -r -p /Applications/CellProfiler-3.1.8.app/Contents/Resources/cellprofiler/data/examples/ExampleFly/ExampleFly.cppipe -i /Applications/CellProfiler-3.1.8.app/Contents/Resources/cellprofiler/data/examples/ExampleFly/images/ -o ~/Desktop/

I did have JAVA_HOME set to something else. If I set it to CP’s internal JAVA_HOME, java -version reports version 11.0.1. When I try to run my command above with the new JAVA_HOME, the error is different:

$ printenv JAVA_HOME
/Applications/CellProfiler-3.1.8.app/Contents/Resources/Home
$ [Bubble:~/Documents/UTSW/8KTestSubset] nieder $ /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp -p /Users/nieder/Documents/UTSW/8KTestSubset/CellProfiler/Batch_data.h5 -o /Users/nieder/Documents/UTSW/8KTestSubset/CellProfiler/original_CP -c -r  -f 1 -l 76
[55826] PyInstaller Bootloader 3.x
[55826] LOADER: executable is /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp
[55826] LOADER: homepath is /Applications/CellProfiler-3.1.8.app/Contents/MacOS
[55826] LOADER: _MEIPASS2 is NULL
[55826] LOADER: archivename is /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp
[55826] LOADER: /Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp contains a digital signature
[55826] LOADER: No need to extract files to run; setting extractionpath to homepath
[55826] LOADER: Already in the child - running user's code.
[55826] LOADER: Python library: /Applications/CellProfiler-3.1.8.app/Contents/MacOS/Python
[55826] LOADER: Loaded functions from Python library.
[55826] LOADER: Manipulating environment (sys.path, sys.prefix)
[55826] LOADER: sys.prefix is /Applications/CellProfiler-3.1.8.app/Contents/MacOS
[55826] LOADER: Setting runtime options
[55826] LOADER: Initializing python
[55826] LOADER: Overriding Python's sys.path
[55826] LOADER: Post-init sys.path is /Applications/CellProfiler-3.1.8.app/Contents/MacOS
[55826] LOADER: Setting sys.argv
[55826] LOADER: setting sys._MEIPASS
[55826] LOADER: importing modules from CArchive
[55826] LOADER: extracted struct
[55826] LOADER: callfunction returned...
[55826] LOADER: extracted pyimod01_os_path
[55826] LOADER: callfunction returned...
[55826] LOADER: extracted pyimod02_archive
[55826] LOADER: callfunction returned...
[55826] LOADER: extracted pyimod03_importers
[55826] LOADER: callfunction returned...
[55826] LOADER: Installing PYZ archive with Python modules.
[55826] LOADER: PYZ archive: out00-PYZ.pyz
[55826] LOADER: Running pyiboot01_bootstrap.py
[55826] LOADER: Running pyi_rth_traitlets.py
[55826] LOADER: Running pyi_rth_mplconfig.py
[55826] LOADER: Running pyi_rth_mpldata.py
[55826] LOADER: Running pyi_rth_pkgres.py
[55826] LOADER: Running pyi_rth_multiprocessing.py
[55826] LOADER: Running CellProfiler.py
/Applications/CellProfiler-3.1.8.app/Contents/MacOS/cellprofiler/utilities/hdf5_dict.py:539: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.
/Applications/CellProfiler-3.1.8.app/Contents/MacOS/cellprofiler/utilities/hdf5_dict.py:541: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
font search path ['/Applications/CellProfiler-3.1.8.app/Contents/MacOS/mpl-data/fonts/ttf', '/Applications/CellProfiler-3.1.8.app/Contents/MacOS/mpl-data/fonts/afm', '/Applications/CellProfiler-3.1.8.app/Contents/MacOS/mpl-data/fonts/pdfcorefonts']
Could not open font file /Library/Fonts/NISC18030.ttf
generated new fontManager
Times reported are CPU and Wall-clock times for each module
Mon May  4 17:28:13 2020: Image # 1, module Images # 1: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Mon May  4 17:28:13 2020: Image # 1, module Metadata # 2: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Exception in thread "Thread-1" java.lang.NoClassDefFoundError: org/w3c/dom/ls/DocumentLS
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:802)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:700)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:623)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at org.apache.xerces.jaxp.DocumentBuilderImpl.<init>(Unknown Source)
	at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
	at org.apache.xalan.transformer.TransformerIdentityImpl.createResultContentHandler(TransformerIdentityImpl.java:204)
	at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:296)
	at org.cellprofiler.imageset.ImageSet.decompress(ImageSet.java:177)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
	at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)
	at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
	at org.mozilla.javascript.gen._java_python_bridge__3._c_script_0(<java-python-bridge>:2)
	at org.mozilla.javascript.gen._java_python_bridge__3.call(<java-python-bridge>)
	at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
	at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
	at org.mozilla.javascript.gen._java_python_bridge__3.call(<java-python-bridge>)
	at org.mozilla.javascript.gen._java_python_bridge__3.exec(<java-python-bridge>)
	at org.mozilla.javascript.Context.evaluateString(Context.java:1079)
Caused by: java.lang.ClassNotFoundException: org.w3c.dom.ls.DocumentLS
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 28 more
Error detected during run of module NamesAndTypes
Traceback (most recent call last):
  File "cellprofiler/pipeline.py", line 1782, in run_with_yield
  File "cellprofiler/pipeline.py", line 2034, in run_module
  File "cellprofiler/modules/namesandtypes.py", line 1729, in run
  File "cellprofiler/modules/namesandtypes.py", line 1677, in get_imageset
  File "javabridge/jutil.py", line 384, in run_script
  File "javabridge/jutil.py", line 887, in call
  File "javabridge/jutil.py", line 854, in fn
JavaException: org/w3c/dom/ls/DocumentLS
Mon May  4 17:28:13 2020: Image # 1, module NamesAndTypes # 3: CPU_time = 0.26 secs, Wall_time = 0.18 secs
[55826] LOADER: OK.
[55826] LOADER: Cleaning up Python interpreter.

If I try to run your ExampleFly.cppipe with no JAVA_HOME (or my original JAVA_HOME), it fails. If I set JAVA_HOME to the internal java instance, then ExampleFly.cppipe runs OK.

Do I have something wrong with my pipeline and workflow? I have a CPPROJ file with the last module being ExportToSpreadsheet (manually running it at this point from the GUI works OK). I added the CreateBatchFiles module at the end. I dragged my images into the Images module. Then clicked “Analyze Images”, which created Batch_data.h5. I did not save the CPPROJ file with the loaded images or the CreateBatchFiles after creating the Batch_data.h5 file (since I thought those were contained in the exported h5 file). I then ran the above command.

What is the “something else” your JAVA_HOME was pointed to- is it a version of Java pre-8 or post-11?

AFAICT, everything you did was exactly right, and I have no problem on my own Mac running the command I sent, or just dragging in the cp file with my JAVA_HOME set to something outside the executable- I in fact demo doing just that about 9.5 minutes into this video.

If your CreateBatchFiles module is set up to change your paths at all, that would certainly account for the error, if the new paths actually aren’t correct for your machine; are you able to run headless CellProfiler commands pointing to a pipeline file without CreateBatchFiles in it? Or if you point it to one WITH CreateBatchFiles in it, do you get a batch file out where you expect it?

My system-installed is java8 from oracle. I had JAVA_HOME set because /usr/bin/java was pointing to an old minor version (1.8.0_66) and not the latest java8 that java-update had installed in another directory.

Loading my cpproj file and pressing “view output settings”:
image
My images are in subfolders below that input folder, and the Batch_data.h5 file was saved to that CellProfiler folder listed in the Output settings when I added the CreateBatchFiles module and ran it from the GUI.

… (after some offline testing)…

So I’m not exactly sure what happened, but it’s working now. I went back into the CPPROJ file and rechecked the input/output folders by reselecting them in the folder selection dialog box, and resaved the file. I then ran the this command basically mimicking what you had in that video:

/Applications/CellProfiler-3.1.8.app/Contents/MacOS/cp -c -r \
-p /Users/nieder/Documents/UTSW/8KTestSubset/SC219\ STING\ ORIGINAL.cpproj \
-i /Users/nieder/Documents/UTSW/8KTestSubset/images \
-o /Users/nieder/Desktop/jnkCP -f 1 -l 40

And it ran and analyzed properly. Hand batching other subsets (in parallel) of the images also worked. It’s even working with my JAVA_HOME location still set. The CPPROJ file originally came from Windows, but the output folder had been edited in the ExportToSpreadsheet folder since the Mac is just my test environment. Maybe there was some Windows path still left behind that disagreed with the Mac.

1 Like