Java errors while running on linux cluster

dougb · April 22, 2016, 10:03pm

I recently installed CellProfiler in a linux cluster environment. After reading the documentation for batch processing I am running jobs that analyse a single well’s worth of images per job. The pipeline seems to run and it writes .csv files according to the settings of the ExportToSpreadsheet module, but I see many java Caught exceptions reported to standard output while it is running.

The java exceptions are of two types. The first are reported immediately after the cellprofiler command is executed and the second type are reported for each image that is analyzed. The text of these exceptions are listed below.

My question is will these exceptions alter the data that I will get from the analysis or can I safely ignore them?

Exceptions reported for each image:

Fri Apr 22 17:14:55 2016: Image # 3, module Images # 1: 0.00 sec
Fri Apr 22 17:14:55 2016: Image # 3, module Metadata # 2: 0.00 sec
Warning:  org.apache.xerces.parsers.SAXParser: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized.
Exception in thread "Thread-0" java.io.EOFException: Attempting to read beyond end of file.
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
	at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
	at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:506)
	at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
	... 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread "Thread-0" java.lang.IllegalArgumentException
	at java.nio.Buffer.position(Buffer.java:244)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
	at loci.formats.tiff.TiffParser.<init>(TiffParser.java:113)
	at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread "Thread-0" java.io.EOFException: Attempting to read beyond end of file.
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
	at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
	at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:506)
	at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
	... 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread "Thread-0" java.lang.IllegalArgumentException
	at java.nio.Buffer.position(Buffer.java:244)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
	at loci.formats.tiff.TiffParser.<init>(TiffParser.java:113)
	at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread "Thread-0" java.io.EOFException: Attempting to read beyond end of file.
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
	at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
	at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:506)
	at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
	... 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread "Thread-0" java.lang.IllegalArgumentException
	at java.nio.Buffer.position(Buffer.java:244)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
	at loci.formats.tiff.TiffParser.<init>(TiffParser.java:113)
	at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread "Thread-0" java.io.EOFException: Attempting to read beyond end of file.
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
	at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
	at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:506)
	at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
	... 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread "Thread-0" java.lang.IllegalArgumentException
	at java.nio.Buffer.position(Buffer.java:244)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
	at loci.formats.tiff.TiffParser.<init>(TiffParser.java:113)
	at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Exception in thread "Thread-0" java.io.EOFException: Attempting to read beyond end of file.
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
	at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
	at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:506)
	at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
	... 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread "Thread-0" java.lang.IllegalArgumentException
	at java.nio.Buffer.position(Buffer.java:244)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
	at loci.formats.tiff.TiffParser.<init>(TiffParser.java:113)
	at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Fri Apr 22 17:14:55 2016: Image # 3, module NamesAndTypes # 3: 3.19 sec
Fri Apr 22 17:14:58 2016: Image # 3, module Groups # 4: 0.00 sec
Fri Apr 22 17:14:58 2016: Image # 3, module IdentifyPrimaryObjects # 5: 14.08 sec
Fri Apr 22 17:15:13 2016: Image # 3, module IdentifySecondaryObjects # 6: 7.32 sec
Fri Apr 22 17:15:20 2016: Image # 3, module IdentifyTertiaryObjects # 7: 2.05 sec
Fri Apr 22 17:15:22 2016: Image # 3, module MeasureObjectIntensity # 10: 26.76 sec
Fri Apr 22 17:15:50 2016: Image # 3, module MeasureObjectSizeShape # 15: 132.56 sec
Fri Apr 22 17:18:03 2016: Image # 3, module ExportToSpreadsheet # 17: 0.00 sec
Fri Apr 22 17:18:03 2016: Image # 3, module CreateBatchFiles # 18: 0.00 sec

Exceptions reported immediately after executing cellprofiler command:

> cellprofiler -p /data/hd/analysis/rs/HD42m1/Batch_data.h5 -c -r -f 10 -l 18 -o /data/hd/analysis/rs/HD42m1/2
> Plugin directory doesn't point to valid folder: /home/plugins
> java.lang.IllegalArgumentException: Cannot handle replace call to list in ij.Menus's public static synchronized java.lang.String[] getPlugins()
> 	at net.imagej.patcher.CodeHacker.replaceCallInMethod(CodeHacker.java:725)
> 	at net.imagej.patcher.CodeHacker.replaceCallInMethod(CodeHacker.java:650)
> 	at net.imagej.patcher.LegacyExtensions.addExtraPlugins(LegacyExtensions.java:477)
> 	at net.imagej.patcher.LegacyExtensions.injectHooks(LegacyExtensions.java:175)
> 	at net.imagej.patcher.LegacyInjector.inject(LegacyInjector.java:308)
> 	at net.imagej.patcher.LegacyInjector.injectHooks(LegacyInjector.java:109)
> 	at net.imagej.patcher.LegacyEnvironment.initialize(LegacyEnvironment.java:101)
> 	at net.imagej.patcher.LegacyEnvironment.applyPatches(LegacyEnvironment.java:495)
> 	at net.imagej.patcher.LegacyInjector.preinit(LegacyInjector.java:397)
> 	at net.imagej.patcher.LegacyInjector.preinit(LegacyInjector.java:376)
> 	at net.imagej.patcher.JavaAgent.preinit(JavaAgent.java:147)
> 	at net.imagej.patcher.JavaAgent.premain(JavaAgent.java:95)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
> 	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
> Caused by: java.lang.RuntimeException: java.io.IOException: invalid constant type: 18
> 	at javassist.CtClassType.getClassFile2(CtClassType.java:203)
> 	at javassist.compiler.MemberResolver.lookupMethod(MemberResolver.java:110)
> 	at javassist.compiler.MemberResolver.lookupMethod(MemberResolver.java:96)
> 	at javassist.compiler.TypeChecker.atMethodCallCore(TypeChecker.java:704)
> 	at javassist.compiler.TypeChecker.atCallExpr(TypeChecker.java:681)
> 	at javassist.compiler.JvstTypeChecker.atCallExpr(JvstTypeChecker.java:156)
> 	at javassist.compiler.ast.CallExpr.accept(CallExpr.java:45)
> 	at javassist.compiler.CodeGen.doTypeCheck(CodeGen.java:241)
> 	at javassist.compiler.CodeGen.atStmnt(CodeGen.java:329)
> 	at javassist.compiler.ast.Stmnt.accept(Stmnt.java:49)
> 	at javassist.compiler.CodeGen.atIfStmnt(CodeGen.java:390)
> 	at javassist.compiler.CodeGen.atStmnt(CodeGen.java:354)
> 	at javassist.compiler.ast.Stmnt.accept(Stmnt.java:49)
> 	at javassist.compiler.Javac.compileStmnt(Javac.java:568)
> 	at javassist.expr.MethodCall.replace(MethodCall.java:234)
> 	at net.imagej.patcher.CodeHacker$7.edit(CodeHacker.java:686)
> 	at javassist.expr.ExprEditor.loopBody(ExprEditor.java:191)
> 	at javassist.expr.ExprEditor.doit(ExprEditor.java:90)
> 	at javassist.CtBehavior.instrument(CtBehavior.java:618)
> 	at net.imagej.patcher.CodeHacker$EagerExprEditor.instrument(CodeHacker.java:1278)
> 	at net.imagej.patcher.CodeHacker.replaceCallInMethod(CodeHacker.java:670)
> 	... 17 more
> Caused by: java.io.IOException: invalid constant type: 18
> 	at javassist.bytecode.ConstPool.readOne(ConstPool.java:1027)
> 	at javassist.bytecode.ConstPool.read(ConstPool.java:970)
> 	at javassist.bytecode.ConstPool.<init>(ConstPool.java:127)
> 	at javassist.bytecode.ClassFile.read(ClassFile.java:693)
> 	at javassist.bytecode.ClassFile.<init>(ClassFile.java:85)
> 	at javassist.CtClassType.getClassFile2(CtClassType.java:190)
> 	... 37 more
> java.lang.IllegalArgumentException: Cannot handle app name in ij.ImageJ's public <init>(java.applet.Applet applet, int mode)
> 	at net.imagej.patcher.CodeHacker.replaceAppNameInCall(CodeHacker.java:446)
> 	at net.imagej.patcher.LegacyExtensions.insertAppNameHooks(LegacyExtensions.java:406)
> 	at net.imagej.patcher.LegacyExtensions.injectHooks(LegacyExtensions.java:291)
> 	at net.imagej.patcher.LegacyInjector.inject(LegacyInjector.java:308)
> 	at net.imagej.patcher.LegacyInjector.injectHooks(LegacyInjector.java:109)
> 	at net.imagej.patcher.LegacyEnvironment.initialize(LegacyEnvironment.java:101)
> 	at net.imagej.patcher.LegacyEnvironment.applyPatches(LegacyEnvironment.java:495)
> 	at net.imagej.patcher.LegacyInjector.preinit(LegacyInjector.java:397)
> 	at net.imagej.patcher.LegacyInjector.preinit(LegacyInjector.java:376)
> 	at net.imagej.patcher.JavaAgent.preinit(JavaAgent.java:147)
> 	at net.imagej.patcher.JavaAgent.premain(JavaAgent.java:95)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
> 	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
> Caused by: java.lang.RuntimeException: java.io.IOException: invalid constant type: 18
> 	at javassist.CtClassType.getClassFile2(CtClassType.java:203)
> 	at javassist.CtClassType.subtypeOf(CtClassType.java:303)
> 	at javassist.CtClassType.subtypeOf(CtClassType.java:318)
> 	at javassist.compiler.MemberResolver.compareSignature(MemberResolver.java:247)
> 	at javassist.compiler.MemberResolver.lookupMethod(MemberResolver.java:119)
> 	at javassist.compiler.MemberResolver.lookupMethod(MemberResolver.java:96)
> 	at javassist.compiler.TypeChecker.atMethodCallCore(TypeChecker.java:704)
> 	at javassist.compiler.TypeChecker.atCallExpr(TypeChecker.java:681)
> 	at javassist.compiler.JvstTypeChecker.atCallExpr(JvstTypeChecker.java:156)
> 	at javassist.compiler.ast.CallExpr.accept(CallExpr.java:45)
> 	at javassist.compiler.CodeGen.doTypeCheck(CodeGen.java:241)
> 	at javassist.compiler.CodeGen.atStmnt(CodeGen.java:329)
> 	at javassist.compiler.ast.Stmnt.accept(Stmnt.java:49)
> 	at javassist.compiler.Javac.compileStmnt(Javac.java:568)
> 	at javassist.expr.MethodCall.replace(MethodCall.java:234)
> 	at net.imagej.patcher.CodeHacker$4.edit(CodeHacker.java:427)
> 	at net.imagej.patcher.CodeHacker$4.edit(CodeHacker.java:441)
> 	at javassist.expr.ExprEditor.loopBody(ExprEditor.java:219)
> 	at javassist.expr.ExprEditor.doit(ExprEditor.java:90)
> 	at javassist.CtBehavior.instrument(CtBehavior.java:618)
> 	at net.imagej.patcher.CodeHacker$EagerExprEditor.instrument(CodeHacker.java:1278)
> 	at net.imagej.patcher.CodeHacker.replaceAppNameInCall(CodeHacker.java:402)
> 	... 16 more
> Caused by: java.io.IOException: invalid constant type: 18
> 	at javassist.bytecode.ConstPool.readOne(ConstPool.java:1027)
> 	at javassist.bytecode.ConstPool.read(ConstPool.java:970)
> 	at javassist.bytecode.ConstPool.<init>(ConstPool.java:127)
> 	at javassist.bytecode.ClassFile.read(ClassFile.java:693)
> 	at javassist.bytecode.ClassFile.<init>(ClassFile.java:85)
> 	at javassist.CtClassType.getClassFile2(CtClassType.java:190)
> 	... 37 more
> Version: 2016-04-22T21:02:41 Unknown rev. / 20160422210241
> Failed to stop Ilastik
> Pipeline saved with CellProfiler version 20140723173957
> vigra import: failed to import the vigra library. Please follow the instructions on
> "http://hci.iwr.uni-heidelberg.de/vigra/" to install vigra
> Traceback (most recent call last):
>   File "/home/anaconda2/lib/python2.7/site-packages/cellprofiler/modules/classifypixels.py", line 50, in <module>
>     import vigra
> ImportError: No module named vigra
> vigra import: failed to import the vigra library. Please follow the instructions on
> "http://hci.iwr.uni-heidelberg.de/vigra/" to install vigra
> Traceback (most recent call last):
>   File "/home/anaconda2/lib/python2.7/site-packages/cellprofiler/modules/ilastik_pixel_classification.py", line 47, in <module>
>     import vigra
> ImportError: No module named vigra
> Could not load cellprofiler.modules.exporttocellh5
> Traceback (most recent call last):
>   File "/home/anaconda2/lib/python2.7/site-packages/cellprofiler/modules/__init__.py", line 294, in add_module
>     m = __import__(mod, globals(), locals(), ['__all__'], 0)
>   File "/home/anaconda2/lib/python2.7/site-packages/cellprofiler/modules/exporttocellh5.py", line 29, in <module>
>     import cellh5
>   File "/home/anaconda2/lib/python2.7/site-packages/cellh5/__init__.py", line 12, in <module>
>     from cellh5 import *
>   File "/home/anaconda2/lib/python2.7/site-packages/cellh5/cellh5.py", line 18, in <module>
>     import matplotlib.pyplot as plt
>   File "/home/anaconda2/lib/python2.7/site-packages/matplotlib/pyplot.py", line 114, in <module>
>     _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
>   File "/home/anaconda2/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
>     globals(),locals(),[backend_name],0)
>   File "/home/anaconda2/lib/python2.7/site-packages/matplotlib/backends/backend_wxagg.py", line 11, in <module>
>     from . import wx_compat as wxc
>   File "/home/anaconda2/lib/python2.7/site-packages/matplotlib/backends/wx_compat.py", line 24, in <module>
>     raise ImportError(missingwx)
> ImportError: Matplotlib backend_wx and backend_wxagg require wxPython >=2.8.12
> could not load these modules: cellprofiler.modules.exporttocellh5
> Times reported are CPU times for each module, not wall-clock time

karhohs · May 10, 2016, 6:51pm

@dougb There is a known issue with a couple modules within CP that throw java errors, specifically the RunImageJ and ClassifyPixels modules. Most modules will work without a problem. Feel free to share your pipeline, and I can take a look just to make sure.