Memory on CP2, ExportToSpreadsheet

Hi,
I got something that looks like a memory problem in the middle of execution where Export to spreadsheet should be taking place. A while later at image 79 I got a time-out from the cluster. I am not sure if the two are related, but any help with this will be greatly appreciated. I imagine you call Java there, but what memory parameter do you use, and if this needs reset, where do I do that? Apparently the cluster assigns 1GB per job, unless specified differently, and I suppose setting this up for Java to use up to 1GB needs to be done somewhere within CP2. What is the default java memory setup you have put into CP2? What size of file should I expect from the analysis of 203 images?
Regards
Ela

Tue Mar 8 17:56:07 2011: Image # 75, module CreateBatchFiles # 8: 0.00 sec (bg)
Tue Mar 8 17:56:07 2011: Image # 76, module LoadImages # 1: 0.21 sec (bg)
Tue Mar 8 17:56:08 2011: Image # 76, module IdentifyPrimaryObjects # 2: 6.26 sec (bg)
Tue Mar 8 17:56:14 2011: Image # 76, module MeasureObjectIntensity # 3: 0.39 sec (bg)
Tue Mar 8 17:56:14 2011: Image # 76, module MeasureObjectSizeShape # 4: 14.84 sec (bg)
Tue Mar 8 17:56:29 2011: Image # 76, module TrackObjects # 5: 31.87 sec (bg)
Tue Mar 8 17:57:01 2011: Image 2011-03-08 17:59:16
Full thread dump Java HotSpot™ 64-Bit Server VM (16.3-b01 mixed mode):

“Thread-0” prio=10 tid=0x0000000017ff4800 nid=0x1b5b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

“Low Memory Detector” daemon prio=10 tid=0x0000000017fef000 nid=0x1b79 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

“CompilerThread1” daemon prio=10 tid=0x0000000017fed000 nid=0x1b78 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

“CompilerThread0” daemon prio=10 tid=0x0000000017fe8000 nid=0x1b77 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

“Signal Dispatcher” daemon prio=10 tid=0x0000000017fe5800 nid=0x1b76 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

“Finalizer” daemon prio=10 tid=0x0000000017fc2000 nid=0x1b75 in Object.wait() [0x0000000041663000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaac8f811e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaac8f811e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

“Reference Handler” daemon prio=10 tid=0x0000000017fc0000 nid=0x1b74 in Object.wait() [0x0000000041562000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaac8f81048> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaac8f81048> (a java.lang.ref.Reference$Lock)

“main” prio=10 tid=0x0000000017ec9800 nid=0x1b70 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

“VM Thread” prio=10 tid=0x0000000017fbb800 nid=0x1b73 runnable

“GC task thread#0 (ParallelGC)” prio=10 tid=0x0000000017ece800 nid=0x1b71 runnable

“GC task thread#1 (ParallelGC)” prio=10 tid=0x0000000017ed0800 nid=0x1b72 runnable

“VM Periodic Task Thread” prio=10 tid=0x0000000017ff2000 nid=0x1b7a waiting on condition

76, module ExportToSpreadsheet # 6: 0.00 sec

Tue Mar 8 17:57:01 2011: Image # 76, module SaveImages # 7: 0.09 sec (bg)
Tue Mar 8 17:57:02 2011: Image # 76, module CreateBatchFiles # 8: 0.00 sec (bg)
PipelineWithExport1.cp (6.96 KB)
logbrutus.cp (54.3 KB)

Hi Ela, how are you?

CellProfiler is running the JVM and the JVM hooks into the native exception signaling mechanism and ends up reporting any unhandled signals (like illegal memory access). I have a feeling that the Java stack dump is a red herring and that the problem is caused by a problem in CP, possibly in your case (Linux?) a miscompiled module?

You can override the default JVM memory size when you start CP with the --jvm-heap-size switch which is the same format as for Java, for instance, “cellprofiler --jvm-heap-size 4g”.

–Lee