Too many open files error

Hi folks,

I’ve got an error in a pipeline that I’m running for a collaborator (see attached), and am getting this too many open files error, where the input files are .flex files. I’m running version 11570 from SVN. I recall having encountered a flex file open files error with a prior version of CP2 before. Is it possible that somewhere in the file processing code that .flex files are not being closed properly? The sample .flex images are too large to attach, so I’ve stored one at http://www.cs.toronto.edu/~zamparo/images/sample_image_sets/003001000.flex

Here’s the output error message:

[quote]Tue Oct 18 14:21:42 2011: Image # 191, module LoadImages # 1: 0.93 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:43 2011: Image # 191, module RescaleIntensity # 2: 0.49 sec (bg)
Tue Oct 18 14:21:43 2011: Image # 191, module RescaleIntensity # 2: 0.49 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:43 2011: Image # 191, module RescaleIntensity # 3: 0.48 sec (bg)
Tue Oct 18 14:21:43 2011: Image # 191, module RescaleIntensity # 3: 0.48 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:44 2011: Image # 191, module MakeProjection # 4: 0.03 sec (bg)
Tue Oct 18 14:21:44 2011: Image # 191, module MakeProjection # 4: 0.03 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:44 2011: Image # 191, module MakeProjection # 5: 0.04 sec (bg)
Tue Oct 18 14:21:44 2011: Image # 191, module MakeProjection # 5: 0.04 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:44 2011: Image # 191, module GrayToColor # 6: 0.40 sec (bg)
Tue Oct 18 14:21:44 2011: Image # 191, module GrayToColor # 6: 0.40 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:45 2011: Image # 191, module GrayToColor # 7: 0.41 sec (bg)
Tue Oct 18 14:21:45 2011: Image # 191, module GrayToColor # 7: 0.41 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:45 2011: Image # 191, module GrayToColor # 8: 0.58 sec (bg)
Tue Oct 18 14:21:45 2011: Image # 191, module GrayToColor # 8: 0.58 sec (bg)
INFO:PipelineStatistics:Tue Oct 18 14:21:46 2011: Image # 191, module SaveImages # 9: 0.02 sec
Tue Oct 18 14:21:46 2011: Image # 191, module SaveImages # 9: 0.02 sec
INFO:PipelineStatistics:Tue Oct 18 14:21:46 2011: Image # 191, module SaveImages # 10: 0.02 sec
Tue Oct 18 14:21:46 2011: Image # 191, module SaveImages # 10: 0.02 sec
INFO:PipelineStatistics:Tue Oct 18 14:21:46 2011: Image # 191, module SaveImages # 11: 0.02 sec
Tue Oct 18 14:21:46 2011: Image # 191, module SaveImages # 11: 0.02 sec
Exception in thread “main” java.io.FileNotFoundException: /mnt/morphology/Estyles/Silvia_Pombe/2011_Sept27_PombeTest_3/Meas_01(2011-09-27_09-32-04)/003010000.flex (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:216)
at loci.common.NIOFileHandle.(NIOFileHandle.java:100)
at loci.common.NIOFileHandle.(NIOFileHandle.java:111)
at loci.common.NIOFileHandle.(NIOFileHandle.java:119)
at loci.common.Location.getHandle(Location.java:284)
at loci.common.Location.getHandle(Location.java:255)
at loci.common.RandomAccessInputStream.(RandomAccessInputStream.java:80)
at loci.formats.in.FlexReader.groupFiles(FlexReader.java:1055)
at loci.formats.in.FlexReader.initFlexFile(FlexReader.java:481)
at loci.formats.in.FlexReader.initFile(FlexReader.java:333)
at loci.formats.FormatReader.setId(FormatReader.java:1072)
ERROR:cellprofiler.utilities.jutil:Caught exception when executing closure
Traceback (most recent call last):
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 299, in synchronous_closure
File “/home/kfounk/cp2/CellProfiler/bioformats/formatreader.py”, line 377, in fn
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 502, in method
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 440, in call
JavaException: /mnt/morphology/Estyles/Silvia_Pombe/2011_Sept27_PombeTest_3/Meas_01(2011-09-27_09-32-04)/003010000.flex (Too many open files)
Caught exception when executing closure
Traceback (most recent call last):
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 299, in synchronous_closure
File “/home/kfounk/cp2/CellProfiler/bioformats/formatreader.py”, line 377, in fn
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 502, in method
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 440, in call
JavaException: /mnt/morphology/Estyles/Silvia_Pombe/2011_Sept27_PombeTest_3/Meas_01(2011-09-27_09-32-04)/003010000.flex (Too many open files)
WARNING:cellprofiler.pipeline:Intercepted exception while running module
Traceback (most recent call last):
File “/home/kfounk/cp2/CellProfiler/cellprofiler/pipeline.py”, line 328, in run
File “/home/kfounk/cp2/CellProfiler/cellprofiler/modules/loadimages.py”, line 2186, in run
File “/home/kfounk/cp2/CellProfiler/cellprofiler/modules/loadimages.py”, line 3075, in provide_image
File “/home/kfounk/cp2/CellProfiler/bioformats/formatreader.py”, line 469, in load_using_bioformats
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 308, in run_in_main_thread
JavaException: /mnt/morphology/Estyles/Silvia_Pombe/2011_Sept27_PombeTest_3/Meas_01(2011-09-27_09-32-04)/003010000.flex (Too many open files)
Intercepted exception while running module
Traceback (most recent call last):
File “/home/kfounk/cp2/CellProfiler/cellprofiler/pipeline.py”, line 328, in run
File “/home/kfounk/cp2/CellProfiler/cellprofiler/modules/loadimages.py”, line 2186, in run
File “/home/kfounk/cp2/CellProfiler/cellprofiler/modules/loadimages.py”, line 3075, in provide_image
File “/home/kfounk/cp2/CellProfiler/bioformats/formatreader.py”, line 469, in load_using_bioformats
File “/home/kfounk/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 308, in run_in_main_thread
JavaException: /mnt/morphology/Estyles/Silvia_Pombe/2011_Sept27_PombeTest_3/Meas_01(2011-09-27_09-32-04)/003010000.flex (Too many open files)[/quote]

MakeProjection_GraytoColor_SaveImages.cp (10.3 KB)

The following change may fix this. It’s in r11574, but you could also just apply it yourself.

Index: bioformats/formatreader.py

— bioformats/formatreader.py (revision 11573)
+++ bioformats/formatreader.py (revision 11574)
@@ -456,6 +456,7 @@
image.shape = (height,width)

     rdr.close()
  •    jutil.call(stream, 'close', '()V')
       del rdr
       #
       # Run the Java garbage collector here.

This fix seems to have done the trick. Thanks Thouis!

Lee.

[quote=“thouis”]The following change may fix this. It’s in r11574, but you could also just apply it yourself.

Index: bioformats/formatreader.py

— bioformats/formatreader.py (revision 11573)
+++ bioformats/formatreader.py (revision 11574)
@@ -456,6 +456,7 @@
image.shape = (height,width)

     rdr.close()
  •    jutil.call(stream, 'close', '()V')
       del rdr
       #
       # Run the Java garbage collector here.[/quote]