LoadImages failure: too many open files

Hi,

I’m running CP2.0 on a linux cluster. When I try to generate my batch file though, I’m encountering this error:

Exception in thread “Thread-0” java.io.FileNotFoundException: /home/morphology/Yolanda/Willis/BODIPY/BODIPY_plate02s/010021000_plate02s.flex (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at loci.common.NIOFileHandle.(NIOFileHandle.java:100)
at loci.common.NIOFileHandle.(NIOFileHandle.java:111)
at loci.common.NIOFileHandle.(NIOFileHandle.java:119)
at loci.common.Location.getHandle(Location.java:284)
at loci.common.Location.getHandle(Location.java:255)
at loci.formats.ImageReader.getReader(ImageReader.java:153)
at loci.formats.ImageReader.setId(ImageReader.java:665)
Traceback (most recent call last):
File “/usr/cp2/CellProfiler/cellprofiler/pipeline.py”, line 1218, in prepare_run
File “/usr/cp2/CellProfiler/cellprofiler/modules/loadimages.py”, line 1357, in prepare_run
File “/usr/cp2/CellProfiler/cellprofiler/modules/loadimages.py”, line 1779, in prepare_run_of_flex
File “/usr/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 439, in method
File “/usr/cp2/CellProfiler/cellprofiler/utilities/jutil.py”, line 377, in call
JavaException: /home/morphology/Yolanda/Willis/BODIPY/BODIPY_plate02s/010021000_plate02s.flex (Too many open files)

I’ve attached the pipeline file. The biologists I work with who have used earlier CP versions haven’t encountered this bug before. Any ideas?
test_pipeline.cp (10.6 KB)

We’re using CellProfiler SVN revision 10833, in case that helps.

Can you tell me what prior version of CP they were using? Was it CP 1.0 or an earlier svn version of CP 2.0?
-Mark

Hi Mark,

It was some version of 1.0 (I think CellProfiler 7746). We’ve only recently installed CP 2.0 on our cluster, and are still working out the kinks.

Edit: Also, going through the code I find that the reported version in CellProfiler.py is 10789, while the output suggests otherwise:

Subversion revision: 10833 Pipeline saved with CellProfiler SVN revision 10833 0 [Thread-0] WARN loci.formats.FormatHandler - mismatch between image count, names and factors (count=4, names=8, factors=8)

Hope this is illuminating,

Lee.

Hi Lee,

We haven’t been able to replicate the error that you reported as yet. However, we found that flex file handling was incomplete for batch creation as it is, so we have checked in a fix in the svn repository. Try updating to the newest version (r10892), re-running the pipeline and let us know if you get the same error.

Cheers,
-Mark

Thanks for your report. I needed to make some repairs to LoadImages to get CreateBatchFiles to work with flex files. I’d appreciate you trying the latest (SVN revision 10892). You will need to update both the client and server code bases for CellProfiler to this revision to get the fix to work. Trunk builds for Windows and the Mac are available at http://cellprofiler.org/releases/.

Hi Mark,

Updated successfully, but no change unfortunately. The stack trace (below),

Exception in thread "Thread-0" java.io.FileNotFoundException: /home/morphology/Yolanda/Willis/BODIPY/BODIPY_plate02s/010021000_plate02s.flex (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:100) at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:111) at loci.common.NIOFileHandle.<init>(NIOFileHandle.java:119) at loci.common.Location.getHandle(Location.java:284) at loci.common.Location.getHandle(Location.java:255) at loci.formats.ImageReader.getReader(ImageReader.java:153) at loci.formats.ImageReader.setId(ImageReader.java:665) Traceback (most recent call last): File "/usr/cp2/CellProfiler/cellprofiler/pipeline.py", line 1218, in prepare_run File "/usr/cp2/CellProfiler/cellprofiler/modules/loadimages.py", line 1357, in prepare_run File "/usr/cp2/CellProfiler/cellprofiler/modules/loadimages.py", line 1779, in prepare_run_of_flex File "/usr/cp2/CellProfiler/cellprofiler/utilities/jutil.py", line 439, in method File "/usr/cp2/CellProfiler/cellprofiler/utilities/jutil.py", line 377, in call JavaException: /home/morphology/Yolanda/Willis/BODIPY/BODIPY_plate02s/010021000_plate02s.flex (Too many open files)

tells me that this java exception is caused by somenot closing file descriptors that have been processed. Is there some way of debugging CP to keep a count of how many open file descriptors are present when generating a batch file? I’ve done a quick search, and found some similar problems http://www.vnoel.com/Computers-Related-Issues/Java-problem-Too-many-open-files.html

Thanks,

Is it possible for you to send us the flex file that’s causing the problem (/home/morphology/Yolanda/Willis/BODIPY/BODIPY_plate02s/010021000_plate02s.flex)? The message that you’re getting is from a library we use to open the files. It would help us diagnose the problem.

Thanks,

–Lee

Sure, how should I send it to you?

[quote=“LeeKamentsky”]Is it possible for you to send us the flex file that’s causing the problem (/home/morphology/Yolanda/Willis/BODIPY/BODIPY_plate02s/010021000_plate02s.flex)? The message that you’re getting is from a library we use to open the files. It would help us diagnose the problem.

Thanks,

–Lee[/quote]

I’ve tried to attach the flex file here, but I after a long delay I’m taken to a page which says

Can I just email it to you, or upload it somewhere?

Thanks,

[quote=“LeeKamentsky”]Is it possible for you to send us the flex file that’s causing the problem (/home/morphology/Yolanda/Willis/BODIPY/BODIPY_plate02s/010021000_plate02s.flex)? The message that you’re getting is from a library we use to open the files. It would help us diagnose the problem.

Thanks,

–Lee[/quote]

I just private messaged you about the file upload.
-Mark

Hi Lee,
I have updated the file “cellprofiler/modules/loadimages.py” (SVN revision 10913) to explicitly close the flex file in all cases. I can’t recreate the bug on my system; I’m guessing that the problem might be operating-system specific (I’m testing with Windows), but what I fixed is a possible cause of the problem you’re seeing.

The trunk builds for SVN revision 10913 should be ready in a few minutes at http://cellprofiler.org/releases/ and should include the fix. If you’re running from source, the url for loadimages is https://svn.broadinstitute.org/CellProfiler/trunk/CellProfiler/cellprofiler/modules/loadimages.py

–Lee

Great, thanks Lee. I’ll test this weekend or early next week, hopefully this does the trick.

[quote=“LeeKamentsky”]Hi Lee,
I have updated the file “cellprofiler/modules/loadimages.py” (SVN revision 10913) to explicitly close the flex file in all cases. I can’t recreate the bug on my system; I’m guessing that the problem might be operating-system specific (I’m testing with Windows), but what I fixed is a possible cause of the problem you’re seeing.

The trunk builds for SVN revision 10913 should be ready in a few minutes at http://cellprofiler.org/releases/ and should include the fix. If you’re running from source, the url for loadimages is https://svn.broadinstitute.org/CellProfiler/trunk/CellProfiler/cellprofiler/modules/loadimages.py

–Lee[/quote]

I was able to generate a batch file with the whole data set, so hopefully this did the trick. Now I just need to figure out how to run these as smaller jobs submitted to our cluster.

Thanks for all your help,

Lee.

[quote=“LeeZamparo”]Great, thanks Lee. I’ll test this weekend or early next week, hopefully this does the trick.

[quote=“LeeKamentsky”]Hi Lee,
I have updated the file “cellprofiler/modules/loadimages.py” (SVN revision 10913) to explicitly close the flex file in all cases. I can’t recreate the bug on my system; I’m guessing that the problem might be operating-system specific (I’m testing with Windows), but what I fixed is a possible cause of the problem you’re seeing.

The trunk builds for SVN revision 10913 should be ready in a few minutes at http://cellprofiler.org/releases/ [/quote]

and should include the fix. If you’re running from source, the url for loadimages is https://svn.broadinstitute.org/CellProfiler/trunk/CellProfiler/cellprofiler/modules/loadimages.py

–Lee[/quote]