Cellprofiler_v4.0.4 batch processing

Usage & Issues
#1

Hi CP team,
We’ve been running cellprofiler_v3.1.3 on our computing cluster for a few years without any issues. IT have just installed cellprofiler_v4.0.4 for us, but when I try to submit the jobs to the cluster I’m getting the following error:
File “/config/binaries/miniconda3/4.8.3/envs/cellprofiler_v4.0.4/lib/python3.8/site-packages/cellprofiler_core/image/abstract_image/file/_file_image.py”, line 159, in cache_file
raise IOError(
OSError: Test for access to directory failed. Directory: ///W:/CS_DATA_SHARE_03/2020Oct26/A50363_201029120001
Fri Nov 27 08:41:00 2020: Image # 1, module NamesAndTypes # 3: CPU_time = 0.55 secs, Wall_time = 0.49 secs

Something seems to be going wrong when the Batch_data.h5 is created because CellProfiler is looking for the images in the local root path rather than in the cluster root path. I’m using the exact same setup that we’ve always used for v3. Any idea what could be happening? I’ll attach the pipeline and a copy of the sbatch script and output log files.

Thank you!
Karla
Batch-6414636_1.txt (3.0 KB) illum_pipeline_example.cppipe (6.1 KB) image_list sbatch_illum.txt (1.2 KB)

#2

Hi @Karla ,

Thanks for reporting this. This might be a strange suggestion, but in your CreateBatchFiles module would you be able to try swapping \ separators with / instead? So W:\CS_DATA_SHARE_03\2020Oct26 becomes W:/CS_DATA_SHARE_03/2020Oct26. If that gets it working then I think I might have a better fix we can put into the next release.

#3

Hi @DStirling,
Thanks for the reply! Didn’t solve it, unfortunately! I’ve attached the updated pipeline and the error.
Karla

Batch-7053756_1.txt (3.2 KB) illum_pipeline_example2.cppipe (6.1 KB)

#4

Hi @Karla,

Thanks for doing that, is there any chance you could upload the .cpproj file instead of .cppipe? This will have your local image path names embedded in it, which might help with testing this. You may need to compress that into a .zip archive to upload directly onto the forum.

#5

Hi @DStirling,
Sure! Thank you, that would be great!
Karla
CP4_test.zip (1.0 MB)

#6

Hmm, it seems like I can get the replacement working if I take that latest pipeline and switch the / back to \ in the Windows paths, but I can’t get the same to work when loading the original illum pipeline with those settings already present. I’m not sure if the file may have been corrupted somehow, but you might want to try updating the paths in that way just to be sure.