Pipeline unable to find the IlumXX.npy for illumination correction


I have a pipeline that perform illumination correction and segmentation. Similar to the one the “advance segmentation” from the tutorial Headless CellProfiler/ DistributedCellProfiler Tutorial.
I have a load_data_with_illum.csv that contains the necessary info to load the .tiff images, and the .npy (calculated beforehand).
When I run the pipeline headless in my machine, It works. However, when I tried to run it in the cloud (GCP), using pretty much the same headless command, It doesn’t work. And I keep getting the following error:

Error detected during run of module CorrectIlluminationApply
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/cellprofiler_core/pipeline/_pipeline.py”, line 958, in run_with_yield
self.run_module(module, workspace)
File “/usr/local/lib/python3.8/dist-packages/cellprofiler_core/pipeline/_pipeline.py”, line 1275, in run_module
File “/usr/local/lib/python3.8/dist-packages/cellprofiler/modules/correctilluminationapply.py”, line 233, in run
self.run_image(image, workspace)
File “/usr/local/lib/python3.8/dist-packages/cellprofiler/modules/correctilluminationapply.py”, line 249, in run_image
illum_function = workspace.image_set.get_image(illum_correct_name)
File “/usr/local/lib/python3.8/dist-packages/cellprofiler_core/measurement/_measurements.py”, line 1534, in get_image
raise ValueError(
ValueError: The IllumER image is missing from the pipeline.

What have I tried already?

  • I’ve verified and re-verified that the .npy files were there, and the path in the load_csv_will_illum.csv were the correct ones, But still it doesn’t find those images.

-If I uncheck the illumination correction from the pipeline, save it, and upload it, the pipeline run without any issue. Meaning that it can find the tiff images. But for some reason it can not find the .npy

Any thoughts?

Thank you in advance,

Can you share your pipeline and load data CSV? Thanks!

Also, it would be great if you can report the exact command (ie cellprofiler -c -r -p etc ) you are using to execute this.

Thank you Beth,

Find below the command and the files used.
In my VM home directory ($csv_dir) I have the file load_data_with_illum.csv.
And two folders: /data with the .tiff images, and /illum with the .npy
It works fine if I use a cell profiler pipeline that do not perform illumination correction.

The command, and the files used:
# run cellprofiler pipeline
cellprofiler --run --run-headless
-p ~{cppipe_file}
-o output
-i $csv_dir

load_data_with_illum.csv (1.5 KB)
AnalysisPipeline_AddedMeasurements.cppipe (43.8 KB)

Thank you so much!

Can you try running with, rather than -i $csv_dir, --data-file=$csv_dir/load_data_with_illum.csv ?

If that doesn’t fix it, the issue may be with your CSV itself; if you compare your columns, you’ll see that the ones not for illum have headers of FileName_Chan and PathName_Chan, and the ones for illum have headers of FileNameChan and PathNameChan . It’s possible/likely CellProfiler can’t parse the latter structure.

1 Like

Thank you so much Beth!
The name of the columns was the issue.
I will look into the append_illum_cols.py script and see if I can fix it there, because that csv was automatically generated using an edited version of the script from your repo. I sure did something.

Thanks again and have a wonderful long weekend!