Error in Headless Run on a High Performance Cluster

cellprofiler

#1

Hi there!

I’m running CellProfiler version 3.0.0rc1 (it appears this is a currently unreleased version?), obtained from the GitHub within an Anaconda installation, on a CentOS7 based High Performance Cluster.

We want to analyse microscope slides in the form of tiles using CellProfiler. To this extent, I’m using a pipeline designed in a previous version of CellProfiler. I want to keep using the LoadImages module instead of the new design. I was having problems reading in files and on this forum, someone with a similar problem was advised to keep using the LoadImages module, as it still works and is an easier solution when processing directories of files via the command line.

When I run this from the most recent beta version (2.3) on my macbook, it runs without problems. When I run the pipeline via the command line on the cluster, I get the error below.

I fear the error may have something to do with differences between the versions I am currently running. Due to various restrictions and peculiarities on our cluster set up, we have experienced great difficulty installing CellProfiler on it for our group (and now we have finally gotten it to work). For this reason, I wanted to ask if someone could help me solve this problem, hoping for an easy fix, and that installing another version of CellProfiler on the cluster is not necessary.

Thanks!

Below, the pipeline file and the error generated:

CellProfiler Pipeline: http://www.cellprofiler.org
Version:3
DateRevision:20160308191757
GitHash:b91da8e
ModuleCount:13
HasImagePlaneDetails:False

LoadImages:[module_num:1|svn_version:\'Unknown\'|variable_revision_number:11|show_window:False|notes:\x5B\'# Pipeline created for HPC (b.g.l.nelissen@umcutrecht.nl)\', \'# Goal\x3A Measure total tissue area and DAB area\', \'# Outlines\x3A Tissue blue, DAB red\', \'\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    File type to be loaded:individual images
    File selection method:Text-Exact match
    Number of images in each group?:3
    Type the text that the excluded images have in common:.counted
    Analyze all subfolders within the selected folder?:All
    Input image file location:Default Input Folder\x7CNone
    Check image sets for unmatched or duplicate files?:No
    Group images by metadata?:No
    Exclude certain files?:Yes
    Specify metadata fields to group by:
    Select subfolders to analyze:
    Image count:1
    Text that these images have in common (case-sensitive):tissue
    Position of this image in each group:1
    Extract metadata from where?:File name
    Regular expression that finds metadata in the file name:^(?P<STUDYNR>\x5B^.\x5D*)\\\\.(?P<STAIN>\x5B^.\x5D*).*\\\\.X(?P<X>\x5B0-9\x5D{1,4}).*\\\\.Y(?P<Y>\x5B0-9\x5D{1,4})
    Type the regular expression that finds metadata in the subfolder path:.*\x5B\\\\\\\\/\x5D(?P<Date>.*)\x5B\\\\\\\\/\x5D(?P<Run>.*)$
    Channel count:1
    Group the movie frames?:No
    Grouping method:Interleaved
    Number of channels per group:3
    Load the input as images or objects?:Images
    Name this loaded image:Original
    Name this loaded object:Nuclei
    Retain outlines of loaded objects?:No
    Name the outline image:NucleiOutlines
    Channel number:1
    Rescale intensities?:No

ColorToGray:[module_num:2|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Create Grayscale image for finding total tissue area\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Select the input image:Original
    Conversion method:Combine
    Image type:RGB
    Name the output image:OrigGray
    Relative weight of the red channel:1
    Relative weight of the green channel:1
    Relative weight of the blue channel:1
    Convert red to gray?:Yes
    Name the output image:OrigRed
    Convert green to gray?:Yes
    Name the output image:OrigGreen
    Convert blue to gray?:Yes
    Name the output image:OrigBlue
    Convert hue to gray?:Yes
    Name the output image:OrigHue
    Convert saturation to gray?:Yes
    Name the output image:OrigSaturation
    Convert value to gray?:Yes
    Name the output image:OrigValue
    Channel count:1
    Channel number:Red\x3A 1
    Relative weight of the channel:1
    Image name:Channel1

Smooth:[module_num:3|svn_version:\'Unknown\'|variable_revision_number:2|show_window:False|notes:\x5B\'Smooth grayscale image to remove dust and small open holes\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Select the input image:OrigGray
    Name the output image:OrigGraySmooth
    Select smoothing method:Gaussian Filter
    Calculate artifact diameter automatically?:No
    Typical artifact diameter:20
    Edge intensity difference:0.1
    Clip intensities to 0 and 1?:Yes

ImageMath:[module_num:4|svn_version:\'Unknown\'|variable_revision_number:4|show_window:False|notes:\x5B\'Invert grayscale image for futher analysis\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Operation:Invert
    Raise the power of the result by:1
    Multiply the result by:1
    Add to result:0
    Set values less than 0 equal to 0?:Yes
    Set values greater than 1 equal to 1?:Yes
    Ignore the image masks?:No
    Name the output image:Inverted
    Image or measurement?:Image
    :OrigGraySmooth
    :1
    Measurement:
    Image or measurement?:Image
    :
    :1
    Measurement:

IdentifyPrimaryObjects:[module_num:5|svn_version:\'Unknown\'|variable_revision_number:10|show_window:False|notes:\x5B\'Find tissue using the grayscale image.\', \'No use of magick here; white is background, non-white is foreground.\', \'From now on, outlined foreground is called Tissue\', \'Diameter is choosen after some testing\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Select the input image:Inverted
    Name the primary objects to be identified:Tissue
    Typical diameter of objects, in pixel units (Min,Max):40,99999999
    Discard objects outside the diameter range?:Yes
    Try to merge too small objects with nearby larger objects?:No
    Discard objects touching the border of the image?:No
    Method to distinguish clumped objects:None
    Method to draw dividing lines between clumped objects:Intensity
    Size of smoothing filter:10
    Suppress local maxima that are closer than this minimum allowed distance:7
    Speed up by using lower-resolution image to find local maxima?:Yes
    Name the outline image:Tissue
    Fill holes in identified objects?:Never
    Automatically calculate size of smoothing filter for declumping?:Yes
    Automatically calculate minimum allowed distance between local maxima?:Yes
    Retain outlines of the identified objects?:Yes
    Automatically calculate the threshold using the Otsu method?:Yes
    Enter Laplacian of Gaussian threshold:0.5
    Automatically calculate the size of objects for the Laplacian of Gaussian filter?:Yes
    Enter LoG filter diameter:5
    Handling of objects if excessive number of objects identified:Continue
    Maximum number of objects:500
    Threshold setting version:1
    Threshold strategy:Manual
    Thresholding method:Otsu
    Select the smoothing method for thresholding:Automatic
    Threshold smoothing scale:1
    Threshold correction factor:0.9
    Lower and upper bounds on threshold:0.000000,1.000000
    Approximate fraction of image covered by objects?:0.01
    Manual threshold:0.05
    Select the measurement to threshold with:None
    Select binary image:None
    Masking objects:From image
    Two-class or three-class thresholding?:Three classes
    Minimize the weighted variance or the entropy?:Weighted variance
    Assign pixels in the middle intensity class to the foreground or the background?:Foreground
    Method to calculate adaptive window size:Image size
    Size of adaptive window:10
    Use default parameters?:Default
    Lower outlier fraction:0.05
    Upper outlier fraction:0.05
    Averaging method:Mean
    Variance method:Standard deviation
    # of deviations:2.0

MeasureImageAreaOccupied:[module_num:6|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Measure all Tissue areas\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Hidden:1
    Measure the area occupied in a binary image, or in objects?:Objects
    Select objects to measure:Tissue
    Retain a binary image of the object regions?:No
    Name the output binary image:Stain
    Select a binary image to measure:Tissue

UnmixColors:[module_num:7|svn_version:\'Unknown\'|variable_revision_number:2|show_window:True|notes:\x5B\'Unmix the colors, and extract DAB\', \'Create one grayscale DAB channel\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Stain count:1
    Select the input color image:Original
    Name the output name:DAB
    Stain:DAB
    Red absorbance:0.363092
    Green absorbance:0.651435
    Blue absorbance:0.666181

MaskImage:[module_num:8|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Use selected Tissue object to create a mask\', "Only analyse what\'s in the \'Tissue\' area"\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Select the input image:DAB
    Name the output image:DAB_masked
    Use objects or an image as a mask?:Objects
    Select object for mask:Tissue
    Select image for mask:Tissue
    Invert the mask?:No

IdentifyPrimaryObjects:[module_num:9|svn_version:\'Unknown\'|variable_revision_number:10|show_window:False|notes:\x5B"Find DAB positive area\'s", \'You need to test which thresholding method works best for you. Read the help menu for more info.\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Select the input image:DAB_masked
    Name the primary objects to be identified:DAB_area
    Typical diameter of objects, in pixel units (Min,Max):8,4000000
    Discard objects outside the diameter range?:Yes
    Try to merge too small objects with nearby larger objects?:No
    Discard objects touching the border of the image?:No
    Method to distinguish clumped objects:None
    Method to draw dividing lines between clumped objects:Intensity
    Size of smoothing filter:10
    Suppress local maxima that are closer than this minimum allowed distance:7
    Speed up by using lower-resolution image to find local maxima?:Yes
    Name the outline image:DAB_area
    Fill holes in identified objects?:Never
    Automatically calculate size of smoothing filter for declumping?:Yes
    Automatically calculate minimum allowed distance between local maxima?:Yes
    Retain outlines of the identified objects?:Yes
    Automatically calculate the threshold using the Otsu method?:Yes
    Enter Laplacian of Gaussian threshold:0.5
    Automatically calculate the size of objects for the Laplacian of Gaussian filter?:Yes
    Enter LoG filter diameter:5
    Handling of objects if excessive number of objects identified:Continue
    Maximum number of objects:500
    Threshold setting version:1
    Threshold strategy:Global
    Thresholding method:RobustBackground
    Select the smoothing method for thresholding:Automatic
    Threshold smoothing scale:1
    Threshold correction factor:1
    Lower and upper bounds on threshold:0,1.0
    Approximate fraction of image covered by objects?:0.01
    Manual threshold:0.8
    Select the measurement to threshold with:None
    Select binary image:None
    Masking objects:From image
    Two-class or three-class thresholding?:Two classes
    Minimize the weighted variance or the entropy?:Weighted variance
    Assign pixels in the middle intensity class to the foreground or the background?:Background
    Method to calculate adaptive window size:Image size
    Size of adaptive window:10
    Use default parameters?:Default
    Lower outlier fraction:0.05
    Upper outlier fraction:0.05
    Averaging method:Mean
    Variance method:Standard deviation
    # of deviations:2.0

MeasureImageAreaOccupied:[module_num:10|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Measure all DAB areas (within Tissue)\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Hidden:1
    Measure the area occupied in a binary image, or in objects?:Objects
    Select objects to measure:DAB_area
    Retain a binary image of the object regions?:No
    Name the output binary image:Stain
    Select a binary image to measure:Tissue

OverlayOutlines:[module_num:11|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Create outlines for visualisation.\', \'The Tissue outline has a thick blue like\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Display outlines on a blank image?:No
    Select image on which to display outlines:Original
    Name the output image:Orig_Overlay
    Outline display mode:Color
    Select method to determine brightness of outlines:Max of image
    Width of outlines:4
    Select outlines to display:Tissue
    Select outline color:Blue
    Load outlines from an image or objects?:Image
    Select objects to display:None

OverlayOutlines:[module_num:12|svn_version:\'Unknown\'|variable_revision_number:3|show_window:False|notes:\x5B\'Create outlines for visualisation.\', \'The DAB outline has a thin green line\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Display outlines on a blank image?:No
    Select image on which to display outlines:Orig_Overlay
    Name the output image:Orig_Overlay
    Outline display mode:Color
    Select method to determine brightness of outlines:Max of image
    Width of outlines:1
    Select outlines to display:DAB_area
    Select outline color:Red
    Load outlines from an image or objects?:Image
    Select objects to display:None

SaveImages:[module_num:13|svn_version:\'Unknown\'|variable_revision_number:11|show_window:False|notes:\x5B\'Save original images with outlines so we can visually check what the pipeline did.\'\x5D|batch_state:array(\x5B\x5D, dtype=uint8)|enabled:True|wants_pause:False]
    Select the type of image to save:Image
    Select the image to save:Orig_Overlay
    Select the objects to save:None
    Select the module display window to save:None
    Select method for constructing file names:From image filename
    Select image name for file prefix:Original
    Enter single file name:OrigBlue
    Number of digits:4
    Append a suffix to the image file name?:Yes
    Text to append to the image name:.overlay
    Saved file format:png
    Output file location:Default Output Folder\x7CNone
    Image bit depth:8-bit integer
    Overwrite existing files without warning?:Yes
    When to save:Every cycle
    Rescale the images? :No
    Save as grayscale or color image?:Grayscale
    Select colormap:gray
    Record the file and path information to the saved image?:No
    Create subfolders in the output folder?:No
    Base image folder:Default Input Folder
    Saved movie format:avi

Your pipeline version is 20160308191757 but you are running CellProfiler version 300. Loading this pipeline may fail or have unpredictable results.
Times reported are CPU times for each module, not wall-clock time
Exception in thread "Thread-0" java.io.EOFException: Attempting to read beyond end of file.
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:350)
	at loci.common.RandomAccessInputStream.readFloat(RandomAccessInputStream.java:501)
	at loci.formats.in.PQBinReader.isThisType(PQBinReader.java:96)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:506)
	at java.nio.HeapByteBuffer.getFloat(HeapByteBuffer.java:463)
	at loci.common.NIOFileHandle.readFloat(NIOFileHandle.java:348)
	... 2 more
Caught exception in class loci.formats.in.PQBinReader.isThisType
Exception in thread "Thread-0" java.lang.IllegalArgumentException
	at java.nio.Buffer.position(Buffer.java:244)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:603)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
	at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
	at loci.formats.tiff.TiffParser.<init>(TiffParser.java:113)
	at loci.formats.in.FlowSightReader.isThisType(FlowSightReader.java:101)
Caught exception in class loci.formats.in.FlowSightReader.isThisType
Thu Mar  9 15:57:09 2017: Image # 1, module LoadImages # 1: 1.72 sec
Thu Mar  9 15:57:11 2017: Image # 1, module ColorToGray # 2: 0.19 sec
Thu Mar  9 15:57:11 2017: Image # 1, module Smooth # 3: 0.98 sec
Thu Mar  9 15:57:12 2017: Image # 1, module ImageMath # 4: 0.01 sec
Thu Mar  9 15:57:12 2017: Image # 1, module IdentifyPrimaryObjects # 5: 2.07 sec
Thu Mar  9 15:57:14 2017: Image # 1, module MeasureImageAreaOccupied # 6: 0.08 sec
Thu Mar  9 15:57:15 2017: Image # 1, module UnmixColors # 7: 0.67 sec
Thu Mar  9 15:57:15 2017: Image # 1, module MaskImage # 8: 0.03 sec
Thu Mar  9 15:57:15 2017: Image # 1, module IdentifyPrimaryObjects # 9: 2.01 sec
Thu Mar  9 15:57:17 2017: Image # 1, module MeasureImageAreaOccupied # 10: 0.07 sec
Error detected during run of module OverlayOutlines
Traceback (most recent call last):
  File "/hpc/local/CentOS7/dhl_ec/software/CellProfiler/src/CellProfiler/src/cellprofiler-master/cellprofiler/pipeline.py", line 1753, in run_with_yield
    self.run_module(module, workspace)
  File "/hpc/local/CentOS7/dhl_ec/software/CellProfiler/src/CellProfiler/src/cellprofiler-master/cellprofiler/pipeline.py", line 1997, in run_module
    module.run(workspace)
  File "/hpc/local/CentOS7/dhl_ec/software/CellProfiler/src/CellProfiler/src/cellprofiler-master/cellprofiler/modules/overlayoutlines.py", line 202, in run
    pixel_data = self.run_color(workspace, base_image)
  File "/hpc/local/CentOS7/dhl_ec/software/CellProfiler/src/CellProfiler/src/cellprofiler-master/cellprofiler/modules/overlayoutlines.py", line 302, in run_color
    objects = workspace.object_set.get_objects(outline.objects_name.value)
  File "/hpc/local/CentOS7/dhl_ec/software/CellProfiler/src/CellProfiler/src/cellprofiler-master/cellprofiler/object.py", line 803, in get_objects
    return self.__objects_by_name[name]
KeyError: u'None'
Thu Mar  9 15:57:17 2017: Image # 1, module OverlayOutlines # 11: 0.01 sec