Starfish threading issue with local_max_peak_finder.py

Hi,

I’m trying to use starfish-0.2.1 to find spots in my images, but I get a strange threading error when I try to run the max peakfinder:

AttributeError: '_thread._local' object has no attribute 'state'

The error occurs when I call local_max_peakfinder.run.

I’ve tried to downgrade my version of Starfish, but this didn’t help either.

Here is the code, if it helps:

import numpy as np
import imageio
from starfish import data, FieldOfView
from starfish.types import Axes, Features
from starfish.image import Filter
from starfish.core.imagestack.imagestack import ImageStack
from starfish.spots import DecodeSpots, FindSpots
from starfish import Codebook

def find_spots(path, outpath, intensity_percentile=99.99, filter_width=2, small_peak_min=4, small_peak_max=100,
               big_peak_min=25, big_peak_max=10000, small_peak_dist=2, big_peak_dist=0.75, block_dim_fraction=0.5,
               spot_pad_pixels=2, keep_existing=False):
    """
    Find and keep only spots from stitched images.

    """

    image_stack = imageio.volread(path)

    print(image_stack.shape)

    # If DAPI, skip this (TODO: definitely shouldn't do this here)
    if 'DAPI' in outpath.split('/')[-1]:
        imageio.imsave(outpath, image_stack.max(0))
        return

    thresholded_image = np.copy(image_stack)

    _, height, width = image_stack.shape

    threshold = np.percentile(thresholded_image, intensity_percentile)
    thresholded_image[thresholded_image > threshold] = threshold + (np.log(thresholded_image[thresholded_image > threshold] - threshold)/np.log(1.1)).astype(thresholded_image.dtype)

    #May need to fiddle with the sigma parameters in each step, depending on the image.

    #High Pass Filter (Background Subtraction)
    gaussian_high_pass = Filter.GaussianHighPass(sigma=(1, filter_width, filter_width), is_volume=True)

    # enhance brightness of spots
    laplace_filter = Filter.Laplace(sigma=(0.2, 0.5, 0.5), is_volume=True)
    local_max_peakfinder_small = FindSpots.LocalMaxPeakFinder(
        min_distance=small_peak_dist,
        stringency=0,
        min_obj_area=small_peak_min,
        max_obj_area=small_peak_max,
        min_num_spots_detected=2500,
        is_volume=False,
        verbose=False
    )

    local_max_peakfinder_big = FindSpots.LocalMaxPeakFinder(
        min_distance=big_peak_dist,
        stringency=0,
        min_obj_area=big_peak_min,
        max_obj_area=big_peak_max,
        min_num_spots_detected=2500,
        is_volume=False,
        verbose=False
    )

    synthetic_codebook= Codebook.synthetic_one_hot_codebook(n_round=1, n_channel=1, n_codes=1)
    decoder = DecodeSpots.PerRoundMaxChannel(codebook=synthetic_codebook)

    block_dimension = int(max(thresholded_image.shape) * block_dim_fraction)
    spot_coordinates= np.zeros((0, 2), dtype=np.int64)

    # Finding spots by block_dimension x block_dimension size blocks
    # We skip the blocks at the edges with the - 1 (TODO: pad to full block size)
    for row in range(0, height - 1, block_dimension):
        for column in range(0, width - 1, block_dimension):
            # Cutout block and expand dimensions for channel and round
            block = thresholded_image[np.newaxis, np.newaxis, :, row:row+block_dimension, column:column+block_dimension]
            images = ImageStack.from_numpy(block)
high_pass_filtered = gaussian_high_pass.run(images, verbose=False, in_place=False)
            laplace = laplace_filter.run(high_pass_filtered, in_place=False,verbose=False)

            small_spots = local_max_peakfinder_small.run(laplace.reduce({Axes.ZPLANE}, func="max"))
            decoded_intensities = decoder.run(spots=small_spots)
            small_spot_coords = np.stack([decoded_intensities[Axes.Y.value], decoded_intensities[Axes.X.value]]).T

            big_spots = local_max_peakfinder_big.run(laplace.reduce({Axes.ZPLANE}, func="max"))
            decoded_intensities = decoder.run(spots=big_spots)
            big_spot_coords = np.stack([decoded_intensities[Axes.Y.value], decoded_intensities[Axes.X.value]]).T

            all_spot_coords = np.vstack([small_spot_coords, big_spot_coords])
            all_spot_coords += (row, column)

            spot_coordinates = np.vstack([spot_coordinates, all_spot_coords])

    # Copying over only non-zero pixels
    image_spots = np.zeros((height, width), dtype=np.uint16)
    for spot_coordinate in spot_coordinates:
        spot_column, spot_row = spot_coordinate
        for row in range(max(0, spot_column-spot_pad_pixels), min(spot_column+spot_pad_pixels+1, height)):
            for column in range(max(0, spot_row-spot_pad_pixels), min(spot_row+spot_pad_pixels+1, width)):
                # Max projecting over z-stack
                image_spots[row, column] = image_stack[:, row, column].max(0)

    imageio.imsave(outpath, image_spots)

    return image_spots

Any idea what the problem is?

My first guess would be that it’s because LocalMaxPeakFinder does not support is_volume=False (ref: how to localmaxpeakfinder). Let me know if setting that to True fixes the problem.

I tried changing it to is_volume=True, but that didn’t fix it.

I don’t think that’s the problem. Indeed, I think this exact code was working for my colleague some months ago, but there must be something different with the environment in which it is being run…

Could you provide the full traceback for the error? And a minimal example I could run to reproduce the error?

It’s very possible this error is dependent on your OS or environment. Can you describe how you installed and run starfish?

Here is the full traceback:

Traceback (most recent call last):
  File "starfish_test.py", line 2, in <module>
    sfs.find_spots("BigStitcherResults/B3_round1_fused_tp_0_ch_1.tif", "FilteredCompositeImages/B3_round1_fused_tp_0_
ch_1.tif")
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/InSitu_Transcriptomics/cisipy/cisipy/preprocessing/spot_finder_star
fish.py", line 77, in find_spots
    small_spots = local_max_peakfinder_small.run(laplace.reduce({Axes.ZPLANE}, func="max"))
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/starf
ish/core/pipeline/algorithmbase.py", line 23, in helper
    result = func(*args, **kwargs)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/starf
ish/core/spots/FindSpots/local_max_peak_finder.py", line 341, in run
    n_processes=n_processes
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/starf
ish/core/imagestack/imagestack.py", line 949, in transform
    return list(zip(results, selectors))
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/miniconda3/lib/python3.7/concurrent/futures/_base.py", l
ine 586, in result_iterator
    yield fs.pop().result()
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/miniconda3/lib/python3.7/concurrent/futures/_base.py", l
ine 432, in result
    return self.__get_result()
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/miniconda3/lib/python3.7/concurrent/futures/_base.py", l
ine 384, in __get_result
    raise self._exception
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/miniconda3/lib/python3.7/concurrent/futures/thread.py", 
line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/starf
ish/core/imagestack/imagestack.py", line 964, in _processing_workflow
    return worker_callable(sliced, *args, **kwargs)  # type: ignore
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/starf
ish/core/spots/FindSpots/local_max_peak_finder.py", line 234, in image_to_spots 
    optimal_threshold, thresholds, spot_counts = self._compute_threshold(data_image)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/starf
ish/core/spots/FindSpots/local_max_peak_finder.py", line 211, in _compute_threshold
    return self._select_optimal_threshold(thresholds, spot_counts), thresholds, spot_counts
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/starf
ish/core/spots/FindSpots/local_max_peak_finder.py", line 171, in _select_optimal_threshold
    dst = line.distance(p)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/geometry/line.py", line 1265, in distance
    if self.contains(other):
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/geometry/line.py", line 1231, in contains
    return Point.is_collinear(other, self.p1, self.p2)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/geometry/point.py", line 557, in is_collinear
    return Point.affine_rank(*points) <= 1
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/geometry/point.py", line 330, in affine_rank
    return m.rank(iszerofunc = lambda x:
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/matrices/matrices.py", line 159, in rank
    return _rank(self, iszerofunc=iszerofunc, simplify=simplify)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/matrices/reductions.py", line 234, in _rank
    d = M.det()
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/matrices/matrices.py", line 123, in det
    return _det(self, method=method, iszerofunc=iszerofunc)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/matrices/determinant.py", line 585, in _det
    return _get_intermediate_simp(_dotprodsimp)(m)
  File "/ahg/regevdata/projects/CTS/ImageAnalysis/codeSource/virtualenvs/allpurpose/lib/python3.7/site-packages/sympy
/matrices/utilities.py", line 34, in _get_intermediate_simp
    if dotprodsimp is False or _dotprodsimp_state.state is False:
AttributeError: '_thread._local' object has no attribute 'state'

I installed starfish using pip and virtualenv, and I’ve tried running the code in both the interpreter and by calling a script from the command line; both have the same error.

Here is a minimal example that throws the same error for me:

import imageio
import numpy as np

from starfish.types import Axes, Features
from starfish.core.imagestack.imagestack import ImageStack
from starfish.spots import DecodeSpots, FindSpots

def local_max_peakfinder_test(path, intensity_percentile=99.995, filter_width=2, small_peak_min=4, small_peak_max=100,
               big_peak_min=25, big_peak_max=10000, small_peak_dist=2, big_peak_dist=0.75, block_dim_fraction=0.5,
               spot_pad_pixels=2, keep_existing=False):
    """
    Debugging FindSpots.LocalMaxPeakFinder
    """

    image_stack = imageio.volread(path)
    
    # enhance brightness of spots
    local_max_peakfinder_small = FindSpots.LocalMaxPeakFinder(
        min_distance=small_peak_dist,
        stringency=0,
        min_obj_area=small_peak_min,
        max_obj_area=small_peak_max,
        min_num_spots_detected=2500,
        is_volume=False,
        verbose=False
    )

    images = ImageStack.from_numpy(image_stack[np.newaxis, np.newaxis, :])
    premature_spots = local_max_peakfinder_small.run(images.reduce({Axes.ZPLANE}, func="max"))

if __name__ == "__main__":
    local_max_peakfinder_test("BigStitcherResults/B3_round1_fused_tp_0_ch_1.tif")

@mattcai @alam-shahul I can replicate this error with pip install starfish in a fresh conda environment.on macOS 10.13.6.

# packages in environment at /Users/brian/anaconda2/envs/starfish_test:
#
# Name                    Version                   Build  Channel
appnope                   0.1.0                    pypi_0    pypi
attrs                     19.3.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
bleach                    3.1.5                    pypi_0    pypi
boto3                     1.14.23                  pypi_0    pypi
botocore                  1.17.23                  pypi_0    pypi
ca-certificates           2020.6.24                     0  
certifi                   2020.6.20                py37_0  
chardet                   3.0.4                    pypi_0    pypi
click                     7.1.2                    pypi_0    pypi
cycler                    0.10.0                   pypi_0    pypi
dataclasses               0.6                      pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
defusedxml                0.6.0                    pypi_0    pypi
diskcache                 4.1.0                    pypi_0    pypi
docutils                  0.15.2                   pypi_0    pypi
entrypoints               0.3                      pypi_0    pypi
h5py                      2.10.0                   pypi_0    pypi
idna                      2.10                     pypi_0    pypi
imageio                   2.9.0                    pypi_0    pypi
importlib-metadata        1.7.0                    pypi_0    pypi
ipykernel                 5.3.2                    pypi_0    pypi
ipython                   7.16.1                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                7.5.1                    pypi_0    pypi
jedi                      0.17.1                   pypi_0    pypi
jinja2                    2.11.2                   pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
joblib                    0.16.0                   pypi_0    pypi
jsonschema                3.2.0                    pypi_0    pypi
jupyter-client            6.1.6                    pypi_0    pypi
jupyter-core              4.6.3                    pypi_0    pypi
kiwisolver                1.2.0                    pypi_0    pypi
libcxx                    10.0.0                        1  
libedit                   3.1.20191231         h1de35cc_1  
libffi                    3.3                  hb1e8313_2  
markupsafe                1.1.1                    pypi_0    pypi
matplotlib                3.3.0                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
mpmath                    1.1.0                    pypi_0    pypi
nbconvert                 5.6.1                    pypi_0    pypi
nbformat                  5.0.7                    pypi_0    pypi
ncurses                   6.2                  h0a44026_1  
networkx                  2.4                      pypi_0    pypi
notebook                  6.0.3                    pypi_0    pypi
numpy                     1.19.0                   pypi_0    pypi
openssl                   1.1.1g               h1de35cc_0  
packaging                 20.4                     pypi_0    pypi
pandas                    1.0.5                    pypi_0    pypi
pandocfilters             1.4.2                    pypi_0    pypi
parso                     0.7.0                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    7.2.0                    pypi_0    pypi
pip                       20.1.1                   py37_1  
prometheus-client         0.8.0                    pypi_0    pypi
prompt-toolkit            3.0.5                    pypi_0    pypi
ptyprocess                0.6.0                    pypi_0    pypi
pygments                  2.6.1                    pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
pyrsistent                0.16.0                   pypi_0    pypi
python                    3.7.7                hf48f09d_4  
python-dateutil           2.8.0                    pypi_0    pypi
pytz                      2020.1                   pypi_0    pypi
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    5.3.1                    pypi_0    pypi
pyzmq                     19.0.1                   pypi_0    pypi
read-roi                  1.6.0                    pypi_0    pypi
readline                  8.0                  h1de35cc_0  
regional                  1.1.2                    pypi_0    pypi
requests                  2.24.0                   pypi_0    pypi
s3transfer                0.3.3                    pypi_0    pypi
scikit-image              0.15.0                   pypi_0    pypi
scikit-learn              0.23.1                   pypi_0    pypi
scipy                     1.5.1                    pypi_0    pypi
semantic-version          2.8.5                    pypi_0    pypi
send2trash                1.5.0                    pypi_0    pypi
setuptools                49.2.0                   py37_0  
showit                    1.1.4                    pypi_0    pypi
six                       1.15.0                   pypi_0    pypi
slicedimage               4.1.1                    pypi_0    pypi
sqlite                    3.32.3               hffcf06c_0  
starfish                  0.2.1                    pypi_0    pypi
sympy                     1.6.1                    pypi_0    pypi
terminado                 0.8.3                    pypi_0    pypi
testpath                  0.4.4                    pypi_0    pypi
threadpoolctl             2.1.0                    pypi_0    pypi
tk                        8.6.10               hb0a8c7a_0  
tornado                   6.0.4                    pypi_0    pypi
tqdm                      4.48.0                   pypi_0    pypi
trackpy                   0.4.2                    pypi_0    pypi
traitlets                 4.3.3                    pypi_0    pypi
urllib3                   1.25.9                   pypi_0    pypi
validators                0.16.0                   pypi_0    pypi
wcwidth                   0.2.5                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
wheel                     0.34.2                   py37_0  
widgetsnbextension        3.5.1                    pypi_0    pypi
xarray                    0.16.0                   pypi_0    pypi
xz                        5.2.5                h1de35cc_0  
zipp                      3.1.0                    pypi_0    pypi
zlib                      1.2.11               h1de35cc_3

@berl @mattcai I thought that it might be an issue with Conda, so I decided to build Python from scratch. This took me a couple days to figure out correctly, because I’m working on a cluster on which I don’t have root access. However, after all that, and after using venv with my fresh Python installation and doing pip install starfish, I get the same error.

This seems to be some sort of issue with how starfish and/or sympy does threading, I think? Here is the most relevant StackOverflow post I could find:

I’m not sure if there is some sort of multithreading going on, but if there is, is there a way I can tell starfish to not do any parallelism for now?

Thank you @berl and @alam-shahul for troubleshooting this issue and narrowing it down. I believe the problem is the new sympy version 1.6.1. When I did a fresh install with pip install starfish I also got the error, but downgrading to sympy==1.5.1 (pip install sympy==1.5.1) fixed it for me.

2 Likes