Stardist for Big Datasets in 3D

Hi @mweigert @uschmidt83 ,

Thanks for the quick response on twitter about dealing with large datasets in 3D. I tried the stardist-dev and the keras datasequence for dynamically loading images for training.

Without GPU tools at the start of first epoch I get worker died warning/error and then there is no screen output/running process even though the CPU’s and the memory is still occupied.

I am trying with gpu tools to see if that helps in some way, my batch size is 1 and images are variable size but some of them have even 1000 Z’s as the data is coming from light sheet microscope.

Did you try it on an example dataset and encountered any issues like these?

Thanks again for looking into it so quickly.

No with GPU tools I get error:

  File "/data/usr_app/VarunAnaconda/lib/python3.7/site-packages/pyopencl/__init__.py", line 887, in kernel_call
    return self._enqueue(self, queue, global_size, local_size, *args, **kwargs)
  File "<generated code>", line 10, in enqueue_knl_filter_3_x
pyopencl._cl.MemoryError: clEnqueueNDRangeKernel failed: MEM_OBJECT_ALLOCATION_FAILURE

Hello again,

I took a very small subset of my dataset, 10 ZYX image pairs and after waiting for the lazy load I got this error at the start of the training and then nothing happens after that. If we could solve this issue I think we are good to go.

Epoch 1/200
data_utils.py (718): An input could not be retrieved. It could be because a worker has died.We do not have any information on the lost sample.

After a bit more digging it was doing this because after loading the labeled images I was doing relabelling of them and that was causing the timeout, but now it works very well. Many thanks!

1 Like