Thanks for the quick response on twitter about dealing with large datasets in 3D. I tried the stardist-dev and the keras datasequence for dynamically loading images for training.
Without GPU tools at the start of first epoch I get worker died warning/error and then there is no screen output/running process even though the CPU’s and the memory is still occupied.
I am trying with gpu tools to see if that helps in some way, my batch size is 1 and images are variable size but some of them have even 1000 Z’s as the data is coming from light sheet microscope.
Did you try it on an example dataset and encountered any issues like these?
Thanks again for looking into it so quickly.