I have successfully trained CSBDeep (CARE) on my data using Python on the GPU. I am using GeForce RTX 2090 Ti (12 GBs) and ofcourse compatible CUDA, CUDnn, Tensorflow, and Keras.
I find that our sample sizes are much bigger than what CSBDeep was designed for; we use 25x and 40x objectives and image nuclei much larger than the Tarabetium sample used originally. I understand that the training model is dynamic based on the dimensions of input voxels (16x64x64 originally). I would like to increase these dimensions to include at-least two cells/nuclei/ROIs within one voxel, however just going up to 32x128x128 throws OOM errors. I believe this is because the neural network model is built within the GPU and would be limited by GPU memory.
My question is: is it possible to bypass the GPU and construct the model in the RAM (I have 128G so much better). I do not mind the training time - can leave it to train over the weekend. Also, a discussion on what is the optimal voxel size would also be very helpful. Is it even necessary? My principal input resolutions are ~150x1000x1000 (‘ZYX’).
Furthermore, the problem can also be translated at prediction time where the input image would be broken into n_tiles. Do the internal functions break each tile to the same voxel size as the network’s dimensions?