Stardist 3D Training Error with Original Config Values

Hi, I’ve hand-annotated a small set of z-stacks which I have injected in the Demo3D dataset and tried to train the model using those. But at the training stage, I encounter the error quoted below. Any suggestions? Thank you!

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-40-cab340fc00c9> in <module>()
     13     model = StarDist3D.from_pretrained('3D_demo')
     14 else:
---> 15     model.train(X_trn, Y_trn, validation_data=(X_val,Y_val), augmenter=augmenter)
     16 None;

3 frames
/usr/local/lib/python3.6/dist-packages/csbdeep/utils/utils.py in _raise(e)
     88 
     89 def _raise(e):
---> 90     raise e
     91 
     92 

TypeError: exceptions must derive from BaseException

Hi @elevnn, welcome to the forum!

the error message is unfortunately not really helpful because it doesn’t show where the problem comes from. However, I have a suspicion. Our code likely tried to show an error message, but we have a little bug in there preventing the message from being shown. Concretely, one of these two lines likely caused this:

I.e. your own annotated images are either too small, or input image and corresponding label image don’t have the same shape (pixel dimensions).

Best,
Uwe

1 Like

Hi Uwe,

Sorry about the late response, I realised I wanted to finish experimenting with my new process before posting, as I was trying to explain in my initial thread response:

Instead of hand-annotating, I am now trying to generate a new set of 512x512xZ annotated 3D masks using surfaces extracted from Imaris (.ims) files containing segmented nuclei (I have a large quantity of those .ims stacks). With some processing, get to the point where, in my annotated mask, each distinct nucleus is represented by voxels with the same uint16 value in a 3D numpy array. The picture below is one slice from the z-stack/array, which contains voxel values ranging 0-126 corresponding to the number of nuclei.
slice
I have then saved those masks as .tif using the python tifffile library.

I have replaced the first 5 stacks in the demo3D training set with my own data, as well as the first stack in the testing data.

Unfortunately, I am still encountering the exact same error as mentioned in my first thread post. I don’t really understand what I am doing wrong, as per your previous advice:

  • I have checked that my files aren’t too small. They are now 512x512xZ.
  • I have checked that the images and the masks match in terms of shape (see blow the side-by-side comparison of the metadata of an image-mask pair)

Apologies for the inconvenience, and thank you for your help!

Best,
Elena

Hi Elena, please run this after the images have been loaded (cell [5]) in the 1_data.ipynb notebook:

[print(x.shape, y.shape) for x,y in zip(X,Y)];

For the provided training data, the output is:

(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)

Best,
Uwe

Ok, I’ve just seen your PM with a link to your data.

For your data, I get

(34, 512, 512) (34, 512, 512)
(22, 512, 512) (22, 512, 512)
(26, 512, 512) (26, 512, 512)
(26, 512, 512) (26, 512, 512)
(31, 512, 512) (32, 512, 512)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)
(64, 128, 128) (64, 128, 128)

That means your data has correct axis order and shapes of input images and output labels masks match except for this image: (31, 512, 512) (32, 512, 512). You need to fix that.

I’ve also opened your images in Fiji and they look good to me. You now need to change the default training parameters in 2_training.ipynb, which use a patch size that is too large for your data (train_patch_size must fit your smallest training image). Specifically, please change cell [10] to something like:

# 96 is a good default choice (see 1_data.ipynb)
n_rays = 96

# Use OpenCL-based computations for data generator during training (requires 'gputools')
use_gpu = False and gputools_available()

# Predict on subsampled grid for increased efficiency and larger field of view
grid = (1,1,1)

# Use rays on a Fibonacci lattice adjusted for measured anisotropy of the training data
rays = Rays_GoldenSpiral(n_rays, anisotropy=anisotropy)

conf = Config3D (
    rays             = rays,
    grid             = grid,
    anisotropy       = anisotropy,
    use_gpu          = use_gpu,
    n_channel_in     = n_channel,
    # adjust for your data below (make patch size as large as possible)
    train_patch_size = (16,256,256),
    train_batch_size = 2,
)
print(conf)
vars(conf)

Also remove all other training images that are not yours.

Hope that helps,
Uwe

2 Likes

Amazing, it worked very smoothly. Thank you so much for your help!

So far, I have prepared and tried two sets of training data, one containing tobacco plants and the other arabidopsis roots. They are all Zx512x512.

I have only used the short demo training with Google Colab GPU so far as I am still waiting for institutional GPU access so I expect the results with 400 epochs to be quite different from what they are now. However, one thing I am worried about is that, for both of my datasets, the actual nuclei are very small but it seems the predicted nuclei are always much bigger than the GT nuclei, as seen in an arabidopsis root example below (using 7 training stacks and 1 validation stack):

  1. Is this an issue with my data, as in, the nuclei are smaller than the minimum required size?
    OR
  2. Is this something I can tweak in the model parameters?
    OR
  3. Will this just get better once I’ve properly trained it for 400 epochs? (My main worry is that with the demo3D dataset, even the demo training with 2 epochs yielded quite good results compared to what I get with the image above)

Once again, thank you for being so responsive to user questions, it means a lot.

Elena

Hi @elevnn,

Which notebook did you use and what do you mean by “demo training”?

Cheers,
M

Hi Martin,
I’ve used the notebooks found in here https://github.com/mpicbg-csbd/stardist/tree/master/examples/3D and adapted them for my own purposes.

By ‘demo training’ I meant that in the second notebook ‘Training’, there is the following option, which I left as is because Colab seems to crash every time I wish to do proper training, due to lack of RAM.

quick_demo = True

if quick_demo:
    print (
        "NOTE: This is only for a quick demonstration!\n"
        "      Please set the variable 'quick_demo = False' for proper (long) training.",
        file=sys.stderr, flush=True
    )

Best,
Elena

I’d suspect it would.