ZeroCostDL4Mic: an open platform to use Deep-Learning in Microscopy

Hi everyone,

Some exciting news on our side. We are pleased to announce that we released a major update of our ZeroCostDL4Mic platform. This release is accompanied by an update of our bioRxiv preprint.

ZeroCostDL4Mic allows the use of popular Deep Learning neural networks capable of carrying out tasks such as image segmentation and object detection (using U-Net, StarDist and YOLOv2), image denoising and restoration (using CARE and, Noise2Void), super-resolution microscopy (using Deep-STORM) and image-to-image translations (using Label-free prediction fnet, pix2pix and CycleGAN). With ZeroCostDL4Mic, researchers with little or no coding expertise are able to train (and re-train), validate, and use DL networks, though a browser and for free, thanks to the Google Collab engine the platform uses.

Some helpful videos:

Latest Update In brief:

  • We released five new notebooks that train 3D U-Net, YOLOv2, Deep-STORM, CycleGAN and pix2pix.
  • Release of associated training datasets on Zenodo
  • Updated Preprint and Manual
  • We illustrate our notebooks’ applicability by providing examples of results that can be generated using our notebooks (6 new figures + 9 new videos).
  • We illustrate the power of performing transfer learning (See SI of our preprint).
  • We illustrate the usefulness of data augmentation (See SI of our preprint).

If you have any questions or comments, do not hesitate to get in touch with us! We regularly provide updates on twitter about the developments we are implementing, to check them out follow #ZeroCostDL4Mic on Twitter.

We hope you find ZeroCostDL4Mic useful for your work (or your hobbies!).

The ZeroCostDL4Mic team

23 Likes

Hi,

I’m trying to use pix2pix, although not running in colab but in a personal server with a Tesla P100 16Gb GPU.
I converted your colab notebook to a .py file and I’m testing with the Actin filaments dataset you provided.
My doubt is, is it normal being so slow (almost 3 min per epoch)? Even with gpu processing?

Thank you

Mafalda

Hi Mafalda,
Yes the training can take some time. The pix2pix dataset we provide contains thousand images. If you have access to multiple GPU you can speedup the process by using more than one (you will need to modify the code of our notebook a bit and increase the batch size https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/300e84a78e77e22f08668180c65949971386175b/docs/tips.md). In Google Colab, training 200 epochs took a bit over than 8 hours.
Cheers
Guillaume

Hi @Guillaume_Jacquemet

thank you for the tip, it’s already running on both GPUs and a lot faster. Now I need to limit the percentual usage of the GPU… :slight_smile:

Cheers,

Mafalda

1 Like

Dear Guillame

I would like to ask you if the google Colab U-net notebook has any restrictions according to the number and size of the images the network is able to process?
Previously I had been working with 160 images of size 896X896 pixel and 788Kb and the network was running without any problems however now I would like to work with 170 images if size 2048X2048 and 4Mg however when I start uploading my data set (cell 3.1. Parameters and paths) the notebook crashes and this warning appears in the log:

“WARNING:root:kernel 6fbc0215-5eb7-470b-a567-c70c7b7a9c44 restarted”

If I try to runt the same cell again (3.1. Parameters and paths) I got this message:

NameError Traceback (most recent call last)
in ()
27 # ------------- Initialising folder, variables and failsafes ------------
28 # Create the folders where to save the model and the QC
—> 29 full_model_path = os.path.join(model_path, model_name)
30 if os.path.exists(full_model_path):
31 print(R+’!! WARNING: Folder already exists and will be overwritten !!’+W)

NameError: name ‘os’ is not defined

It seems obvious that the problem is the size of my images, to big maybe, but what can I do if I need to use these images?

Thanks for your time

1 Like

Hi,

Thanks for reaching out! yes, it looks like the notebook crashed.
The second error you get is because the notebook restarted and now some of the variables are missing (since they are loaded in the previous cell).

At this stage, it is very unlikely that the notebook crashed because of the size of your dataset.
Are you using the latest version of the notebook?
Could you share a screenshot of the first error you get (maybe including the settings you chose)?

Cheers

Guillaume

1 Like

Thanks for your quick answer! Here I send you the error the google Colab is giving me, the first thing that appears is the blue circle next to the “Default advanced parameters enabled”, after the messages of "the notebook has crashed appears and if I click on the link that appears next to that message the right window is displayed.

I am not sure if I am using the latest version of the notebook, I went to ZeroCostDL4Mic GitHub page and open the U-Net (2D) link and make a copy of it in my google drive. I already try using batch size2 and 1 as is suggested by the notebook. When I use batch size=1 I have this message

“Your session crashed after using all available RAM”.

1 Like

Hi xgalindo!
I have just tried to reproduce your error and cannot seem to be able to do so.
I have used 28 images with sizes 2048x2048 without any problems.

The notebook does not actually load the patches to memory so, from the RAM point of view, it should not matter how many images or even how big your dataset is. And the batch size will only matter to the GPU memory and not RAM really.

Could you please see if you can consistently reproduce your error?
For this, please start from scratch by opening the notebook from our Wiki and re-run everything from a clean slate. Let me know how you get on !

Romain

1 Like

Hello Romain

Thank you for your quick answer.

I am currently working with 2 image microscopy stacks of 85 images each of 2048x2048 pixel size. if I try to use both stacks (or more) I have the same error over and over. However, if I use just one stack the erro does not apear again (until now).

I already started from scratch as you suggested and I the same errors appear, " The notebook as crashed and re-started" and “Your session crashed after using all available RAM”.

Xareni

Hi Xareni,
I wonder whether it’d be better if you used individual images, rather than stacks.
If you follow our instructions on how we did it with our demo data here:

You could split all of your stacks in individual images and try this way.
I suspect that this will work. Have a try and let me know !

Romain

They are already into individual images, I save each image of the stack in Google drive as an individual image and did the same with its corresponding binary mask.

Dear Xareni,
I have found what may be causing you an issue and have made modifications to the notebook (patch generation is now handling data slightly differently).
You can now use the latest version of the notebook on our Wiki page:

Please have a try and let me know how things work for you!
I hope this helps.

Best,

Romain

1 Like

Dear Romain

I thank you for your quick answer, I have decided to use just 85 images for my training because if I use more than this the training process lasts more than 8 hours and is interrupted due to Google colab limitations, however now I have encountered a new problem. After the training process is done, and I choose the option to use the current model to perform the evaluation, the cell shows me the next error message. I have already tried to use another model previously developed using U-net as well using its corresponding path, however, I have the same error. The training process seems to be done correctly but after I do not know why this error is appearing. Thank you very much for all your help.

P.D. I started from scratch using the link you sent me previously.

1 Like

Hi Xareni,
First thing first: did the fix I made solve the loading problem at least?
Disregarding how long it’d take to train with all your images, could you at least start the training successfully?

As an aside, you could also consider finding out what’s the maximum number of images you can use while remaining within the 12h runtime, and also you could disable data augmentation and use all of your data.

Now, onto your new issue, I have seen what you describe when the session is restarted between training and evaluation. In this case, all variables are gone and the notebook does not know what was the latest model. In this case, you can simply add the path to the model you want to evaluate and untick the box “Use_current_trained_model”.

Also, please check that the model folder is there and contains the saved model and correct csv files.
I hope this helps!

Romain

Dear Romain

I just finished a small training using all the images I have and the augmentation process and the training part seems to be working correctly now :slight_smile:. However, I have the same problem with the evaluation cell, I get the same error in both cases, If the “Use_current_trained_model” box is unticked and if I put the full path of the model. O leave you the screenshot of the error with the full display of the folder containing the model.

Thanks for all your help in advance

Best
Xareni

Hi Xareni,
So it looks like your training was unsuccessful. What allows me to say that is the following:

  • there is no model called weights_best.hdf5, which means that at no point there was an improvement in model performance
  • losses are NaN! See epoch verbose

So, to confirm this, could you please open the training_evaluation.csv file. If all values are NaN, then that’s what happened.

Now, onto why this happened: could you please let me know what file format, bit depth you use for your training? Not sure whether you’ve noticed in our wiki:
" To train U-Net network you need matching images of input (8-bit TIF) and masks (8-bit TIF). Additionally, the masks within the training dataset should have values being either 0 or 255 ."

Also, please after running section 3.1, you should have folders /content/img_patches/ and /content/mask_patches/. Could you please check whether these images contain data that accurately correspond to data patches?

Maybe you could also share with me 2-3 representative training images and masks so that I can take a look.

Cheers,

Romain

Dear Romain

Indeed my images are 8-bit in both cases, and the values of the binary images are 0 or 255. I check some of the data in /content/img_patches/ and /content/mask_patches/ and they have images containing data that accurately correspond to data patches. Here I send you some of the images I am using.

Thanks for all your help

Best Regards
Xareni Galindo

Hi Xareni,
I just took a look and noticed two things:

  • the masks in your second dataset “Sample S5” are made from an inverted LUT so even though the images look right, the data is wrong (0 and 255 are swapped). So that would confuse terribly the model to combine both dataset for training
  • the issue of loss is due to augmentation, and more specifically to shifts.

So you can run the training again on all your data (making sure masks are corrected to be all the same way around), either without augmentation or disabling the horizontal and vertical shifts specifically (set to 0), keeping other parameters the same.

I ran a quick training on your 5 images Sample S4

That’s not bad considering it’s so few images!

There are a few more things you could consider too. I realised that you’re trying to segment nuclei so you could use the purposely built StarDist network instead. This may perform better than vanilla U-Net.
It would be straight forward to transform your target images into a format compatible with StarDist by using the “Find Connected Regions” plugin on Fiji. In fact, I just ran a small training with this and it worked too. It would be interesting to compare which network performs better with your data.

Since your data are 3D stacks, you can also consider using the 3D versions of U-Net and StarDist similarly, this should give you nice results too, depending on the size of your training dataset as well.

I hope this helps !

Romain

1 Like