U-Net available as ImageJ Plugin?



Recently watched “Research at NVIDIA: AI Can Now Fix Your Grainy Photos by Only Looking at Grainy Photos” https://www.youtube.com/watch?v=pp7HdI0-MIo and was impressed!

The AI was U-Net, I did some checking and found “U-Net: Convolutional Networks for Biomedical Image Segmentation” https://lmb.informatik.uni-freiburg.de/ … index.html there is a short video and at bottom of page “We provide the u-net for download in the following archive: u-net-release-2015-10-02.tar.gz (185MB). It contains the ready trained network, the source code, the matlab binaries of the modified caffe network, all essential third party libraries, the matlab-interface for overlap-tile segmentation and a greedy tracking algorithm used for our submission for the ISBI cell tracking challenge 2015. Everything is compiled and tested only on Ubuntu Linux 14.04 and Matlab 2014b (x64)” Hope it will be ported to Windows!!! it also mentions “If you do not have a CUDA-capable GPU or your GPU is smaller than mine, edit segmentAndTrack.sh accordingly (see there for documentation).”

They have made some ImageJ Plugins https://lmb.informatik.uni freiburg.de/resources/opensource/imagej_plugins.en.html the page only goes to 22.8.2014 so wonder if U-Net is available as a ImageJ Plugin?

If not, how difficult would it be to do this?






UNet is actually a deep learning software and such constructs are divided into three steps:

  1. Create training data using real data and ground truth, example annotations made via tools like ITK snap.
  2. Train the model using the UNet architecture.
  3. Apply the trained model on data not used for training to get the predictions.

I am not sure what the imageJ plugin does, or to be more precise how many steps mentioned above it handles. However there is python code and jupyter notebooks produced by @fjug lab which uses this architecture for image restoration. (https://github.com/CSBDeep/CSBDeep/tree/master/examples)

My understanding of their software is that it uses UNet and ResUNet architecture and if you are interested in segmentation you can input raw data along with annotated data as input(https://github.com/kapoorlab/CAREforCurie/blob/master/UNet/GenerateData.ipynb) and then use the simple_unet model they wrote to obtain a segmentation prediction (https://github.com/kapoorlab/CAREforCurie/blob/master/UNet/BatchProcessing3D.ipynb)
(Please correct me if I am wrong here @fjug, @frauzufall )

In order to use these notebooks you would have to download python, anaconda and then install tensorflow, keras. It is not as simple as just installing a Fiji plugin and using it to get a segmentation output as you first thought, I presume.


For step 3 you can use a Fiji plugin, check out these links about the CSBDeep Fiji plugin installation
and about how to run your own network. You should be able to apply any network exported as a TensorFlow SavedModel with single image input and output (same dimensions). Note that this is a work in progress and we will add support for a wider range of networks in the future.

You could first train your own network with the python code @kapoorlab already mentioned and then do the prediction in python or Fiji or you can try the existing demo Fiji plugins which use pretrained networks.


As already mentioned you can use CSBDeep if you have a running tensorflow implementation and sufficient compute power on your local machine. Your problem is mainly image restoration, therefore CSBDeep might be most promising.

For detection and segmentation tasks you can also try the U-Net Segmentation plugin which is available via the Fiji Updater. The plugin is Client-Server-based and interfaces a caffe installation on the local machine or a remote server, so installing the plugin is half the bet, you also need to install caffe on a Linux machine, to actually use the plugin. If both is installed, the plugin is probably exactly what you are searching for, and as a plus, you can use the server with the caffe installation from any computer running ImageJ.

See https://lmb.informatik.uni-freiburg.de/resources/opensource/unet/ for more details.