Extract the network architecture with Java from a trained keras .h5 model

Hi!

I’m trying to open an already trained model from Keras (python) with Java and use it to analyze new images. My model is saved in HDF5 format which contains the architecture of the network as a .json file and the trained weights as .h5.

In FIJI it is easy to read each of the weight matrices in my network. However, I didn’t find any script to reproduce the architecture of the network without using Keras.

Using DL4J it is possible to reproduce the graph but is it also possible to complete this graph with the weights and process an image? Is there any other Java library which can read the dictionary from .json and build the architecture in a way that I know how to process my image with the trained weights in Java?

Might it be done already for Fiji, CellProfiler or Ilastik?

Thanks a lot!

Hello Estibaliz -

First, a few more words about what you’re trying to accomplish
might help folks on this forum point you to appropriate tools.

However, it appears that the DL4J framework you mention might
do what you want.

Is your goal to write a free-standing java program that does
your image analysis using the pre-trained network? Or do
you specifically want to use Fiji / ImageJ for the processing
(and you want to write a java plugin that uses your network)?

I am not aware of any built-in support for keras / .h5 models in
ImageJ.

According to the DL4J documentation, you can use it to import
and run (“inference”) a pre-trained keras model. Quoting:

String simpleMlp = new ClassPathResource("simple_mlp.h5").getFile().getPath();
MultiLayerNetwork model = KerasModelImport.importKerasSequentialModelAndWeights(simpleMlp);

// ...

INDArray output = model.output(input);

(Supposedly you can also train models and further train imported
models.)

I don’t know anything about CellProfiler, but, on the surface, it
appears that it might be an appropriate platform for what you
want to do. (I don’t know whether CellProfiler offers a practical
mechanism for importing existing keras models, but it does
appear to have some truck with keras.) With more detail about
your actual use case, I’m sure some of the CellProfiler experts
will chime in.

Lastly, let me speculate a little bit, and suggest a “poor-man’s”
approach to avoiding (partially) reinventing the deep-learning
wheel:

For concreteness, let me assume that you have trained an
image classifier that you want to use for further processing.
Rather than load the classifier into to a java program (such
as Fiji) and run it there, perhaps you could use your existing
deep-learning framework (presumably keras) to classify your
images, and write out the classifications, rather than the classifier.
Reading such classifications into java / Fiji would be straightforward.

Of course, without knowing more about your workflow, this is a
speculative suggestion.

Thanks, mm

Hi mountain_man
Thank you for the quick reply

First, I would like to know whether it is written already for Java so I can complete my code with already existing libraries. If it works well, then it is possible to modify the code for a user-friendly Java based software such as Fiji.

It is true, but still, it has some limitations https://deeplearning4j.org/docs/latest/keras-import-overview:

Popular models and applications

We support import for a growing number of applications, check here for a full list of currently covered models. These applications include

  • Deep convolutional and Wasserstein GANs
  • UNET
  • ResNet50
  • SqueezeNet
  • MobileNet
  • Inception
  • Xception

Troubleshooting and support

An IncompatibleKerasConfigurationException message indicates that you are attempting to import a Keras model configuration that is not currently supported in Deeplearning4j (either because model import does not cover it, or DL4J does not implement the layer, or feature).

The main concern is that I’m working with my own architectures so it could be the case that DL4J is not able to read all the layers/workflow properly. That’s why I asked whether there is any Keras interpreter written in Java, so it is possible to reproduce the entire network. Also, how good is DL4J with respect to Keras versions updating?

Thanks again!

Hello Estibaliz -

I don’t have useful answers to your specific questions, but I have
some general comments, below:

I’ve never used DL4J and have no idea how good it is or how
complete its coverage is.

If your architecture implements novel, custom functionality (like
BatchNormalization or PyTorch’s CTCLoss before they were
invented and implemented as standard building blocks), then I
would not expect DL4J to work for you. If you are simply wiring
together bog-standard keras building blocks in a keras-approved
way, then something like DL4J could likely work (but I have no
idea how good DL4J is).

Have you tried DL4J on your architecture, and can you report
specific errors that you’re getting?

My gut reaction is that you risk trying to reinvent the wheel here.
There are a lot of man-hours in keras. Is one better off spending
time writing a java clone of keras (or adding near-complete keras
support to DL4J), or adding new functionality (and fixing bugs) in
keras (or developing new image-processing algorithms, or doing
cool science)?

Writing code to use an existing, pre-trained network for inference
(e.g., giving it images as its input and getting class labels for the
images as the output) would be a significantly simpler project than
writing the code for training such a network. But still, given the
complexity of modern networks, there would be really quite a lot of
man-hours involved. (And if you’re asking whether DL4J keeps up
to date with new versions of keras, that would presumably imply
ongoing man-hours updating your own code base.)

You still haven’t really explained your use case. Just inference?
Fairly standard image classification? Video? Would Fiji / ImageJ
be a probable requirement? Would Fiji / ImageJ be your platform
and/or user interface, or would you just borrow a couple of image
processing algorithms from ImageJ?

It sounds like you want to run a keras (python / c++) network in
java. If somebody has already done the work for you (e.g., DL4J
or some other package that I wouldn’t have any suggestions for),
then great. But if not, I would advise against writing your own java
clone of keras. You could, instead:

  1. Pre-classify your images with keras. (Easy! Someone has
    already done the work of writing the inference engine for you.)

  2. Write your application in python, and use keras directly.

  3. Have your java application shell out a keras session to classify
    your images, as needed (on a per-image basis or for batches).

  4. Write a java bridge to a running keras “server” and have
    your java application use keras to classify your images on a
    client-server basis.

(Me? I’d do number 1, or maybe 2. “Don’t fight your tools.”)

But if you could share the details of what your actual workflow
is, forum participants might well have useful suggestions for
promising approaches or other tools that might make sense.

Thanks, mm

Hi, a different approach would be to load and predict with the model in TensorFlow.
To that end, pleae see imagej-tensorflow and you’ll also need to export your Keras model as a SavedModelBundle, e.g. using this.

We use this approach in the CSBDeep Fiji plugin.

Best,
Uwe

1 Like