Can Deep Learning learn existing Deconvolution (or other) algorithms?

Hi Deep Learning experts

I am curious if anyone is aware if deep learning has been used to learn other algorithms, but execute them more efficiently (speed wise and memory wise).

The specific use case I have in mind is Richardson Lucy Deconvolution with Total Variation. Could a deep learning system learn how to apply several hundred iterations of this algorithm non-iteratively? (Again, in this case the goal would not be to necessarily beat the existing method, but apply it faster)

Brian

1 Like

Not an expert!
That said, the use of deep learning on unknown samples scientists are trying to learn something from worries me greatly, as the deep learning algorithms might specifically ignore new types of objects that the research would be interested in. There is the new FastAI preprint that should be amazing in some situations, but I am worried it could be quickly misapplied :slight_smile:
This amusing video as a visual demo, where breaking the plank is a novel event that wasn’t in the training data:

2 Likes

I know real DCV experts who would argue that unser no circumstances they would trust an image that after DL now has higher resolution (contains higher frequencies) than the original without being able to dissect the analytical process that yielded in those results.

But my take is that this is a very active field of research that has the potential for great results as well as for great misuse or side effects. So try it out but be sceptical.

3 Likes

My 2c…

A neural net is basically a universal approximator. So you could train it (fit it) to mimic any function, including the result of applying an RL algorithm… However, it remains an approximation and you’d loose all the mathematical guarantees that applying an algorithm gives you. That said, DL works, most of the time… And i’s a really fascinating tool! And even having a ‘guaranteed’ algorithm is not enough in itself to blindly trust the results (think of all the choices that go into applying deconvolution: single PSF? theoretical/experimental? n.iterations? stop criterion? algorithm? noise level? regularization or not?..)

3 Likes

I had that same question in my head a few months back as it would be so convenient to create training data for a Deep Convolutional network using a traditional algorithm.

Turns out it is an idea that others have already explored:

At least two of the co-authors (@talley, @jencwaters ) are also on this forum, maybe they can offer some insights.

It also raises some interesting questions, e.g. what if I train a network based on results from a proprietary method in a black-box commercial package?

7 Likes

For both classical Decon and DL clever tests need to be performed to verify any claims of super resolution. For example prove that multiple points, that are closer together than the resolution limit, can be resolved, or that objects of differing sub resolution sizes can be measured. I am not aware of rigorous examples, partly because creating phantom samples seems to be difficult. At the very least a very complex simulation would be interesting, where accurate noise, convolution models and sampling are used to create samples with sub-resolution features.

1 Like

Classical image deconvolution methods can be combined with regularization based on deep learning. We had a paper about that in 2017: http://research.uweschmidt.org/pubs/iccv17kruse.pdf

While the concrete method proposed in this paper is not really relevant for the microscopy community, section 2 might be helpful background reading to see how our algorithm can be understood as a generalization of the classic Wiener filter.

Note that this paper is in the context of natural images, where the PSF typically is due to motion blur and cannot assumed to be the same for every image. Also, the paper needs to make a Gaussian noise assumption to derive the specific method.

PS: The paper also proposes a simple and parameter-free method to reduce restoration artifacts that can be caused by the (typically incorrect) assumption of circular/periodic boundary conditions, which has to be made for FFT-based deconvolution.

Uwe

2 Likes

Last year there was a Nature Methods paper by Weigert et al. (and I think @uschmidt83 in this thread might be 2nd author) where they applied deep learning to different microscopy techniques to reduce SNR requirements to speed up acquisition. (Their code is available online, links in paper)
https://www.nature.com/articles/s41592-018-0216-7

1 Like

Nice video, @Research_Associate. Another parallel to machine learning is that the ‘machine’ in the video came up with a novel way to complete the task that the (very patient) trainers hadn’t thought of.

1 Like

Absolutely! Butt… (tee hee), one that might not generalize as well to later, thinner and more solid, test blocks :slight_smile: There could be other benefits to the new method though, and the important thing is realizing that it isn’t really doing the same thing as the intended method. IE, it didn’t find a more efficient way to break the block with it’s foot. I’m sure all machine learning people appreciate the training vs test set quandaries, my worry was more for those that don’t.

Going to need to try and dig up another paper that did something similar, and where you could see the algorithm “filling in” data that did not appear to be there in the low resolution or high resolution ground truth images. A similar example might be broken branches in a honeycomb structure, if the test set was all complete honecomb structures, might be filled in. If a scientist was measuring those breaks, that could be problematic.

Something something, off to get caffeine!

Edit: Quick addendum to that now that I have liquid fire running through me! One of my favorite things you can do with DL is novel feature extraction, so could be a great possibility here. If a novel mathematical method (that can be traced to some physics/math based reasoning) can be derived from the way the deep learning model processes the data to generate a function that runs faster, I think that would be fantastic.

1 Like

Ah, thanks! That was exactly the paper I was thinking of.


And it made me think of that honeycomb example. The network has additional areas that aren’t even in the ground truth.

1 Like

I’ve often wondered why ML hasn’t been used to derive the basic equations of physics. F=ma,
conservation of momentum and energy, etc. It would be interesting if it came up with alternate descriptions of reality e.g. e=mc^2, warped time space, etc. There is a famous modeling example called the cheese mallot where the most effective method of getting a thin slice of cheese from a block of cheese was explored. The most effective solutions were an infinitely thin wire for cutting a slice off and a malot that would squish the cheese block very thin.
Novel solutions to how the world works, helping us to colonize other planets, or how people work, helping to diffuse world animosities would be of great help, and ML could be programmed to look at solutions to these hard problems without bias (maybe).
I hope that academia is looking at this research and application area.
If you know of any research along these lines please share the citations.

Ron DeSpain

2 Likes
4 Likes

This can happen, especially for very low SNR (and thus ambiguous) images.
In the paper (cf. Fig. 5), we offer tools to assess the reliability of the restoration:

  • Option to predict a probability distribution for the intensity of each pixel. Allows to gauge the spread of likely intensity values.
  • Option to train several neural networks to form an ensemble. If the networks of the ensemble strongly “disagree” for a certain region, then the result should be used with caution.

Developing better tools for these issues is ongoing research.

Best,
Uwe

3 Likes

Exactly! And those were some brutal input images. I don’t have access to the paper right now (will take a look later), but were those tools used on the images from that figure? I was wondering if those options would identify the indicated regions.

If this is the kind of use that is being encouraged (low SNR, improve your microscopy images), I can easily see some users thinking that they can “run a model and get better images.” Full stop. I ran into something similar when I was first introduced to deconvolution software. No one warned me about the potential for deconvolution artifacts, and until I got some exposure and ran similar high resolution “ground truth” scans, I wasn’t sure what was going on. Emphasizing the tools to check the images processed like this is going to be very important. Kind of an offshoot, but it would be interesting to use a model like this with tools that could sift through large numbers of movies/stacks/data for novel events, by looking for areas of low probability (things the model couldn’t match well to something that it knows) and drawing attention to them. Not sure how well it would work in practice, since a DL model can also be very sure and very wrong. Might work for some types of projects.

Anyway, my primary concern would be users that are less experienced than you (probably most of us!) not using the tools if they are not emphasized by default. Or perhaps worse, a PI telling someone in a lab to enhance their images with a deep learning model, while being to busy to worry about the fine details.

And sorry, re-reading that last post, the “even” sounded more aggressive than I intended. I do want to draw attention to the caution that must be used with these tools, though! Much like I wasn’t aware of the dangers with deconvolution when I had the option/software dropped in my lap. Once burned… and yes, I should have done more research on it! Time, time.

2 Likes

In deconvolution, I always tell people to output the PSF from their software. The PSF used is just as important if not more important than the exact algorithm. People think there are only 3 ways to compute a PSF, theoretical, measured, and blind. But each of these approaches has many variations and sources of error. In 10+ years of following microscopy and image processing related list serves I can’t think of one case of users asking questions about the PSF they obtained with their software.

If users have not been trained to ask questions about the PSF in traditional restorations, will they know to ask detailed questions about the training and testing process in deep learning??

In deep learning the PSF will be built into the model along with information about their sample content. The model of course is derived from the training set. In open source applications such as CARE the users can keep track of the training set, and work on their own training sets, have access to the source code. and the authors of the code are available to openly answer questions.

However what about the commercial implementations?? Will users have access to the training set??

In deconvolution, classic algorithms such as Richardson Lucy still provide competitive performance. Ideally every deconvolution vendor would offer validated implementations of RL, Inverse filter and a couple of other algorithms. However different software provides different answers, and few people seem to ask why.

Now that deep learning is being implemented in commercial application how do you test it?? Can you train networks with these applications?? If you trained with the same data, on two different DL implementations, would the result be the same?? Should it??

Brian

2 Likes

Now that I had a chance to take a look again at Figure 5 in the Nature paper, it shows that even the same DL network can generate different results, generally due to the stochastic nature of the learning steps (I think). Completely different networks would almost certainly vary to a greater degree, though you could use a similar approach to look for regions of variance.

1 Like

Just out of curiosity, are you wanting them to look at the shape to look for aberrations, or, I suppose, what is your intention as far as their use of the PSF? Include it in the paper/supplement? I am relatively new to all of this, and while I have seen quite a few pitfalls, there is just way too much still to learn!

2 Likes

Hi @Research_Associate

It would be great to have PSFs in the supplement along with calibration data (ie beads), it may seem like overkill, but it could be needed if anyone wants to reproduce the image processing part of an experiment.

I’ve seen cases where the image is aberrated but the PSF is not, and cases where automatic aberration detection goes wrong, or users incorrectly enter values, and the PSF is aberrated but the image is not. Both can result in artifacts.

Also if the PSF has too large of support (ie is too broad) artificially small measurements and a “donut” effect can occur, where there are artificial holes in small objects.

I’ve seen this happen when the PSF is measured from several beads of known size (ie averaged as in Huygens PSF distiller) however beads in the calibration image clump together so the PSF is too large. I’ve also seen it happen when blind deconvolution does not converge from a artificially large starting point.

I’ve also seen cases where a measured PSF contains bleed through from another channel, or part of a PSF from an adjacent bead, or background for some other reason, and this causes problems.

Here is a case where we ended up getting much better results by taking a close look at the PSF

1 Like

And bringing the conversation back to Deep Learning. Will users of Deep Learning understand how to evaluate these systems?? Should they be required to show the training set from which their network was derived??

When CARE first came out there was a lot of discussion. Tweets of amazing images, would lead to questions, which would lead to quick and thorough replies from the authors and very in depth and valuable public conversations, regarding how to train and apply networks properly.

Recently I’ve seen other systems being advertised from commercial companies and I wonder if the users will know how to evaluate these systems, and ask the right questions.

1 Like