Clear memory - should be part of all batch scripts

Development
#1

After using Qupath for several years, I just discovered this extremely powerful little script. Basically, it solves almost all memory issues when running scripts in batch mode, and frankly should be the default mode for all batch mode scripts so that memory is not overwhelmed, as accumulated RAM usage in batch mode is a really frequent issue and causes so many crashes. In future versions I wish there was e.g. a check-box “Clear memory between each batch processed image” in the script window, or a separate option “Run for project (clear memory between images)”. Just a suggestion.

I now put this at the end of all my scripts before I run them in batch mode, and basically has removed most problems with RAM and QuPath:

Thread.sleep(100)
// Try to reclaim whatever memory we can, including emptying the tile cache
javafx.application.Platform.runLater {
    getCurrentViewer().getImageRegionStore().cache.clear()
    System.gc()
}
Thread.sleep(100)
2 Likes
Qupath script with pixel classifier
#2

Thanks for the suggestion, I’m not sure about the proposed change. I don’t have enough information about what these crashes are, and am not aware of it as being a ‘really frequent issue’. The System.gc() will effectively happen anyway before a crash. In most other cases, if this script helps then it suggests to me that the value of the preference ‘Percentage memory for tile caching’ may be too high.

The problem with clearing the tile cache too often is that this will impact performance for open viewers unnecessarily. I realise that sometimes it may be helpful, but I’m not convinced it should be the default – particularly when there is a) a relevant preference to adjust, and b) the option to add those lines to scripts anyway.

#3

Thank you for the response, Pete. I wasn’t aware of the preference “‘Percentage memory for tile caching”, so might also solve the problem. But will that slow down processing significantly potentially if set too low? Qupath eats memory like pac man in batch mode in my experience, so this script has been really helpful to me at least, but your solution might also be good.

#4

Most Java applications will grab more and more memory while they are running, but then clear it using the garbage collector (System.gc()) whenever it is really needed, before throwing any exception.

The garbage collection will discard everything QuPath doesn’t need, except it won’t empty the tile cache containing image pixels. However, the size of the tile cache is limited to a proportion of the available memory (adjustable in the preferences) to avoid this using up memory indefinitely. This means that when the tile cache is full, older tiles will be discarded. This won’t reduce the apparent memory used, but it should hold it within limits.

If you’re seeing lots of errors that can be solved by this script, then the details of when these errors occur would be important to understand what might be going wrong.

#5

I can try to make a more detailed monitoring of the accumulated memory, but it’s definitely mostly the tile cache that accumulates memory.