Out of memory




CPCluster sometimes crashes on some nodes during a module due to an out-of-memory error.

can one somehow rewrite the code that it saves at least all the results of the images analyzed to a Batch.mat file…such that not all the data from this node is lost?

probably one would have to catch the OUT OF MEMORY error within the module and tell it to save everything that has been done till now…?!



Hi Christian,

The simplest solution to write out the data is to simply use a smaller batch size to begin with. Matlab’s memory footprint grows with every cycle, so smaller batch sizes will have less “overhead” memory usage, and this will also minimize the lost cycles if an error does occur. However I would expect that any lost cycle would make you want to fix it, no? See Help->General Help -> MemoryAndSpeed for general memory minimization tips.

But your third question implies that you want to save the data from the cycles that errors, right up to the module it errors on? I wouldn’t trust the data from that cycle, since it ran out of memory and who knows what happened to the handles structure at that point.

It seems to me that writing intermediate results would have a higher cost overall (slowing down batches that don’t error because they have to access the disk more) than the benfit gained.