Hello all, so related to some previous posts about garbage collection behavior in micro-manager, I’ve been trying to narrow down on what might be the performance-limiting factor in high-framerate acquisitions.
I’ve run tests using recent nightlies of both 1.4 and 2.0g and see a similar limitation, that using Hamamatsu Fusion cameras any exposure shorter than 8.5 ms (fast readout, 0 ms interval, continuous acquisition on internal trigger source) results in frames piling up in sequence buffer.
These tests have run on two different workstations: 1 with dual Xeon Gold 5217’s and one with a brand new AMD Threadripper Pro 3955WX. In both cases I tested acquiring purely to memory and acquiring to fast NVMe SSD RAID arrays, with no difference.
This limitation persists with the default parallel GC (although frames pile up in bigger bursts, presumably when GC is triggered) and with a tuned G1GC set to favor more frequent garbage collection with shorter stop-the-world intervals.
This also seems to be only weakly dependent on the size of each frame as the ~8.5 ms threshold holds for both full chip (2304 x 2304) and images cropped to 400 rows. Acquiring from two cameras simultaneously also doesn’t affect this behavior.
What seems odd about this is that shorter exposure times shouldn’t increase the framerate on the camera at full chip beyond 100 FPS, but we still end up getting frames piling up in the sequence buffer with 5 ms exposures.
Has anyone else with similar cameras (I imagine the Flash 4.0 is the most common) run into similar issues trying to operate at high-ish framerates? Where lies the bottleneck? Is it (as was suggested a while ago by either @nicost or @jondaniels) metadata handling while emptying the sequence buffer? It seems to be something that isn’t addressed by a significantly faster CPU (the Threadripper Pro) with faster memory.
Edit: The fact that this threshold seems to be the same whether running 1 camera or two also seems to suggest against this from a naive standpoint, as I’d imagine there’s twice as many metadata-related operations when running an acquisition with two cameras?
I also imagine that there are folks out there running significantly faster cameras, how do those setups behave? Do you just balloon the sequence buffer to be able to soak the pile-up or is there some configuration where you can continuously acquire without hitting the bottleneck that I seem to be encountering?