MicroManager Multidimensional Acquisition view much slower than Live view

I’m trying to record from two Photometrics Evolve 128 cameras using MicroManager, and after promising start, I’ve encountered a peculiar problem. I’ll start with the most recent 1.4 version.

When I run live view, everything is great, frame rate is as expected, etc. However, when I run multi-dimensional acquisition (mda), with everything but the box determining number of samples unchecked (i.e., recording to RAM and no extra stuff that could slow it down), the live view that opens is pretty slow. What I found happens is that it does record the frames it should, but then keeps displaying them over a fairly long period in slow-motion before they can be stored. For example, recording 10k frames at a certain condition records 22 seconds, but the live view showing what’s recorded is hanging there for over a minute, slowly presenting the images at around 100 FPS, while the actual recording rate is around 400 FPS (this is preventing me from taking another recording soon after the first one, which is unfortunately a critical limitation for our experiments). This happens consistently across camera parameters, binning, etc. It sounds to me quite like the issue described here: http://micro-manager.3463995.n2.nabble.com/quot-Slow-display-quot-td7579145.html"m but I could not find a place to toggle the “slow display” mentioned there.

I also tried version 2.0 of MicroManager, but there I had even less success - Live view is fine, but once I try to do acquisition, the message window first says “received 0 out of 1 images”, which gradually changes to “5 out of 1 images”, but the window where the images recorded should be played remains empty, hanging there with “Waiting for Image”, not progressing further.

I’d be extremely grateful for any assistance with this.

Sorry, I don’t have a solution, but what are your memory settings?

https://micro-manager.org/wiki/Micro-Manager_Configuration_Guide#MemorySettings

There is no support for 1.4, so focus on 2.0-gamma. Start with what works, for instance, can you acquire 10 images from one camera. 100? 10,000? I assume that you setup the Multi-Camera with your two Evolves. Can you snap with the multicamera as the camera, and do you get the two images? Can you run live with the multicamera? Can you acquire 10, 100, 10,000 images (I guess not the latter;).

Once you find out what/where things do no longer work as expected, go to Help > Report a problem, and report the problem.

Thanks a lot for the replies!

@samjlord: 2000 MB for ImageJ and 1500 MB in MM (v1.4) or 250 (v2.0). It does not look at the first sight that these would be wrong based on what I found online, but maybe there is some hidden surprise :slight_smile:

@nicost : Right, my motivation for using 1.4 was that it is far more responsive - with 2.0, everything takes longer (not because of UI, it’s just the software response is a lot slower), and it also seems to function less well out of the box (beyond the slowness, it seemed to get not as far with the MDA too). But I understand it is the developed version, so it’s worth focusing at that. I will check the acquisition of a small number of frames next week - I know that 1000 or 10000 didn’t work for sure. Live view with the multicamera is fine (and yes, we have a MultiCamera with two physical Evolves). Thanks for the tips, I’ll see how it goes!

For 2.0 you will need to set the ImageJ memory much higher (are you using a recent nightly build? The change in needed memory assignment happened last April or so, and 250MB is likely filled up in no time, and a likely cause for everything feeling slow as you force it to do a lot of garbage collection, set to at least half your system memory, see https://micro-manager.org/wiki/Micro-Manager_Configuration_Guide#Memory_Settings).

We have a machine vision camera that runs at ~900 fps at 320x240 pixels and I now have the display keeping up with it in 2.0-gamma (by skipping display of frames). Should work for you too, at least with one camera.

OK, I will test increasing the memory, and will trial also a single-camera setup. Many thanks for the suggestions!

If increasing the memory limits proves insufficient, try watching the sequence buffer during acquisition using the sequence buffer monitor plugin. If you see the buffer staying mostly empty with occasional blips that cause many frames to pile up and continue to accumulate over time, try changing the garbage collector.

For our diSPIM, running at >200 FPS (total) across two cameras required us to run micro-manager in an alternate garbage collector. I had success by switching to the CMS GC which is being deprecated in favor of the G1 GC. I have not tested the G1 yet, as that requires some parameter tuning. This can be selected at run-time by launching micro-manager from the command prompt using this syntax: java -XX:+UseConcMarkSweepGC -jar imagej.jar from inside the micro-manager install directory. The impression I got from testing this against the default parallel GC is that the default causes large pauses during GC causing hundreds of frames to pile up in the sequence buffer, eventually filling it up. The CMS and G1 GC are meant to minimize application latency by minimizing the time spent with the entire application paused for GC.

1 Like

@pavak_shah: Very good point: I created an issue on Github to help remember that we need to change the default in the launcher code: https://github.com/micro-manager/micro-manager/issues/905

@pavak_shah Thank you for the tip! I tried what you suggest, but am getting an error below (the PC is 64 bit):

  • I hope it’s ok I used “ij.jar”, given that “imagej.jar” isn’t there
  • in general, based on the observation below, this sounds to me a bit more like an issue with allocation, rather than gc…

@nicost Thank you for the tips - I ended up generating a report, but just to briefly summarize (in case someone gets the same issue later):
a) the most recent nightly build (18th) seems to work better than the previous one (from sometime at the start of this month) in that it does not greet me with an error on startup (related to circular buffer size) and seems capable of recording something with dual camera, which is great.
b) the problem I seem to be having is the same for single or dual camera recording now - and that is that there is a potentially fairly major delay before the acquisition itself starts. For 1000 frames, this is <1 second delay, for 10000, it is around 9 seconds, and for 20000, it is around 40 seconds. By delay, I mean the time before the live view accompanying the acquisition shows, and also the first timestamp during the recording corresponds to the delay.

The error is most likely caused by the “java” on your path being a 32-bit java. You can check by running “java -version”. You’ll need to run using a 64-bit java.

I tried to reproduce the delay you describe using the demo camera. I set the camera to 128x128 pixels, 16-bit (2 bytes), Mode to “Noise”, circular buffer to 400MB, exposure time to 2.5ms (to get to 400 fps). The first time I ran this, there was an about 9 s delay before the viewer showed an image. The second time it started immediately. This delay is caused by allocating the circular buffer memory (but only the circular buffer size should change the delay, not the number of images you request, so there could be something camera specific there). Viewer updates are sporadic, it now and then shows an image, but on average maybe 1 fps. Viewer updates became much better after setting histogram update rate (under gear icon in the inspector) to 0.5Hz. There still are occasional periods where the UI is sluggish, but no data loss seems to occur. The GC flag did not seem to make a big difference for me (but it certainly could in other situations). It should be easier to do this with a real camera since the computer will not need to generate the images, however, you may want to run the same tests I did with the demo camera so that we can be sure if issues are related to your camera/software or to the Micro-Manager common code. Let me know if these pointers are useful.

Thank you for the great ideas to try! I will do that once I get a slot in the lab again. Curious to see how it will go with the demo camera…