Fast Time Series Acquisition for 10GigE Camera?

Hi Everyone,

We are looking into acquiring a fast time series (>500 fps) over the course of about 10 seconds with a FLIR Oryx 10GigE Camera. We have demonstrated on our system that this is possible without losing frames in the native Spinnaker software. However, we wish to pair this with a few other types of acquisitions (z-stacks at wider FOV) through custom acquisition scripts with pycromanager/MM core commands. I was wondering if anyone has successfully done an acquisition at this speed through micromanager? Here are the issues I have been running into thus far:

To test, I would run an MDA with 10,000 frames, at 0 ms interval so acquire the data as fast as possible (10 sec time series at 1ms exposure). Micromanager is seemingly able to capture about 6000 frames but for the remaining frames, it is unable to grab the images from the camera, then MM times out, and the acquisition fails.

I am able to do this fine on a camera at the same fps but with a USB3.0 connection. It seems to not be an issue with the buffer size, as this acquisition is only ~11 GB in total and the buffer size was set to 50 GB (64 GB RAM computer). I also changed the Java Memory under: ImageJ > Edit > Options > Memory & Threads to 40 GB. I also monitored both this memory and it did not ever send images to the sequence buffer, but got up to 23% memory in the memory monitor. I tried this acquisiton in pycromanager, and received similar issues where the acquisition would freeze and never complete. It is not an issue with where the data is being saved, as there is over 1TB of free space on the disk.

For smaller acquisitions (say 500 frames) the acquisition will sometimes complete successfully, however unreliably, and in this case, it will start sending images to the sequence buffer. It is not an issue with the Camera, nor the 10GigE connection, as running the same acquisition in its Native software seems to work just fine. I have attached some images of the what it looks like when the Acquisition fails, and below is the core log:

2020-10-22T11:04:27.574758 tid19272 [IFO,dev:Oryx ORX-10G-51S5M] Spinnaker: Failed waiting for EventData on NEW_BUFFER_DATA event. [-1011]

2020-10-22T11:04:27.574758 tid19272 [IFO,dev:Oryx ORX-10G-51S5M] SeqAcquisition interrupted by the user

2020-10-22T11:04:28.290828 tid19272 [IFO,dev:Oryx ORX-10G-51S5M] Sequence thread exiting

2020-10-22T11:04:28.440560 tid17256 [IFO,App] Acquisition failed.
[ ] java.lang.Exception: Timed out waiting for image to arrive from camera. in Thread[clojure-agent-send-off-pool-0,6,main]
[ ] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[ ] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[ ] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[ ] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[ ] at clojure.lang.Reflector.invokeConstructor(Reflector.java:183)
[ ] at org.micromanager.acq_engine$throw_exception.invoke(acq_engine.clj:77)
[ ] at org.micromanager.acq_engine$pop_tagged_image_timeout.invoke(acq_engine.clj:380)
[ ] at org.micromanager.acq_engine$pop_burst_image.invoke(acq_engine.clj:390)
[ ] at org.micromanager.acq_engine$pop_burst_images$fn__1034.invoke(acq_engine.clj:416)
[ ] at org.micromanager.acq_engine$queuify$fn__1027$fn__1028.invoke(acq_engine.clj:407)
[ ] at org.micromanager.acq_engine$queuify$fn__1027.invoke(acq_engine.clj:407)
[ ] at clojure.core$binding_conveyor_fn$fn__3713.invoke(core.clj:1817)
[ ] at clojure.lang.AFn.call(AFn.java:18)
[ ] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[ ] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[ ] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[ ] at java.lang.Thread.run(Thread.java:748)

2020-10-22T11:04:28.768601 tid17184 [IFO,App] 6883 images stored in 83245 ms.

2020-10-22T11:08:35.377993 tid13912 [IFO,App] EDTHangLogger: Stopping monitoring of EDT hangs

2020-10-22T11:08:35.940350 tid13912 [IFO,dev:COM4] TERM_TIMEOUT error occured!

2020-10-22T11:08:35.940350 tid13912 [ERR,Core] Error occurred in device COM4: Error in device "COM4": (Error message unavailable) (107)

2020-10-22T11:08:36.932530 tid13912 [ERR,Core] Attempt to log message from unregistered device: Destructing MonitoringThread

MMfailure|689x413

Thanks for this summary @cfoltz.

I am looping in @henrypinkard, @nicost.
Hi Henry, Nico, the experiment is to acquire a fast time-series, do a z-stack in two channels, move to next position, and repeat.

We will appreciate your input in diagnosing where we are losing the data throughput. Our interim idea is to use the camera API (pyspin) for acquiring the data at high speed, and use pycro-manager/micro-manager for the rest of the coordination.

Thanks
Shalin

There is a simultaneous discussion at github (https://github.com/micro-manager/micro-manager/issues/987) where @marktsuchida provided some ideas/things to test.

Where to best conduct these kind of discussions is often a bit nebulous. I suspect that image.sc will get more eyes, and if it is not directly a bug/clear issue in the MM code, this forum may be the better place.

If you think this may be a better avenue for discussion, I am happy to remove the github issue and continue the conversation here.

You choose (it mainly is confusing to have the same issue in different places).

Replying to a point made by @marktsuchida on Github:

This sounds like a tough problem and I only have vague suggestions.

The fact that it works with USB3, together with the error being generated by Spinnaker, suggests that it is something happening at the driver or OS level, whereas the fact that it works with the vendor software suggests otherwise. So one possibility is that there is some long-range interaction going on due to Micro-Manager using more resources than the vendor software.

It might be worth trying with both the sequence buffer and the ImageJ memory set to smaller values (just enough to not run out, and in total less than your physical RAM). These settings, when huge, can affect the layout and management of memory by the operating system in significant ways. (Although very unlikely to be related, it also happens that the Java Virtual Machine tends to perform slightly better when its memory (ImageJ memory) is set to 32 GiB or less.)

Another possibility is that Micro-Manager is using the CPU (or, more likely, memory bandwidth) too much, or simply not retrieving images from the driver fast enough because of doing too much work on the same thread in between, such that the driver’s buffer is overflowing. Maybe disable display of MDA images (Tools > Options) or try running the acquisition from a script using only MMCore methods. This would have to be a subtle effect, though, because Micro-Manager should be doing the same things when using USB3. Nonetheless, it might help to get an idea of the time scale. If you decrease the frame rate instead of the frame count, at what point does the acquisition start to run reliably?

I have retried the Acquisition by lowering the Java-side memory to 20GB, and the sequence buffer to 40GB at a variety of framerates. I also disabled the option that displays MDA images. The Acquisition completes at 150 fps with 500 images at 1 ms exposure (~4GB total). The Acquisition fails at 170 fps with 500 images at 1ms exposure (changing down pixel value to 8bit with same ROI).I can explore using straight core commands, but this acquisition also failed using pycromanager MDA.

The reproducibility of this error seems to vary for each attempt over time, where sometimes it can acquire a time series at 300 fps and other attempts only 100 fps. I was able to workaround this issue by using a python wrapper for the vendor software for the time being.

@Elliot_Steele, any ideas?

Hi all,

Not too surprised this isn’t working, the adapters were written for much slower cameras than the 10GigE.

I’ve had a quick look over the source code and they’re its currently doing some unnecessary allocations and deallocations when frames come in. That could definitely slow it down enough to drop frames, particularly with large images. I’ve also spotted a bug in that area of the code so I’ll try making a few tweaks. Unfortunately, I don’t have a point grey camera at the moment so I’m going to need someone to test any changes before we merge them

In the meantime, the only other thing that I can think of off the top of my head is to avoid the 12-bit modes (if they’re available). They require some extra processing of images before they can be handed off to micromanager, that could also easily cause missed frames

Hi Elliot,

I would be happy to test any changes that are made to the device adapter and report back. Thanks for looking into this! We can correspond over email if thats easier for you.

@Elliot_Steele. We currently work with a modified version of the Spinnaker adapter courtesy of @ieivanov to which he added a few important functions. You can find the code that we work with here:

https://github.com/ieivanov/micro-manager/tree/SpinnakerCamera/DeviceAdapters/Spinnaker

Ah, that’s awesome. I’ll use the modified version as a starting point for any changes I make. I’ll let you know once I’ve had a chance to play with it

@ieivanov are you happy to have your changes merged back into the main micromanager branch, they look like really useful additions even if we don’t manage to fix the missing frame issues

Elliot

I would try with a 6GiB sequence buffer if testing with ~4GiB images, just in case. Chances are not high that this will actually help, but it’s good to eliminate the possibility that it has to do with the overhead of allocating and accessing a large block of memory. (It is generally counterproductive to make the buffer larger than necessary, because the operating system then needs to shuffle around memory pages (4KiB blocks) upon first access. If the sequence buffer is larger than the available physical RAM, it may even need to swap to disk, or swap some other important memory to disk, which makes behavior even more unpredictable. The MMCore sequence buffer uses the frame buffers in cyclic order (reusing the most recently used buffers would have been better, at least for the case where the total buffer size is large), so this can have a surprisingly large impact under certain conditions.)

Running straight core commands is perhaps still worth trying, because pycro-manager will have much higher overhead. Also, I would compare the results from identical settings (for things like pixel bit depth) with only exposure changed, although you’ve perhaps already tried that.

@Elliot_Steele I’ll work on merging my changes into the main micromanager branch. In the meantime, you’re welcome to work off of my fork.

Ivan

@Elliot_Steele I was going to mention that to make the Oryx camera work we’d need to update the Spinnaker SDK version that the device adapter relies on. The instructions on the micro-manager website (https://micro-manager.org/wiki/Spinnaker) call for version 1.20.0.15, which is available through Cairn Research (I’m not sure what their connection here is). Through the FLIR website (https://www.flir.com/products/spinnaker-sdk/) we can access archived versions 1.29.0.5 and 2.0.0.147. Version 2.0.0.147 is what I have been using (maybe you asked them to add it to the archive?) but I’m happy to update to a more current version of the SDK if FLIR would keep an archived copy of it for us.

What do you think? I’m happy to test the latest version of the SDK to make sure there are no problem and then we can ask FLIR to add it to their archive. We’d then update the instruction for the device adapter on the micro-manager website. Does that sounds like a good plan?