We are looking into acquiring a fast time series (>500 fps) over the course of about 10 seconds with a FLIR Oryx 10GigE Camera. We have demonstrated on our system that this is possible without losing frames in the native Spinnaker software. However, we wish to pair this with a few other types of acquisitions (z-stacks at wider FOV) through custom acquisition scripts with pycromanager/MM core commands. I was wondering if anyone has successfully done an acquisition at this speed through micromanager? Here are the issues I have been running into thus far:
To test, I would run an MDA with 10,000 frames, at 0 ms interval so acquire the data as fast as possible (10 sec time series at 1ms exposure). Micromanager is seemingly able to capture about 6000 frames but for the remaining frames, it is unable to grab the images from the camera, then MM times out, and the acquisition fails.
I am able to do this fine on a camera at the same fps but with a USB3.0 connection. It seems to not be an issue with the buffer size, as this acquisition is only ~11 GB in total and the buffer size was set to 50 GB (64 GB RAM computer). I also changed the Java Memory under: ImageJ > Edit > Options > Memory & Threads to 40 GB. I also monitored both this memory and it did not ever send images to the sequence buffer, but got up to 23% memory in the memory monitor. I tried this acquisiton in pycromanager, and received similar issues where the acquisition would freeze and never complete. It is not an issue with where the data is being saved, as there is over 1TB of free space on the disk.
For smaller acquisitions (say 500 frames) the acquisition will sometimes complete successfully, however unreliably, and in this case, it will start sending images to the sequence buffer. It is not an issue with the Camera, nor the 10GigE connection, as running the same acquisition in its Native software seems to work just fine. I have attached some images of the what it looks like when the Acquisition fails, and below is the core log:
2020-10-22T11:04:27.574758 tid19272 [IFO,dev:Oryx ORX-10G-51S5M] Spinnaker: Failed waiting for EventData on NEW_BUFFER_DATA event. [-1011] 2020-10-22T11:04:27.574758 tid19272 [IFO,dev:Oryx ORX-10G-51S5M] SeqAcquisition interrupted by the user 2020-10-22T11:04:28.290828 tid19272 [IFO,dev:Oryx ORX-10G-51S5M] Sequence thread exiting 2020-10-22T11:04:28.440560 tid17256 [IFO,App] Acquisition failed. [ ] java.lang.Exception: Timed out waiting for image to arrive from camera. in Thread[clojure-agent-send-off-pool-0,6,main] [ ] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [ ] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [ ] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [ ] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [ ] at clojure.lang.Reflector.invokeConstructor(Reflector.java:183) [ ] at org.micromanager.acq_engine$throw_exception.invoke(acq_engine.clj:77) [ ] at org.micromanager.acq_engine$pop_tagged_image_timeout.invoke(acq_engine.clj:380) [ ] at org.micromanager.acq_engine$pop_burst_image.invoke(acq_engine.clj:390) [ ] at org.micromanager.acq_engine$pop_burst_images$fn__1034.invoke(acq_engine.clj:416) [ ] at org.micromanager.acq_engine$queuify$fn__1027$fn__1028.invoke(acq_engine.clj:407) [ ] at org.micromanager.acq_engine$queuify$fn__1027.invoke(acq_engine.clj:407) [ ] at clojure.core$binding_conveyor_fn$fn__3713.invoke(core.clj:1817) [ ] at clojure.lang.AFn.call(AFn.java:18) [ ] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [ ] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [ ] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [ ] at java.lang.Thread.run(Thread.java:748) 2020-10-22T11:04:28.768601 tid17184 [IFO,App] 6883 images stored in 83245 ms. 2020-10-22T11:08:35.377993 tid13912 [IFO,App] EDTHangLogger: Stopping monitoring of EDT hangs 2020-10-22T11:08:35.940350 tid13912 [IFO,dev:COM4] TERM_TIMEOUT error occured! 2020-10-22T11:08:35.940350 tid13912 [ERR,Core] Error occurred in device COM4: Error in device "COM4": (Error message unavailable) (107) 2020-10-22T11:08:36.932530 tid13912 [ERR,Core] Attempt to log message from unregistered device: Destructing MonitoringThread