Occasional crashes when starting acquisition

Usage & Issues
#1

We’re seeing occasional crashes when starting an acquisition in Micro-Manager 2.0 gamma, 11-19-2020 nightly build. It does not happen often enough to affect our ability to do experiments, but has happened enough times that we thought we should report it.

The error appears to be some kind of timeout, but I don’t know enough about the guts of the acquisition engine to troubleshoot further. I’ll attach the core log file, which has our configuration details at the top and the crash at the end.

Thanks for any advice.

CoreLog20201124T074217_pid8008.txt (9.4 MB)

#2

Looks like a deadlock. The interesting part is the output of the EDT monitoring thread (designed to catch these occurrences):

Thread 19 [AWT-EventQueue-0] BLOCKED
[ ] at org.micromanager.internal.SnapLiveManager.setSuspended(SnapLiveManager.java:202)
[ ] at org.micromanager.internal.ConfigGroupPad$StateTableData.setValueAt(ConfigGroupPad.java:199)
[ ] at javax.swing.JTable.setValueAt(JTable.java:2741)

The AWT is the thread running the UI, so it is blocked you will not be able to do much. The other thread that is blocked is the one running the acquisition engine:

Thread 302 [AcquisitionEngine2010 Thread (Clojure)] BLOCKED
[ ] at java.awt.Component.enable(Component.java:1488)
[ ] at javax.swing.JComponent.enable(JComponent.java:3623)
[ ] at java.awt.Component.enable(Component.java:1513)
[ ] at java.awt.Component.setEnabled(Component.java:1478)
[ ] at javax.swing.JComponent.setEnabled(JComponent.java:2680)
[ ] at javax.swing.AbstractButton.setEnabled(AbstractButton.java:2091)
[ ] at org.micromanager.quickaccess.internal.controls.SnapButton$3.onLiveMode(SnapButton.java:114)
[ ] at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
[ ] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ ] at java.lang.reflect.Method.invoke(Method.java:498)
[ ] at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
[ ] at com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
[ ] at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
[ ] at com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:304)
[ ] at com.google.common.eventbus.EventBus.post(EventBus.java:275)
[ ] at org.micromanager.events.internal.DefaultEventManager.post(DefaultEventManager.java:46)
[ ] at org.micromanager.internal.SnapLiveManager.setLiveModeOn(SnapLiveManager.java:181)
[ ] at org.micromanager.SnapLiveManager.setLiveMode(SnapLiveManager.java:78)
[ ] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ ] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ ] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ ] at java.lang.reflect.Method.invoke(Method.java:498)
[ ] at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:92)
[ ] at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:30)
[ ] at org.micromanager.acq_engine$run_acquisition.invoke(acq_engine.clj:783)
[ ] at org.micromanager.acq_engine$run$fn__1421.invoke(acq_engine.clj:982)
[ ] at clojure.lang.AFn.run(AFn.java:24)
[ ] at java.lang.Thread.run(Thread.java:748)

So, for whatever reason, both threads seem to be talking to the SnapLiveManager, and somehow one is holding a lock, but it is waiting for the other thread to finish, whereas that other thread is waiting for the lock to be released that is held by the first.

I am wondering if this could be caused by a callback from the shutter. What is the “488 + 638” shutter (i.e., what kind of device is that shutter)? Would be worth to look if it calls back when it is opened or closed. The deadlock is certainly a bug, but trying to figure out what specific thing in your setup could cause it (as I have not seen this one reported before;).

Tagging @marktsuchida as this should be fixed before 2.0 release.

#3

Hi Nico,
Thanks for the reply and sorry for taking a couple days to get back to you. The “488 + 638” shutter is a multi-shutter device, created from the Utilities device menu in the configurator. The individual shutters are 488 nm and 638 nm Oxxius lasers.

#4

Hi @dannyhmg,

I pushed a change that - I think - will fix this deadlock. It should be in builds 20201221 and later. Curious to head if that indeed fixed the problem for you.

#5

Thanks Nico, I installed the latest nightly build and so far everything seems to work. Since we were only seeing these deadlocks occasionally, it may be a while before we know for sure whether the problem has been solved, but I’ll let you know if we have any more issues.
Cheers,
Dan

1 Like