Hardware triggering behavior

My saga to increase z-stack acquisition speeds continues. I had finally figured out how to handle camera-to-camera hardware triggering and gotten sequencing working with our ASI MS-2000 stage. Trying to use the “Fast sequencing” mode where the stage no longer requires serial commands and steps through a ring of z-positions based on trigger inputs from the cameras starts generating odd behaviors.

Over the course of an acquisition, the frame sequence and z position seem to become unsynchronized, causing the sample to jump up/down in the z stack every few hundred frames as though triggers were being missed / not generated / or excess triggers were being received.

Without access to an o-scope, this has been difficult to nail down, but one additional test seems to point to a micro-manager related issue rather than hardware.

If I initialize the ring buffer and arm fast sequencing but then run an acquisition from the MDA window that has only timepoints and no z-stack, then rearrange the hyperstack after the acquisition, there’s no jumping!

Something about running with timepoints and z-stacks in MDA with fast sequencing enabled seems to cause problems. I’m trying to get my hands on an o-scope or re-flash the firmware on my triggerscope to use it as a pulse counter to figure out exactly what’s going wrong with the MDA in terms of the number of pulses coming from the camera to trigger the stage, but for folks more intimately familiar with the acquisition engine / MDA code, does this behavior make sense in any way?

I seem to recall that you have 2 cameras… is the Multi Camera device acting as your acquisition camera?

If so maybe that is part of the mystery. In the diSPIM plugin we have to take special care to read which camera each image comes from to make sure it gets filed appropriately, and maybe the MDA isn’t doing the equivalent with Multi Camera.

In my experience, tagged frames from each camera still enter the sequence buffer in a deterministic order when using multi-camera. I just ran a test using only the camera that’s the trigger-source and I still see the lost / extra (still not sure which) z position jumps. Running as a time-only MDA with fast-sequencing on works properly just as with multi-camera enabled.

My guess is that, with z-stacks enabled, the acquisition engine is still running each timepoint as its own sequence acquisition with frames = number of slices causing extra overhead (there seems to be a roughly 200 ms extra delay between timepoints in this mode). Running in time only probably just initializes one long sequence acquisition with frames = number of timepoints that works faster (since each timepoint doesn’t need to be software re-triggered) and thus behaves normally.

In order to explain the behavior I see, that would mean that starting sequence acquisitions by software over and over again generates an extra trigger event on the camera(s) for some reason, but only occasionally? That would also be very odd and I have a difficult time imagining what could cause that. Or is the acquisition engine still sending serial commands to the stage, even with fast sequencing turned on, and that occasionally causes the stage controller to misbehave in some way? Does the acquisition engine ever send anything other than move commands to the stage? Would a flurry of serial move commands hitting the stage while it’s in fast sequence mode cause it to occasionally misfire? Presumably the controller is ignoring move commands but must still be parsing serial commands.

A non-zero interval also doesn’t seem to affect the results, as long as the MDA acquisition is time-only with z-stacks disabled, the fast sequenced acquisition executes properly without skipping / jumping in Z.

I’m not sure whether the MDA sends additional serial commands to the controller/stage even with the “Use Fast Sequence” feature enabled/armed. You can monitor the serial traffic using a serial logger/spy program or else after the fact by either starting a problem report and viewing before sending or else turning on debug logging and looking in the resulting CoreLog file.

It appears from the device adapter’s source code that the “Use Fast Sequence” property only short-circuits sequence-related operations and busy queries. If it turns out that the MDA is doing other things (e.g. sending other serial commands) then maybe other operations also need to be short-circuited.

So when using fast sequence and keeping z-stacks enabled in MDA, it does indeed look like each timepoint is being configured as a separate burst acquisition (hence leading to the inter-timepoint overhead I see). That’s probably not practical to “fix” since the acquisition engine would then need better a priori information about how fast each timepoint takes, whether that fits perfectly into the configured interval, etc. For the fastest possible acquisitions it seems clear that running in time-only mode is the way to go. It also seems that the acquisition engine is indeed trying to manually fly-back and start a stage sequence even though fast sequencing is turned on and that shouldn’t be necessary.

I should note that comparing the set-up commands issued right before a timepoint where a skip occurs and every other timepoint shows no differences.

2020-10-14T15:14:56.244387 tid14300 [dbg,App] [AE] BEGIN channel properties and exposure
2020-10-14T15:14:56.245385 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraDevice)
2020-10-14T15:14:56.245385 tid14300 [dbg,App] [AE] --> Multi Camera
2020-10-14T15:14:56.245385 tid14300 [dbg,App] [AE] END channel properties and exposure
2020-10-14T15:14:56.245385 tid14300 [dbg,App] [AE] acq-sleep
2020-10-14T15:14:56.245385 tid14300 [dbg,App] [AE] BEGIN set z position
2020-10-14T15:14:56.245385 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc waitForDevice “ZStage”)
2020-10-14T15:14:56.245385 tid14300 [dbg,Core] Waiting for device ZStage…
2020-10-14T15:14:56.245385 tid14300 [dbg,Core] Finished waiting for device ZStage
2020-10-14T15:14:56.245385 tid14300 [dbg,App] [AE] --> nil
2020-10-14T15:14:56.246384 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc setPosition “ZStage” 54.9)
2020-10-14T15:14:56.246384 tid14300 [dbg,Core] Will start absolute move of ZStage to position 54.90000 um
2020-10-14T15:14:56.246384 tid14300 [dbg,dev:COM4] SetCommand -> M Z=549.000000\r
2020-10-14T15:14:56.276337 tid14300 [dbg,dev:COM4] GetAnswer <- :A \r\n
2020-10-14T15:14:56.276337 tid14300 [dbg,App] [AE] --> nil
2020-10-14T15:14:56.276337 tid14300 [dbg,App] [AE] END set z position
2020-10-14T15:14:56.276337 tid14300 [dbg,App] [AE] pending devices: #{“ZStage”}
2020-10-14T15:14:56.277336 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc waitForDevice “ZStage”)
2020-10-14T15:14:56.277336 tid14300 [dbg,Core] Waiting for device ZStage…
2020-10-14T15:14:56.277336 tid14300 [dbg,Core] Finished waiting for device ZStage
2020-10-14T15:14:56.277336 tid14300 [dbg,App] [AE] --> nil
2020-10-14T15:14:56.277336 tid14300 [dbg,App] [AE] BEGIN acquire
2020-10-14T15:14:56.277336 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getAutoShutter)
2020-10-14T15:14:56.277336 tid14300 [dbg,App] [AE] --> true
2020-10-14T15:14:56.278334 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getSystemStateCache)
2020-10-14T15:14:56.279332 tid14300 [dbg,App] [AE] --> #<Configuration mmcorej.Configuration@39e1b7f7>
2020-10-14T15:14:56.281329 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc setAutoShutter true)
2020-10-14T15:14:56.281329 tid14300 [dbg,Core] Autoshutter turned on
2020-10-14T15:14:56.281329 tid14300 [dbg,App] [AE] --> nil
2020-10-14T15:14:56.281329 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraDevice)
2020-10-14T15:14:56.281329 tid14300 [dbg,App] [AE] --> Multi Camera
2020-10-14T15:14:56.282327 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc hasProperty “Multi Camera” “ExtraTriggers”)
2020-10-14T15:14:56.282327 tid14300 [dbg,App] [AE] --> false
2020-10-14T15:14:56.283326 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getFocusDevice)
2020-10-14T15:14:56.283326 tid14300 [dbg,App] [AE] --> ZStage
2020-10-14T15:14:56.283326 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc startStageSequence “ZStage”)
2020-10-14T15:14:56.283326 tid14300 [dbg,App] [AE] --> nil
2020-10-14T15:14:56.284324 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraDevice)
2020-10-14T15:14:56.284324 tid14300 [dbg,App] [AE] --> Multi Camera
2020-10-14T15:14:56.285323 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc hasProperty “Multi Camera” “OutputTriggerFirstMissing”)
2020-10-14T15:14:56.285323 tid14300 [dbg,App] [AE] --> false
2020-10-14T15:14:56.285323 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc startSequenceAcquisition 9 0 true)
2020-10-14T15:14:56.286367 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraDevice)
2020-10-14T15:14:56.286367 tid14300 [dbg,App] [AE] --> Multi Camera
2020-10-14T15:14:56.286367 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc hasProperty “Multi Camera” “OutputTriggerFirstMissing”)
2020-10-14T15:14:56.287319 tid14300 [dbg,App] [AE] --> false
2020-10-14T15:14:56.287319 tid14300 [dbg,Core] Will start sequence acquisition from default camera
2020-10-14T15:14:56.287319 tid14300 [dbg,App] Notification for Device: LaserShutter Property: State changed to value: 1
2020-10-14T15:14:56.287319 tid14300 [dbg,Core] Waiting for device LaserShutter…
2020-10-14T15:14:56.287319 tid14300 [dbg,Core] Finished waiting for device LaserShutter
2020-10-14T15:14:56.332249 tid14300 [dbg,Core] Waiting for device LaserShutter…
2020-10-14T15:14:56.332249 tid14300 [dbg,Core] Finished waiting for device LaserShutter
2020-10-14T15:14:56.375182 tid14300 [dbg,Core] Did start sequence acquisition from default camera
2020-10-14T15:14:56.375182 tid14300 [dbg,App] [AE] --> nil
2020-10-14T15:14:56.375182 tid14300 [dbg,App] [AE] collecting image(s)
2020-10-14T15:14:56.376226 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraDevice)
2020-10-14T15:14:56.376226 tid14300 [dbg,App] [AE] --> Multi Camera
2020-10-14T15:14:56.376226 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc hasProperty “Multi Camera” “OutputTriggerFirstMissing”)
2020-10-14T15:14:56.377218 tid14300 [dbg,App] [AE] --> false
2020-10-14T15:14:56.377218 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getNumberOfCameraChannels)
2020-10-14T15:14:56.377218 tid14300 [dbg,App] [AE] --> 2
2020-10-14T15:14:56.377218 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraChannelName 0)
2020-10-14T15:14:56.378220 tid14300 [dbg,App] [AE] --> LongWavelength Camera
2020-10-14T15:14:56.378220 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraChannelName 1)
2020-10-14T15:14:56.378220 tid14300 [dbg,App] [AE] --> ShortWavelength Camera
2020-10-14T15:14:56.378220 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc getCameraDevice)
2020-10-14T15:14:56.378220 tid14300 [dbg,App] [AE] --> Multi Camera
2020-10-14T15:14:56.379215 tid14300 [dbg,App] [AE] <-- (. org.micromanager.mm/mmc hasProperty “Multi Camera” “OutputTriggerFirstMissing”)
2020-10-14T15:14:56.379215 tid14300 [dbg,App] [AE] --> false
2020-10-14T15:14:56.380176 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.397176 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.403139 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.409129 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.414121 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.420112 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.426103 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.434090 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.440081 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.447069 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.453060 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.461049 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.467039 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.473030 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.478022 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.485011 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.491001 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms
2020-10-14T15:14:56.492998 tid25964 [dbg,App] Notification for Device: LaserShutter Property: State changed to value: 0
2020-10-14T15:14:56.497990 tid26284 [dbg,App] [AE] waiting for burst image with timeout 20100.06044117647 ms

Good sleuthing to see that the MDA is using a SetPosition call (which results in a serial command) between stacks.

I just committed a change to the ASIStage device adapter that short-circuits that SetPosition call when “Use Fast Sequence” is set to “Armed”. (Specifically, the SetPosition call still happens via the MDA but the device adapter will do nothing, similar to the waiting step which returns instantaneously in the “Armed” state.) I would appreciate you testing and letting me know if it resolves the problem. Best-guess is that currently the SetPosition move and TTL pulse are happening about the same time and on rare occasions the TTL pulse is somehow being missed (if so, the controller’s firmware could be improved).

The change is in the SVN repo. I’m not sure the process for getting that into the MM2 builds, but I do know it’s not automatic like it is with 1.4.x.

Thanks Jon!

Hi @nicost, would you be able to help us pull Jon’s new adapter code into the MM2 nightlies?

Merged. Should be in the 20201015 build and later.

B.t.w., I really like the Saleae logic analyzers. They have an educational/hobbyist price that is reasonable, and they are super useful for trouble shooting hardware triggering.