Micro-Manager crashing with Andor Sona camera

Dear Micro-Manager developers and other users,

I was wondering whether you might be able to direct us to someone who could help us debug the cause of the following Micro-Manager crashes? We urgently need Micro-Manager to work.

The steps that lead to the crashes are as follows:

System configuration:
Camera: Andor Sona-4BV6U 4.2B-6
Driver: AndorDriverPack3Setup-3.14.30022.0
USB 3.1 card: StarTech PEXUSB312A2 (provided with the Andor Sona camera)
PCIe slot: X16
Motherboard: X399 AORUS XTREME (rev. 1.0)
Micro-Manager: MMSetup_64bit_2.0.0-gamma1_20200527.exe 27-May-2020 19:53 150M
RAM: 8 GB
Sequence buffer size: 5120 MB

Crash scenario 1:

  1. Start Micro-Manager
  2. Click “Multi-D Acq.”
  3. Select “Multiple Positions (XY)”
  4. Click “Edit Positions List” and add a couple of positions by moving the stage and clicking “Mark”
  5. Click “Acquire!”
  6. Error message displayed: “Error: Internal inconsistency: unknown system exception occurred”
    CoreLog attached: CoreLog20200528T005528_pid15864-crashed-when-clicked-Acquire-in-multiD-with-XY-selected.txt
    Note to aid in debugging: I’m able to acquire images at multiple time points (as opposed to XY positions) in Multi-Dimensional Acquisition, as long as no other “dimensions” (e.g. XY, channels, etc.) are selected.

Crash scenario 2:

  1. Start Micro-Manager
  2. Click “Live” (images are displayed, no problem)
  3. Click autofocus button (binoculars)
  4. Error message displayed: “Error: Internal inconsistency: unknown system exception occurred”
    CoreLog attached: CoreLog20200528T005013_pid5688-crashed-when-clicked-autofocus-while-live-view-on.txt"
    Note to aid in debugging: Autofocus works without crashing if autofocus is started when no image is being displayed, as long as I first click menu item Tools > Refresh GUI.

Crash scenario 3:

  1. Start Micro-Manage
  2. Click “Live” (images are displayed, no problem)
  3. In the image window, click “Live” to stop acquisition, so “Snap” button becomes active
  4. Error message displayed: “Error: Internal inconsistency: unknown system exception occurred”
    CoreLog attached: CoreLog20200528T004907_pid11412-crashed-when-clicked-snap-after-stopping-live-view.txt

Note to aid in debugging: Note that images from the Sona are displayed without any problem by starting Micro-Manager and clicking “Live”. It’s when we try to do other things, like autofocus, multidimensional acquisition, or snap a picture, that Micro-Manager crashes with the “Error: Internal inconsistency” message. If I try to do anything after that error then all of the MicroManager windows close and I need to restart Micro-Manager.

The CoreLog files indicate that the “Internal inconsistency” error occurs when Micro-Manager calls “CMMCore::getImage()”.

The Andor SDK3 Micro-Manager page mentions the speed of the PCIexpress slot as a possible cause of the “Internal inconsistency” error, however our PCIexpress slot is X16 and we are able to acquire images even at high frame rates without crashes using the “Live” button.

I would be happy to provide additional information if needed to help debug.

Another question I have is, how do we keep the driver for the Andor Sona up to date? I was only able to get the version we have (AndorDriverPack3Setup-3.14.30022.0) by asking our sales person who provided a link to that one file. The link on the Andor SDK3 Micro-Manager page links to a page that says “The resource cannot be found”.

Thank you in advance for any help!

Best regards,

John Ramunas
ramunas@rejuvenationtech.com

CoreLog20200528T004907_pid11412-crashed-when-clicked-snap-after-stopping-live-view.txt (120.5 KB)

CoreLog20200528T005013_pid5688-crashed-when-clicked-autofocus-while-live-view-on.txt (118.0 KB)

CoreLog20200528T005528_pid15864-crashed-when-clicked-Acquire-in-multiD-with-XY-selected.txt (240.1 KB)

The Andor SDK3 device adapter was updated recently, and most likely not widely tested since. I contacted the engineers at Andor and expect they will look into it soon.

You may want to test an earlier version of Micro-Manager. The last major change happened March 17th 2020, so please do test a version from before that date (it will be helpful to know whether or not earlier versions have the same problem). Just running live with the camera, stopping live, then pressing Snap would be enough.

I also failed to find the Andor Driver Pack online. Andor re-organized their website, so it may just have gotten lost in the shuffle. Please do ask your Andor rep whether or not this driver pack will remain publicly accessible (you have a lot more sway with Andor than I do;).

B.t.w., you did a great job assembling all relevant data. We made this easier by providing the Help > Report a Problem menu entry that lets you directly submit issues like this.

Hi Nico,

Thanks for the fast response. Following your suggestion, I tested the March 16, 2020 version of Micro-Manager (MMSetup_64bit_2.0.0-gamma1_20200316.exe), running live with the camera, stopping live, then pressing Snap, and the same error occurred:

2020-05-28T19:50:40.883168 tid14500 [ERR,Core] Error occurred in device CMMCore::getImage(): Internal inconsistency: unknown system exception encountered
2020-05-28T19:50:40.883168 tid14500 [IFO,App] Failed to snap image

The CoreLog is attached: CoreLog20200528T194951_pid13952.txt (121.5 KB)

I also attach here screen shots of the three crash scenarios described in my original post above, in case they help:

Following your suggestion I asked Andor’s Product Support Engineer Mohammed Mustafees Khan whether the Andor Driver Pack 3 has moved to a new location on Andor’s site, and whether it will remain publicly accessible. I’ll follow up if the information is available.

Thank you again,
John

I realize this response is very late, but for what it’s worth, I have tested the newer Andor Drivers (from July, provided direct from Andor) on a recent mm2.0-gamma build (11-2-2020), and it is more stable. There are only some issues with live-mode streaming (it’s delayed – could be our computer), but MDA acquisition worked just fine.