[Micro-manager] Impossible to modify AnswerTimeout of adapter

Hello,

I 'm using MM to drive an ASI© XYZ stage + Piezo + LED and I am having troubles while trying the software autofocus routines (mostly during MDA, it occurs less during test mode). In fact, when I launch the MDA with autofocus, MM returns an “error in device : Serial command failed. Is the device connected to the serial port? (14)” for all the ASI serial adapters. Everything works like a charm without the autofocus routine ON.

Therefore, my guess is that the ASI devices are a bit slower than the AnswerTimeout of the adapter (500ms) in some cases, and return the serial error. However, I can’t change this value in the adapter manager, it’s simply not applying when I enter a new value.

Additional info:

  • I’m using USB to control the ASI adapters on w10 and will try RS232 ASAP.
  • Very last version of MM
  • Energy mode of w10 is set on high performances and I disabled USB sleep.
  • When I go the the adapter managers, MM tells me “Your driver library is not up todate. We strongly recommend to update the device driver. Your library version: 3.2.4. Recommanded library version: >= 3.2.7” but I don’t get why since everything is up to date (Windows, MM, ASI drivers) and it tells me the same even when I remove all adapters. What is it referring to?

Thus:

  1. Do you know if the serial error during the autofocus routine of the MDA could arise from the timeout being too low?
  2. Do you know how to change the AnswerTimeout value of the ASI adapters?
  3. Could my problem come from the “USB drivers library not to date”?

I hope Nico sees this message :slight_smile: !
Thanks!
Théo

1 Like

Hi Théo,

I did see your message;).

The timeout you see indicates that the ASI controller does not respond to a command send by the computer. If you have reason to believe that the controller needs more time, you can increase the time out (which is set in the serial port that you select in the Hardwar Configuration Wizard, default is 500ms). However, it most likely means that the commands send by Micro-Manager during software autofocus confused the controller (and/or the device adapter code). To see what is happening, generate a Trouble Report (Help > Report a Problem). If you send it, I’ll have a look at it, and see if there is something obvious (and possible ask ASI for ideas).

The driver library error apparently originates in one of the device adapters. @marktsuchida and Max figured out which one, and I thought that they removed the code generating the warning. In any case, you can safely ignore that warning.

1 Like

Dear Nico,

Thanks a lot for this super fast answer.

The problem is that I can’t do that, it doesn’t work! Whatever value I put, it gets back to 500ms.

What makes me think it’s a timeout problem it’s because the appearance of the error is erratic (= not 100% systematic). Out of the MDA, the Autofocus works without error 90% of time and in the MDA with multiposition ON, it sends the serial error 95% of the time. So my hypothesis is that the answer time of my device must be close to 500ms.

Anyway, here is the report: You can stop reading after Line 1400 because the same round of serial errors keeps appearing at each Z of the autofocus.
Problem Report.txt (198.3 KB)

Thank you so much Nico!

EDIT : I tried to drive the device in RS232, the problem remains the exact same.

Glancing through the problem report I see something amiss. Specifically either the device adapter doesn’t wait for the controller to acknowledge the LED X command or else the controller doesn’t offer an acknowledgement. The next command leads to a serial timeout. I’m guessing the erratic nature of the bug relates to whether that mismatch in communication gets cleared in time (on either the PC or controller side).

I am looking into it.

1 Like

I cannot replicate the behavior here; the controller always acknowledges the LED X command for me.

I did correct an obvious bug in the device adapter. Previously the device adapter simply looked the other way when the controller did not acknowledge the LED X command. Now it will give an internal error like it should. I doubt it will fix @Theo_A’s problem but at least it should make the root cause more clear.

@nicost, can you please get my SVN checkin into the MM2 gamma build for @Theo_A to try? @Theo_A, can you please generate a similar problem report with the updated build and see if anything is different for you?

1 Like

Dear @jondaniels,

Thank you so much for your help.
I will try again as soon as the new build is released.

Your help is much appreciated, I love this community!

Problem may be related to two different threads talking through the same serial port at the same time. See:

2020-11-13T20:48:17.438896 tid13272 [dbg,dev:COM17] SetCommand -> LED X=20\r
2020-11-13T20:48:17.438896 tid12452 [dbg,dev:COM17] SetCommand -> W F\r
2020-11-13T20:48:17.940733 tid12452 [IFO,dev:COM17] TERM_TIMEOUT error occured!
2020-11-13T20:48:17.940733 tid13272 [IFO,dev:COM17] TERM_TIMEOUT error occured!

The tid is the thread number. @Theo_A, you do not happen to have the Stage control panel open and have “polling updates” checked?

@jondaniels, is it conceivable that the problem is indeed related to two threads sending a command to the controller at the same time? Just from memory, the serial adapter likely has a lock, and I believe that there is also some per-device adapter lock in place. If relevant, @marktsuchida may remember (or we can look at the code;).

Also, I just ported svn changes to 2.0gamma including @jondaniels ’ latest bug fix.

@Theo_A: The serial port in your report is set to a 500ms timeout. Are you saying that the HCW does not let you change that number? Let me know if that is really the case (that would be a new bug), but in the mean time you can always edit the config file in a text editor.

1 Like

Ok I managed to get more details about the reproducibility of the bug. (I didn’t built yet with the new SVN).

No the Stage control has always been closed (and polling updates unticked). However from that, I noticed that the autofocus routine worked perfectly UNTIL I open the Stage Position List. From then I get the serial error (from the threads conflict?) while trying to autofocus, in any conditions, even after closing the Stage Position List. That also explains the behavior of the bug that I described as “erratic” (in reality, it’s not).

Attached is a new report. To generate it, I did 1) An autofocus without having opened the Stage Position List, then 2) Opening of the Stage Position List 3) Closing of the Stage Position List 4) Autofocus --> serial error
Problem Report_2.txt (88.5 KB)
I think (not sure) that 4) starts at Line 1123.

Yes, exactly this. The HCW lets me change everything of the adapter BUT the AdapterTimeout. (But now as I understand, timeout was not the source of my problem)

I can build MM with the last SVN and do another try now.

Thanks a lot.

ADDENDUM: To rule out one thing, this bug occurs identically if the Core focus is set to the linear Z stage or to the Piezo stage.

@nicost good catch about the different threads talking to the controller! I’m guessing that’s the root cause of the problem.

The ASIStage device adapter allows multiple MM devices, all talking to the same controller over the same comm port (this is an old device adapter architecture; these days a hub device would be used instead but I think this device adapter is quite old). I don’t know how a lock mechanism would be implemented but we probably need something like that… There is at least an ASIBase class that handles communication across the different device classes.

1 Like

I reproduced the bug with the 13/11 nightly build (I guess it contains Jon’s update).

Attached is the report, this time annotated for more readibility. Again, I did:

  1. Launching a fresh µM session --> autofocus routine without having opened the Stage Position List: no error
  2. Opening of the Stage Position List
  3. Closing of the Stage Position List (or not, doesn’t change anything)
  4. Autofocus --> serial error
    Problem Report_3.txt (74.4 KB)

From my limited understanding and your remarks, I think that opening the Stage Position List opens a new tid (here, tid4600), that then conflicts with the tid used for the Autofocus routine (here, tid9336).

@nicost, I tried reproducing the bug with the Stage control panel opened and polling updates ticked, but it did not appear. Only opening the Stage Position List could make it occur.

Of note, this bug doesn’t occur if the shutter used for the routine is not the ASI LED (I tried the AF routine with the same setup but with a Thorlabs4100 as shutter, and could not get the bug).

I wish I could help you more.
Thanks again!

Most likely, there are multiple issues at play. One of them is that two different threads are talking through the (emulated) serial port to the ASI controller. @jondaniels: there is locking in place in the Core, however, it only locks all devices of the same type (for instance, GetXYPosition locks all XYStages). I am not sure what the easiest fix is at this point. I guess this could all be handled in the ASI adapter, but most solutions would be ugly (independent classes would need to share a lock somehow), and the same situation will occur in other places. It just occurred to me that adding the possibility for the code to acquire a lock on the serial port itself could be useful. @marktsuchida: any ideas of something that could be done quick to solve this?

The other part of it is the Stage Position panel itself. The thread that is asking for an update of the piezo position is a callback coming from the Core, informing the Stage Position Panel that a stage position changed. The Stage Position Panel then asks (i.e. asks the hardware) all stages in the system about their current position. Looking at this for a few seconds, I have the feeling this is not needed, since the callback already contains the new position of the stage that moved. I may try to see what happens if we use the callback information directly to update the current position info and the display, rather then ask the hardware again (which is bound to have significant overhead). If that works, @Theo_A’s problem should disappear.

@Theo_A : is not opening the Stage Position List or using another shutter a doable work-around until we figure this out?

1 Like

Thanks Nico for spending time on this.

Naïve question, but would it be possible to put a checkbox in the stage position panel to disable auto-update for example? Or to disable auto-update when the Position Panel is closed?

Well I’m doing timelapses on multiple positions so I need to open the Stage Position List at least once…I’ll wait it’s alright!
Will keep you updated if I find more info or a viable workaround to share with other people using Autofocus with ASIStage+LED. In the meantime I’m ofc available if you need to try any potential patch.

Thank you again!

Hi @nicost and @marktsuchida,

What do you think is the best way forward on this? Can there be a lock on the serial port? (realizing that the serial port can shared by multiple devices)

Would it for sure resolve the issue if someone (probably me) bit the bullet and wrote an “ASIStage2” device adapter with a hub device? (Many people have generously contributed to the ASIStage device adapter over the years (since at least 2007), but I think you’ll agree that the code is long in the tooth… The ASITiger device adapter would be a better starting point I think.)

1 Like

I just submitted a PR (https://github.com/micro-manager/micro-manager/pull/1009) that stops the PositionListDlg from talking to the hardware after receiving a callback that a stage position was updated. The rational for going back to the hardware rather than using the position information in callback was not clear to me to begin with, and this should be beneficial for everyone. This change should be in tonight’s (20201116) and later builds.

This should solve @Theo_A’s issue. The longer term questions are whether or not the API should provide a method to lock the serial port (@marktsuchida ?), and the status of the ASI Stage device adapter. Hopefully the fix to the PositionListDlg will make these issue less pressing.

1 Like

Dear Nico and Jon,

Thanks again for the time spent on my problem.

This worked! I managed to launch several autofocus routines with the Position List opened without getting any error.
Thanks a lot, @nicost. I’m going to make further tests just to be sure, but I can mark the topic as solved unless you want to continue exchanging about locking the port with Jon and Mark.
Btw I’m available if you want to test anything related to the ASI Stage and controller, such as a new adapter.

I hope to also be able to contribute to the MM community soon!