I'm writing a device adapter for the BDPathway 435 (and 855)

Hi everyone,

I recently “acquired” a BD Pathway 435 and am attempting to write a Micro-Manager device adapter for it.

This rabbit hole goes as far as you wish it to:

  • Filterwheels, XYZ stage, shutters, spinning disk serial commands are well understood and (hopefully soon) will be controllable from my device adapter.
  • I have many additional commands related to liquid dispensing (BD Pathway 855) but there was no liquid handling devices in MM last I checked (?).
  • I haven’t started on the autofocus yet as it uses a NI PCI-6221 and I haven’t had a chance to break out the logic analyzer. Also, my AttoVision install seems to be borked…

At this point, this is just to gauge interest, there can’t be that many BD Pathways left, but they were nice machines way back when, and who knows? With a new camera, new filters and an LED light source, they could still have a very fruitful second life (with CellProfiler for the image analysis). At least that’s what I’m planning for mine.

If Nico is happy with me sending him a device adapter, I can do that once I can reliably talk to my 435.

Cheers,
Egor

Great to see you here Egor! That sounds like fun. Of course I’ll be happy to incorporate the device adapter!

Liquid handling devices are interesting. We could make such an abstraction, but this also clearly treads on other, general, lab automation. There are other projects out there (I am not well educated about them), and it will be very important to be as compatible as possible, or, preferably, use the same code. Quite a big project. Maybe just properties for now?

Thanks Nico, I’ve been lurking here for some time :wink:

Just to give you some insights into the beast, there is a small DirectLOGIC 205 PLC which handles all the commands sent to the BDPathway from a single serial connection, so there isn’t a clear split between what the microscopy part is and what the liquid dispensing part is.

Inside the BDPathway 435:

Easy enough, I wrote a BDPathway hub device, to which I attached a bunch of state devices (like the door), wheel devices and shutters, xy and z stages.

Regarding what a “liquid handling device” would look like, It would be nice if it had some specific methods which could be called as part of the MDA scripting (or from Python for those so inclined).

Top of my head:

  • Set the plates and tips SBS dimensions / formats / number of wells and possibly keep track of liquid levels.
  • Set the number of tips available and maybe automatically move to the next tip / tipbox when depleted (the 855 does this automatically).
  • Query the current nest / well position (from where the current stage coordinates for example).
  • Add a tip, aspirate, dispense, mix, eject the tip in the bin or back into a nest / position (and make sure there isn’t a tip there already).
  • Plus a bunch of properties like aspirate dispense speeds (number of mixes?).

I’m trying to think this in terms of the two I know (the BDPathway 855 and the Cellomics Arrayscan) but I don’t know how these methods would translate to other machines. And unfortunately, I don’t have access to these two anymore…

Soldiering on :slight_smile:

Cheers,
Egor

Before I can release my adapter, I need to make the serial communication with Micro-manager a lot more robust. Not sure if it is the current USB-serial dongle I am using, but I see many flipped bits (even at 19200 baud), which make my life… exciting (I don’t get out much, COVID and all).

2020-07-27T09:27:27.534247 tid8148 [dbg,dev:COM4] SetCommand -> gtZ\r
2020-07-27T09:27:27.585246 tid8148 [dbg,dev:COM4] GetAnswer <- gtJ,-1\r

I’m querying the Z step size with ‘gtZ’ but the PLC tells me the ‘gtJ’ command is not valid? Meanwhile my adapter happily splits the response string at the comma and registers -1 as the step size. Nope, not good enough.

Let’s dig in a bit:

print(bin(ord('Z')))
0b1011010
print(bin(ord('J')))
0b1001010

Oh, that’s rather unfortunate! :confounded:

EDIT: My somewhat robustified “ExecuteCommand” now picks-up command mismatches and sends these kind of strings to the log:

2020-07-27T15:22:48.847586 tid12624 [dbg,dev:COM4] SetCommand -> tY\r
2020-07-27T15:22:48.898587 tid12624 [dbg,dev:COM4] GetAnswer <- 4Y,-1\r
2020-07-27T15:22:48.898587 tid12624 [dbg,dev:BDPathway-XYStage] ExecuteCommand-SerialMismatch: sent 'tY' received '4Y,-1'
2020-07-27T15:22:48.898587 tid12624 [dbg,dev:COM4] SetCommand -> tY\r
2020-07-27T15:22:48.975585 tid12624 [dbg,dev:COM4] GetAnswer <- tY330000\r
...
2020-07-27T15:23:41.357704 tid6008 [dbg,dev:COM4] SetCommand -> TaX550000,Y330000\r
2020-07-27T15:23:41.544703 tid6008 [dbg,dev:COM4] GetAnswer <- TaX550000,Y330\x1000,-1\r
2020-07-27T15:23:41.544703 tid6008 [dbg,dev:BDPathway-XYStage] ExecuteCommand- SerialMismatch: sent 'TaX550000,Y330000' received 'TaX550000,Y33000,-1'
2020-07-27T15:23:41.544703 tid6008 [dbg,dev:COM4] SetCommand -> TaX550000,Y330000\r
2020-07-27T15:23:41.659216 tid6008 [dbg,dev:COM4] GetAnswer <- TaH550000,Y330000,-1\r
2020-07-27T15:23:41.659216 tid6008 [dbg,dev:BDPathway-XYStage] ExecuteCommand-SerialMismatch: sent 'TaX550000,Y330000' received 'TaH550000,Y330000,-1'
2020-07-27T15:23:41.659216 tid6008 [dbg,dev:COM4] SetCommand -> TaX550000,Y330000\r
2020-07-27T15:23:42.132217 tid6008 [dbg,dev:COM4] GetAnswer <- TaX550000,Y330000\r

It tries 3 times before giving up or until the reply matches the command sent to the PLC. But! if the command is corrupted just right, it will still be accepted as valid. Worse case scenario, we could crash the lens into our plate or move the stage to the wrong position without it being obvious, or switch to the wrong position on the filterwheels, or all sorts of fun I am not thinking about right now.

I only have PL2303 based USB dongles but I’ll check a different one next time I’m in the lab and report back.

A quick update, here’s where I’m at, using

  • my BDPathway adapter
  • a webcam with OpenCVgrabber
  • a 24 well plate in the HCS site generator plugin with the two top rows selected.
  • MDA with the generated map.

stage

TODO: I really need to make sure the lens is completely stopped before acquiring my images. And of course the autofocus which I haven’t looked at at all for now.

Still, it’s progress :slight_smile:

Some welcome progress on the dongle front!

It turns out that my particular USB to serial dongle, based on a Prolific PL2303 chip (USB\VID_067B&PID_2303\6&217C136&0&4), wasn’t reliable at all. I have since swapped it for a FT232R based dongle (FTDIBUS\VID_0403+PID_6001+A800CC7GA\0000) and as far as I can tell, all (?) my corruption issues are now gone.

Combine this with receiving empty “” responses when querying the (x, y or z) position while the stage is moving, and I now have a reliable way to tell whether the stage is busy, and wait for the stage to stop before issuing further commands. MDA Acquisition seems to work well enough with the (mechanical) shutter, stage movements and Excitation/Dichroic/Emission filterwheel commands.

Also, it turns out I didn’t have to do anything special to make my BDPathway adapter compatible with MM2.0gamma. Sweet!!

Next I will try to make AttoVision not bork on the old XP machine and look at how the 6221 PCI card controls the autofocus.

Wonderful Egor! Please do keep us updated on your progress here. I enjoy reading about it!

Just added your device adapter code to the 1.4 repo. It should soon make it into 2.0. If there is anyone out there who has access to one of these, you can now start playing with it (but be careful, and don’t blame Egor or me when anything breaks).

I got round my first issue which was: How do I run AttoImage?? We may have lost our original config file, or at least the config file we are trying to launch AttoImage with (on the original XP computer) isn’t quite compatible with our BD Pathway 435. I tried to contact BD, but I doubt I will ever get a reply.

AttoImage (which controls the BDPathway) first opens a serial connection and sends the rd command to query what type of Pathway it’s dealing with. I remember our BDPathway 855 in Manchester was replying with “rdAtto_AAAS”, which interestingly is the answer AttoImage is expecting with the default (?) config file I am using. Instead, our BDPathway sends back “rdAtto_BD_EP”. Was that modified for our lab? I don’t know. However, AttoImage really doesn’t like this answer and shuts down soon after telling us about the “issue”.

My workaround is to put an “Arduino” microcontroller in the middle, which can analyse and modify any string sent back from the BDPathway to the listening PC. The idea of course is to swap the response containing “BD_EP” with one containing “AAAS” instead.

Since I didn’t really want to install any new device driver on the XP machine, I chose a microcontroller with (at least) two hardware serial ports, the STM32F103 blue pill. Also, it has plenty of oomph compared to an 8 bit ATmega, can be programmed with the Arduino IDE if one so wishes, and is very cheap.

I briefly considered stealing power from the RS232 port on the XP machine, but then I thought… Hey, the USB port delivers power but can only expose a serial connection, so why not use it?

Introducing… TCOD for “two controllers, one device”.

For now, TCOD has two different working modes which you can switch by issuing the `D command (for debug) from the USB side.

In normal mode, the blue pill redirects commands from either controllers to the device and sends the device’s answers to both controllers. Since the responses could come out of order, I wouldn’t advice trying to control a (any?) microscope that way, but who knows, as an acid test maybe? This could be very interesting! :thinking:

In debug mode, you simply spy on the traffic between the XP machine (the other controller) and the BDPathway (the device) using any terminal software, like KiTTY or PuTTY.

kitty_debug where “>” denotes PC to device, and “<” device to PC.

Copy-paste into a text file and you have a nice set of commands and responses to analyse in Jupyter:

That’s all! I’ll continue to play with my setup until I break something I guess :slight_smile:

For TCOD, I’m going to buy a nicer enclosure and some isolated RS232 <-> TTL modules because I’m not feeling confident about 3 mains powered machines connected to the same ground.

Also if I could do anything about it, I would have the blue pill expose two USB serial devices, one a “normal” serial communication device, and the other outputting debug info all the time.

If there is any interest in reverse engineering microscopes and writing device adapters for them, I’m happy to share my code and start a page on the micro-manager wiki :slight_smile:

It’s been some time since my last update and I’ve been dreading this next part. I’ve had to wait until I had enough info to try and tackle the BD Pathway’s integrated laser autofocus device. So, here we go :slight_smile:

Controlling the laser autofocus (BD Pathway 435)

Physical examination

On the BDPathway 435, the Laser autofocus is controlled by both a serial command sequence and some calculations done within AttoVision, from an analog readout and communication signals gathered on a NI PCI-6221 card (with a 37 pin connector).

Looking at the cable linking the BDPathway and the control PC, the following pins are connected on the DB37:
imageimage

Name Pin Colour
P0.0 37 green
P0.1 19 brown
P1.1 32 grey
P1.2 33 yellow
P1.3 15 red
P1.5 35 white
P1.6 17 blue
AI.0 1 black
AI.8 20 orange
GND 14 purple

Looking at the live panels in the NI Measurement and Automation Explorer software (test panels tab), we know that Analog In is a differential signal (AI0 +, AI8 -) and its full range is -10V to 10V. We also have P0.0 and P0.1 possibly setup as inputs and/or Open Collector (?), and something happening on the P1 port, bits 1,2,3,5,6:

Other values of interest in AttoVision can be found in Autofocus Setup → Setup plate Type include sampling rate (12KHz), Z sweep relative displacement (160um), gain (? 20), use of a rolling average (20 values) and Algorithm type (one of Double threshold (default), DownTrend, Shoulder Peak, Simple Threshold):


Serial strings:

Two things we can check. In the “Laser Auto-Focus offset calibration” panel:


what happens when we click “Find Laser Z position”? (‘S>’ serial controller (AttoVision) sends, ‘D>’ device responds)

S> TrZ16000
D> TrZ16000
S> tZ
D> tZ70794
S> TaZ-129206
D> TaZ-129206
S> TaZ13894
D> TaZ13894
S> tZ
D> tZ13892

And Laser autofocus position is reported at 138.920 um.

Here, TrZ16000 instructs the BDPathway to initiate a -160 to +160 um sweep from the current position.

TODO I will come back to this sequence later.

Then what happens when we click the “Test” (autofocus) button (with -29.34 um offset at the time of the test)?

S> TrZ16000
D> TrZ16000
S> tZ
D> tZ29894
S> TaZ-170106
D> TaZ-170106
S> TaZ-37340
D> TaZ-37340
S> P00
D> P00
S> MT,1
D> MT,1

TODO I will come back to this sequence. I will note that the transmitted light isn’t switched on with MT,2 but with P00 for some reason.

Switch on:
S> P00
D> P00

Switch off:
S> MT,1
D> MT,1

How does it work? (wild speculation ahead)

The laser autofocus readout is probably based on some kind of photodiode array like in this paper: High-speed and precision auto-focusing system for direct laser lithography from which I’ve pinched this figure:
image
and we’re looking for either a zero crossing (or a max value as described below) readout on AI.0/AI.8

In our case, “Material surface” is the bottom of our multi-well plate.

I am guessing that the serial command initiates a Z sweep and the digital lines are used as a clock signal and maybe some start stop signals. If these are sampled together with the analog signal, maybe that’s how we find how many Z steps there are between the current position and the focal plane.

Can we test any of this?

I just got a small circuit board manufactured, which should allow me to probe the digital signals with sigrok and a super cheap logic analysers (the little 24MHz / 8 channel USB ones).


I’m still waiting for the proper jumpers and headers I was planning on using here but this will do for a quick test.

Future plan(s)

What I really need is a mixed signal analyser to probe the analog and digital signals at the same time. In my head (ha!) I can replace the NI card with an “Arduino” (possibly an STM32F411 black pill) to sample the analog and digital signals and return a serial string telling micromanager the relative number of steps required to reach the focal plane.

Hopefully something along these lines will happen:

  • A Z sweep is requested by Micro-Manager via a TrZ instruction.
  • This activates some sort of signal which triggers the Arduino to sample the analog signal and whichever signals of interest.
  • Z-offset calculations are carried out on the sampled analog signal (rolling average, find max or find zero crossing) and a relative Z value is stored in the STM32F411’s RAM.
  • Micro-Manager can request that offset via a serial command to the black pill.

That’s it for now. Lots of things I’ve possibly missed, assumptions I should not have made, and maybe this adventure has / will shortly reach a dead-end. Whichever way, I will edit this post when I hook it all up and hopefully have more data to show! And if you have any comments, feel free to share them here.

Edited to reflect the latest development with the DT9802 DAQ card.

My dongle sits between the PCI6221 card and cable from the BDPathway and AttoVision doesn’t seem bothered. This will be great for testing.

I now use a Data translation DT9802 USB DAQ card to do my analog, digital and Y-Pipe timestamps combined acquisition:


Several reasons for this: The inputs are isolated, the analog inputs can be used in differential mode, and there is already a micro-manager adapter for this card (but… Windows only).

Going forward, I might be able to read the autofocus signals using either the original PCI6221 DAQ, or the DT9802 DAQ, or if necessary, any other cheap/cross-platform DAQ card. These choices (whichever available) will be given to the user as dangling of the BDPathway device adapter hub in the hardware configuration wizard. The idea is that even if the autofocus isn’t available, the rest of the BDPathway hardware will still be controllable from micro-manager.

For testing, I’m using the free to use QuickDAQ software, with 10s long acquisitions at 10KHz, which I export as CSV files.

With some Python magic, I can combine a CSV file with the serial traffic and timestamps output by the Y-Pipe, into a single graph that shows (hopefully) exactly what chatter is going on between the BDPathway, the AttoVision serial traffic and PCI6221 DAQ inputs.

.
Let’s call this (one of) our baseline(s) chatter and see if we can reproduce it outside AttoVision.

What I’m aiming for: With autofocus on, any absolute Z height will be converted into a distance to the plate bottom (measured) + an offset to bottom of the well (user defined, depends on the plate type?) + a distance into the sample. Then when the autofocus is called subsequently, this distance into the sample can be converted back into an autofocus corrected absolute height.

Unfortunately, the graph shows serial commands issued in response to the various BDPathway TTL outputs (see edit 2 below where we determined the signal directions), which means that the autofocus may not be controlled by a standalone microcontroller like I had planned.

Edits

EDIT 1 - Analog signal (wrongly) measured with a PicoScope

I’ll just leave this here for now. I borrowed my colleague’s PicoScope and here’s what the probe connected between AI.0 (+) and AI.8 (-) actually sees:


In Sigrok / PulseView and mixed signal mode, hopefully I’ll have the analog and the digital traces on the same graph.

EDIT 2 - I used diodes to confirm signal directions

Quick update, I made some special jumpers with a diode across and am recording my signals at the output of the diodes. This way, if a jumper is set the wrong way round (i.e. PC → BDPathway instead of BDPathway → PC), I do not get any output recorded in PulseView.

A bit risky for sure, but we had already established that the NI software thinks all the signals are inputs into the PCI6221 card, this just confirms it.

No smoke, no nasty surprises. Signal directions confirmed! :slight_smile:

EDIT 3 - Y-Pipe serial events monitored in PulseView

This is quite an important upgrade to the Y-Pipe code, a device which I’ve talked about here on the forum.

Long story short (ha!), I was watching one of EEVBlog Dave’s videos where he talks about mixed signals oscilloscopes (MSOs). The take home message is: You don’t really need an MSO if you already have an oscilloscope and a logic analyser and can trigger both to record at the same time.

Which got me thinking, I can definitely adapt that idea to correlate specific digital events recorded with PulseView, with the serial commands that may have triggered those events.

An easy way to add this capability to the Y-Pipe is to generate pulses upon reception of the delimiter character(s), for a particular branch of the Y-Pipe. Then, simply send those pulses to the logic analyser and visualise them in PulseView with the rest of the digital signals.

On the Y-Pipe, each of Serial Device, Serial Controller, USB Controller and Console comment now have their own 5V tolerant digital output (PB12, PB13, PB14, PB15).


The pulse is set low at the beginning of the main loop and high by the delimiter. The wait period at the end of the main loop more or less determines the pulse width (about 20ms). The raising edge of a pulse is the important part and indicates the moment at which the string was sent or received.

Here I’ve overlaid the serial output of the Y-Pipe with the delim pulses for the Serial Controller and Serial Device:

I’m really happy with this new development which brings me one (small) step closer to understanding how the laser autofocus is addressed. And while waiting for my LHT00SU1, I’ll see if I can trigger the picoscope to start recording from the falling edge on P1.2 injected into Channel B. Thanks Dave! :wink:

EDIT 4 - Another day another DAQ, first proper acquisition Well well well, I must've messed my input mode (AC vs DC) on the picoscope, so I updated the main section to show the signals measured by the DT9802 DAQ card instead.

References

First trials, click to expand (moved from the main section)

Here you can see it all hooked-up to a logic analyzer, reading signals from pins I had previously traced inside the DB37 connector.

And this is the very first Sigrok output I got, just after I pressed the “test autofocus” button in AttoVision:

The serial chatter was a bit busier than expected, but that’s because I had just launched AttoVision:

Serial chatter, click to expand
S> TrZ16000
D> TrZ16000
S> tZ
D> tZ-8534
S> TaZ-208534
D> TaZ-208534
S> TaZ-34768
D> TaZ-34768
S> gdA
D> gdA,1
S> Sp00,A,0
D> Sp00,A,0
S> gdC
D> gdC,-1
S> gdc
D> gdc,1
S> Sp00,c,1
D> Sp00,c,1
S> gda
D> gda,1
S> Sp00,a,1
D> Sp00,a,1
S> gdD
D> gdD,1
S> Sp00,D,1
D> Sp00,D,1
S> gdO
D> gdO,-1
S> Sp00,E,1
D> Sp00,E,1
S> Sp00,T,2
D> Sp00,T,2
S> Sp00,G,1
D> Sp00,G,1
S> P00
D> P00
S> MT,1
D> MT,1

If we zoom in on the P1.6 chatter, this is getting super interesting:


We have a 500Hz clock signal and stuff happening with P0.0 and P0.1

I’m not exactly sure what the deal is with P1.3 and P1.5, maybe that’s just noise. And to actually check whether P0.0 and P0.1 are inputs, I’ll make some Dupont jumpers with a diode across and see which way they make AttoVision cry.

Future stuff: To really understand how the Autofocus works, I need Sigrok traces with the analog signal added to the mix, especially the correlation with the 500Hz clock. I don’t have a mixed signal analyser, so I guess I’ll just have to buy the cheapest one I can find on Aliexpress, which also appears to work with Sigrok (LHT00SU1). There may be an issue with AI.0 and AI.8 being differential signals, which I don’t think is a mode compatible with the analyser, but I think I can get away with some sort of differential attenuator I was reading on, which will convert the +10,-10V input to a 0,5V single-ended output:


(from a TI article, simulated in LTSpice with a LT1994 fully differential opamp). If I am to build an Arduino replacement for the NI card, I will also need such a circuit to feed the analog input to the ADC.

That’s it for now! Back to Sigrok and days / weeks of data analysis and waiting for the mailman :wink:

Bonus random photos, click to expand


A rather naked dongle…


Two daisy-chained dongles (oh my!)