To everyone interested in software control of microscopes

Creative new microscopes most often need dedicated software for operation. Creating these software solutions is laborious, and many of us recognize that we can all help each other, facilitate re-use and avoid duplication. Raghav Chhetri, Stephan Preibisch and I organized a workshop about this topic that took place in February 2020 at the HHMI Janelia Research Campus. Workshop participants showed a great desire to build a community (much larger and more inclusive than the Workshop participants) to work together towards modular tool-sets for microscope control that can be used from multiple computer programming languages and platforms.

We summarized some of the ideas that arose during the workshop in a white paper available on arXiv (https://arxiv.org/abs/2005.00082), and started a temporary github repository (https://github.com/nicost/uScopeControl) to provide some means for communication and collaboration. We are partnering with Kevin Eliceiri (LOCI, Univ. Wisconsin, Madison) towards creating a website that can act as an umbrella hub for all open source projects concerned with software control of microscopes, including the Micro-Manager project.

Looking forward to seeing your input/suggestions/encouragement/criticism here. Using a github account, you can also respond to https://github.com/nicost/uScopeControl/issues/1 to make it easier to update you of future developments.

28 Likes

Awesome idea Nico (and Stephan), this will be/is much appreciated…

OpenSPIM should definitely be included, I’m sure that PAvel would approve.

4 Likes

Might also be worth a post on the microforum, though I’m guessing most of those people are on both forums. Bit less activity there, though, so less likely to get lost in the shuffle.

1 Like

Hi @nicost. Thank you for writing this up.

I will be talking to @StephanPreibisch and Raghav Chhetri shortly on the topic and its interface with image analysis. I will be joining them at HHMI Janelia this summer to start working on a part of this. While control, acquisition, and analysis have usually come in sequence, there is now a need to integrate all of these steps. MMCore will be an important common library for this purpose.

Certainly, there is a polyglot issue at hand here with a diversity of programming languages with each having a unique role to play as you outline. Java has both an extensive history in this space as well as speed. Python has beautiful expressive syntax and wide array of numerical processing packages. C++ remains the stalwart of native hardware interface.

Besides Java, Python, and C++, I also note the rise of Julia which fills a niche between Python and C++, which is briefly mentioned. The syntax borrows from some common high-level scripting languages like Python, R, and MATLAB but also has static and inferred typing. It takes advantage of a LLVM backend that can be both statically compiled and also dynamically (Just-In-Time) compiled like Java, except to native machine code rather than JVM bytecode. This also brings the advantage of being able to easily integrate with native C/C++ code. I note that @timholy is a Julia steward, and his lab has pioneered microscopy related Julia packages including control software:
https://github.com/HolyLab (in particular see Imagine and Imagine.jl)

I look forward to helping to meet the challenge of next-generation microscopy and its community.

5 Likes

Good idea… Hardware is also important… consumer-level products should be covered meanwhile…
:+1:

1 Like

Hi @nicost,

I am very excited to hear about the further development of micromanager. It’s really a vital lab resource for us. We use it every day on multiple microscopes so it’s wonderful to hear there are plans to expand access to MMCore in python. I hope further integration with the scijava framework can also be in the plans.

1 Like

Since @marktsuchida now works right next to the principal architect of SciJava (@ctrueden), I have little doubt that further integration will happen;) As always, we’ll have to set priorities because of limited time, so both your input into priorities, as well as help building parts will be essential.

1 Like

This is a topic close to our heart as well. We have been working on both low level hardware control and a microscope GUI both in Python. We are currently writing them up for publication but they are open source and available on github.

python-microscope:

and microscope-cockpit

We have also had collaboration with the PYME developers and integrated some of the hardware control so we have the beginnings of a unified API for python control of microscope hardware.

3 Likes

That sounds great! Is there somewhat of a description of these beginnings of a Python API for microscope hardware control?

It would be wonderful to learn from each other, and to work towards a language agnostic API, so that we can all benefit from the same code interfacing with hardware. Micro-Manager has something to offer here, not only because of the large number of device adapters already available, but also the experience with a device abstraction layer (nevertheless, the MM device abstraction layer needs significant updates).

One interesting question is the use/role of a device abstraction layer. MM strictly separates the “upper” API (facing the user/GUI code), and the “lower” API (interfacing with abstracted devices). It seemed that most Python developers at the meeting did not like that separation. Curious what route you are taking, and the reasoning behind it.

B.t.w., it looks like the Python 3 bindings for MM are now pip installable: https://github.com/micro-manager/pymmcore

2 Likes

@nicost thanks for keeping this going!

My impression was that a chunk of python folks at the meeting just didn’t like the idea of having access to the device drivers exclusively through MMCore. From my perspective, I’d like to see something like the pyMMCore survive, but also add a new layer where python developers could load device drivers directly without the core. @Hazen_Babcock @David-Baddely or @campagnola might have some other thoughts on this- I’ll try to wrangle them over to this forum for the discussion.

@iandobbie I’m a little embarassed to admit that we didn’t find python-microscope until recently! I’ll tag you in one of our python-data-acquisition issues as I think we could benefit from at least a look around if you’re interested.

1 Like

We have a very similar approach to the separation. We actually have 2 separate packages, python-microscope talk to hardware and publishes a standard API for different device types, eg cameras, stages, filterwheels etc… It also provides a deviceserver which runs hardware objects in separate processes to avoid the GIL. The individual hardware objects can either be called directly from your code, or communicated with via Pyro a remote procedure calling package which makes function calls network transparent, meaning you can easily spread hardware across multiple machines.

You can get python-microscope with “pip install microscope” and the current version is python3 only. The release is relatively out of date so probably better to go to the github version.

The GUI end is then handled by microscope-cockpit which provides user interaction and coordination between devices allowing experiments to be run etc… One major feature is navigation of large tile scans to allow easy sample finding on microscopes with no eyepieces. We dump the image data directly to the GPU as textures and this can be zoomed and panned in real time.

Not currently on PyPi due to name clashes. but available from github.

Yes, I wanted to be able to access the device drivers directly. My experience was that MMCore made assumptions that didn’t work with what I wanted to do. For example, and this may have changed, the Python logger that I wanted to use and the MMCore logger conflicted over some system resource. I didn’t know how to turn off the MMCore logger so that meant that when I wanted to use MMCore I couldn’t use my own logger.

I wouldn’t argue for getting rid of MMCore, I just wanted it to be a little more modular.

Splitting out logging is a very good point. Current logging is very nice in the MM environment, but clearly not nice if you have your own logging implemented already. You still will want a mechanism to get messages from the devices to your logging instance. What should that look like, what should the messages themselves look like?
Are there other examples of MMCore assumptions that are in the way?

Reading the documents on https://github.com/MicronOxford/microscope/blob/9a66b121a0f6314e405239d1ed17782b0f825dd6/doc it seems that the design is indeed quite analogous to MMCore. Is there a description (formal or otherwise) of the device types and device interfaces? It would be fun to see if the MM device to Core interface can be wedded to the device interface you are using (by automatically generating Python bindings for MM device adapters). That would greatly expand the number of devices python-microscope supports.

Also, how different/similar are python-microscope and other proposals/ideas in the Python world?

1 Like

Just realized this thread strongly overlaps with a thread like this one: https://github.com/python-data-acquisition/meta/issues/11 (as well as other issues on that repository).
Sorry everyone! Hard to even find a single place to discuss what we clearly all care about deeply!

3 Likes

Maybe, the first thing, also in reference your later posts and the goals of the workshop and the white paper, is to decide on a good forum for these discussions?

Great question!

I made a stab at listing online resources related to microscope automation, including links to several online discussions related to unification (https://github.com/nicost/uScopeControl). Those links contain interesting threads.

Personally, I find it a pity if the discussion were to be about Python-based solutions only. This forum, that targets all software developers and users working with biological imaging, seems very appropriate. If the need arises, we can add a tag to label the discussion so that multiple threads can easily be found.

It seems I’m a bit late to this thread… Thank you @nicost for opening it! I’m curious to have a closer look at some of the projects mentioned here and the associated repositories. Also at the white paper. Somehow I didn’t hear about that meeting in February 2020 – I would have loved to participate in it!

I’m very interested in software control of microscopes. In my case, it’s scanning electron microscopy and a particular acquisition mode: Serial Block-face Electron Microscopy (SBEM).

Welcome to the party @btitze! Sorry to not have invited you (and many others!) to the meeting. The overarching outcome of the meeting was the desire for collaboration, avoidance of duplication and development of easy to use tools for programmatic microscope control. I am afraid that I did not do too much myself (other than ingest lots of suggestions, help write the white paper, started working on a white paper about the MM core with Nenad, and continued MM support).

Maybe you can rekindle this thread! How do you think what a device abstraction layer should look like? What are some of the decisions you made in the past that you are happy with now, and what are decisions that you regret?

2 Likes