Paquo - read/write QuPath projects from Python

Hello everyone,

I would like to introduce you to paquo (PAthological QUpath Obsession) — a Python library that let’s you read and write QuPath projects from Python :snake:

With @sdvillal having done all the groundwork to create paquo, for the past weeks him and I have been polishing the code and documentation to bring paquo into the hands of the community.

paquo was created to allow Python developers to interact with the phenomenal QuPath without having to learn Java or Groovy.

As of now paquo is available for all operating systems for Python versions 3.6, 3.7, and 3.8. paquo is a python wrapper for some of the functionality of QuPath, requiring an installed version of QuPath to operate correctly. It’s using JPype to provide pythonic interfaces to QuPath’s Java internals.
Development takes place on github at: https://github.com/bayer-science-for-a-better-life/paquo
And its documentation is available at: https://paquo.readthedocs.io/

paquo has not yet reached its first stable version release. But it’s already usable and we’d be happy to get feedback from the community :heart: Also please be aware that paquo is under heavy development, and we will be adding more features and potentially break things as we move towards v1.0.0.

We’d love to hear from you and hope that paquo might be useful :blush:

Cheers,
Andreas :smiley:

13 Likes

Sounds great! :smiley: :partying_face:

3 Likes

@poehlmann @sdvillal Thanks a lot for making my last few weeks of work obsolete! :slight_smile:

As the sole developer in my lab with significant Java experience, I worked on an creating an export/import pipeline (groovy export script, python import script) and a script development workflow (see my recent post on scripting through eclipse and https://github.com/arjvik/QuPath-Scripting-Tools). I had always wondered if something like paquo was possible, but I never researched tools to connect between the JVM and CPython interpreter. I did play around with Jython a bit, but that didn’t fit our lab’s needs.

Now that you guys have made paquo, we will no longer need any of that to read in QuPath objects from Python. I know as a fact that this will significantly reduce the development time taken to interface with QuPath in many labs, not just ours. I also love how simple and pythonic you have made the API!

I haven’t had a chance to read your code, so I have a few questions:

  1. Is it true that paquo doesn’t connect to an existing QuPath instance, but rather starts a JVM with a headless version? That’s the impression I get so far.
  2. Is the public-facing API of paquo a wrapper around the java proxy returned by JPype, the java proxy itself, or something else? I’m assuming the java proxy will use python’s magic/dunder methods and thus not have any type information, so it will lack discoverability in IDEs. Does paquo's public API have such type information available?
  3. What features do you plan to implement in the future, and where would you appreciate contributions? I have enough java and python experience to help out, and I would love to contribute to this great piece of software.
  4. I noticed you mention my QuPath-Scripting-Tools project in a Github Issue (https://github.com/bayer-science-for-a-better-life/paquo/issues/7). I would love to help integrate paquo with my tools if possible. I deliberately made that project modular, so in theory writing a client for my RemoteCodeServer.groovy would be relatively easy. What is paquo's usecase for being able to script an existing QuPath instance?
3 Likes

Thank you for your kind words @Arjun.Vikram, they make us happy.

1. and 4. Partially true. Because of how JPype works, paquo needs to control how the JVM is started. But that does not mean that QuPath needs to be headless. In linux, one can start the QuPath GUI from python and easily control it interactively (e.g. from a notebook). We did not have success yet trying this in macos, where it is technically challenging. We have not tried in windows. It is in the context of interactive control being difficult in certain platforms that a client/server architecture, using for example your server, might become handy.

2. You can access both, the low level “proxy” objects and the high-level python wrapper. The latter allows the full IDE experience, and thanks to Andreas relentless work, it even has extensive type annotations to help IDEs and other tools help us better. At the same time, JPype allows to dynamically discover class members, so the experience of interactively using the low level proxys from, let’s say, jupyter, is actually very pleasant. Also JPype is improving at an incredible pace and offers some features we have not used yet. This nice exchange with the JPype folks further elaborates these topics.

3. We do appreciate very much contributions! :slightly_smiling_face:. Our bandwidth is actually quite limited, but there are several directions in which the project can go, and we will prioritize them with feedback from the users. Just to give an example of a dream, if we could put together the interactive bit we could let our anti-java data scientists easily build simple applications controlling how final users see a project upon opening (e.g. open many viewports with some images from the same project, point them to specific areas, turn on certain heatmaps…). Interactions would be, of course, limited by the fact that QuPath cannot talk to python land. This is the price one needs to pay for this multi-language debt :wink:.

5 Likes

Thank you so much @Arjun.Vikram

I’ll quickly add to 2. and 3.

  1. When more of the wrapper features land in JPype (as mentioned by @sdvillal see the linked discussion) it should be possible to thin out many of the paquo wrapper classes and still provide a clean pythonic interface that’s not leaking anything from the java world. And less code is almost always better code. (Unless of course it makes it dramatically more complicated to understand.)

  2. I think right now the most helpful thing to do is thoroughly play with paquo and report bugs, improve the online documentation, and open issues for new feature requests. With that being said, to prevent feature creep I would want to polish the existing functionality until it’s solid and then plan further. PRs with type annotations and PRs with examples for how to do things in paquo are always welcome. But please feel free to open new issues for specific discussions on the issue tracker :sparkling_heart:

Cheers,
Andreas :smiley:

1 Like

I recently wrote a bit on QuPath version numbers here, and how we’re trying to make these a bit stricter and easier to interpret.

Since QuPath itself is changing (and is some way off a v1.0.0 itself), perhaps paquo could track the same major/minor numbering so that it was easy to see which versions should be compatible?

For example, QuPath v0.2.x and paquo v0.2.x should work together. Patch versions could be updated a bit more freely on both sides, since (at least for QuPath) this shouldn’t involve API changes.

This is more or less what JavaFX and ControlsFX seem to be doing these days, with version numbers relating to the minimum JDK version with which they are compatible.

Is that already the plan, or if not what do you think?

3 Likes

Hi @petebankhead

In my mind keeping the versions in sync will be tough if paquo sticks to semver and is still catching up with wrapping more QuPath functionality.
I imagine having a CI job that tests the entire paquo test suite against every released QuPath version to determine a compatible version range (https://github.com/bayer-science-for-a-better-life/paquo/issues/19). And then paquo would throw a big error if the QuPath version used is incompatible.

But my opinion on this is not strong at all and I definitely see the user benefit of: paquo-0.2.X needs QuPath-0.2.X :smiley:

1 Like

We also discussed this early on and Andreas was fairly convincing on not following qupath on versioning:

Fair enough, I can see the arguments for both sides. The main thing that has changed since the discussion is the commitment from the QuPath side to have more standardized versions.

I suspect that following semver independently, paquo’s version numbers will quickly surpass QuPath’s – which may have the slightly confusing result that paquo has a stable v1.0.0 while QuPath doesn’t. Might not really matter though.

1 Like

@poehlmann @sdvillal One thing I’m confused about is why you need to be able to run Groovy code in order to control the QuPath window. Why can’t you just call the static methods in QPEx through JPype? Is it a limitation of JPype that prevents you from being able to run static methods?

As far as I understand (@petebankhead please correct me if I’m wrong) , QuPath is architectured such that you can obtain references to all important objects through the static methods in classes like QPEx, so you should in theory be able to control the entire QuPath interface through JPype.

Also, you mentioned that it is easier to control QuPath from python on Linux than on MacOS. Why is that? Does it have to do with MacOS’s application sandboxing?

If I can get a clearer understanding of why you need to use the scripting interface, I would love to help you guys implement the features you are hoping to get out of them!

2 Likes

The issues in macOS are well described here. Essentially, macOS constrains the event loop of GUI applications to run in the main thread of the process, which means the python process itself gets blocked. Even running python code in separated threads we have not been able to interact with the QuPath GUI.

This is in contrast to linux, where one can simply run the QuPathApp from a different thread, get the running QuPath objects (e.g. viewer = QPEx.getCurrentViewer()) and manipulate them (e.g. viewer.centerImage()) , with the QuPath GUI responding smoothly.

4 Likes

@sdvillal Hmm. so the issue is with how Java implements it’s UI on MacOS. I suppose the only option, then, is to open a side channel to communicate between the Java and Python processes, much like my RemoteCode{Server,Client} scripts. It may be better to rewrite it as a plugin that starts a new thread, so that QuPath isn’t blocked from running when the script is executed.

The interesting thing about my solution is that I don’t use any of QuPath’s internal code (no reference to it’s script-running mechanism), I simply start my OWN embedded script interpreter and pass it input from the TCP socket.

What can I do to help this integration?

1 Like

@Arjun.Vikram I guess we would start by writing a python client and building tooling around it. For the integration to adhere to paquo philosophy, we would need to write a pythonic module on top of the client. A possible approach would be to create drop-in classes that route calls to the server, performing any necessary python -> groovy conversion under the hood. It does not sound too complex if we keep modest goals and build incrementally.

We also would need to tackle issues like security. We probably do not want to have a groovy interpreter open to anyone.

I would propose to move this conversation to an issue in the relevant repositories.

1 Like

I had a look at this, unsuccessfully until discovering %gui osx.

Combining this with JPype’s documentation on launching AWT/Swing applications here, and QuPathGUI’s launch method written to be called from Swing, I can finally launch QuPath from a Juypter notebook and interact with it on a Mac using something like this:

from paquo._config import settings, to_kwargs
from paquo.jpype_backend import start_jvm, JClass

%gui osx

args = to_kwargs(settings)
qupath_version = start_jvm(finder_kwargs=args)
print(qupath_version)

import jpype
import jpype.imports
import java
import javax
from javax.swing import *

qupathGUI = JClass('qupath.lib.gui.QuPathGUI')

@jpype.JImplements(java.lang.Runnable)
class launchQuPath:
    @jpype.JOverride
    def run(self):
        qupathGUI.launchQuPath()
        
javax.swing.SwingUtilities.invokeLater(launchQuPath())

The following then works as expected:

qupath = qupathGUI.getInstance()
print(qupath)
print(qupath.getProject())
qupath.getViewer().zoomIn(10)

I haven’t pushed it terribly far, and guess there might be other threading issues emerge – but it seems fairly promising.

Along the way, I’ve made some changes to the launch method in QuPath so that this can be made easier in v0.3.0 (i.e. you wouldn’t need all the nasty SwingUtilities overhead within Python) – but for now the method above seems to work with QuPath v0.2.2.

More or less… but QPEx isn’t terribly elegant, and is really intended to support scripts run via QuPath’s script editor or command line. Many things will work anyway, but internally QPEx is storing a current ImageData object per-thread – and so won’t automatically know to use the image currently open in the viewer.

For that reason, you should probably use QuPathGUI.getInstance() as your starting point, via which you can request any of the viewers, the project etc.

Still, if you want to use QPEx interactively from Python – and remember to update the current image when needed – the following works for me, thanks to paquo :slight_smile:

qp = JClass('qupath.lib.gui.scripting.QPEx')
qp.setBatchProjectAndImage(qupath.getProject(), qupath.getImageData())

pathObject = qp.getSelectedObject()
pathObject.setPathClass(qp.getPathClass('Stroma'))
qp.fireHierarchyUpdate()

… potentially making it quite possible to use Python-based machine learning libraries to classify QuPath objects (and I suspect much more).

4 Likes

Spoke too soon… it seems all keypresses get intercepted using that approach (well, they end up going to the browser where the notebook is).

However, I’ve made a pull request to change the non-standard QuPath launch method:

And it would appear that (for some reason) %gui qt is rather more helpful.

With that pull request, I can launch QuPath much more easily from Jupyter on a Mac using:

from paquo._config import settings, to_kwargs
from paquo.jpype_backend import start_jvm, JClass

# Just while testing...
args = to_kwargs(settings)
args['qupath_search_dirs'] = ['/path/to/build/dist']
args['qupath_prefer_conda'] = False

qupath_version = start_jvm(finder_kwargs=args)
print(qupath_version)

qupathGUI = JClass('qupath.lib.gui.QuPathGUI')

%gui qt
qupathGUI.launchQuPath()

This has the added benefit of the keyboard working, and allowing me to close and reopen QuPath as needed.

If anyone has time to test it, please let me know if the new launch method is helpful for Paquo and/or causes any trouble.

Since the original static launch method was undocumented, isn’t called anywhere else, and the changes don’t substantially change the behavior, I think it could legitimately be considered a bug fix and included in a v0.2.3 release this week… rather than the (slightly more distant) v0.3.

What do you think?

Edit: Just tested it, and this method seems to work unchanged on Windows as well.

1 Like

Hi @petebankhead

This is great :sparkling_heart: Thank you so much for looking into this. I’ll have some time to verify the changes on all OSes on the weekend. I’ll report back on the PR when I’m done testing.

And regarding the API change: I wouldn’t worry and bump the patch version. Since qupath is at major 0, semver is basically non-restrictive https://semver.org/#spec-item-4

Cheers,
Andreas :smiley:

2 Likes

Thanks @petebankhead, this is really great.

I have tried (on the way I have updated our qupath conda packages). Here are some notes for MACOS:

  • I’am now trying using ipython (not notebooks) as a goal would be to control the GUI from scripts.

  • Interaction with the application is nice and responsive.

  • The “gui qt” magic seems to work best (“matplotlib gui” also helps with an ipython terminal, but not “gui osx”, that still gets us blocked).
    It would be useful to understand what these magics do and whether they can be used independently of having an ipython kernel or REPL running.

  • The open qupath instance does not integrate with the menubar and does not appear in the dock (minor usability glitches that likely cannot be workarounded).

  • I have not been able to make this work from an (.ipy) script run using ipython (but I have not tried a lot).

I do think this is useful, so we should start building a “paquo.gui” module and a demo as soon as we have some time.

2 Likes

Thanks @sdvillal, glad it looks promising!

Forgot to mention this. There are a few things:

  • you can toggle the system menubar on/off in the QuPath preferences
  • if this is turned on, JavaFX might still not be able to use the system menubar depending upon how it is launched
  • system menubar confusion can also cause ‘single key’ shortcut confusion on macOS (e.g. ‘a’ to toggle annotation visibility)… specifically, these events can be fired twice in quick succession.

The only way I could find to reliably fix the shortcut issue is to turn off the system menubar preference. This may need to be done before QuPath is launched (the preference is persistent, so can be set in any running QuPath instance).

(Note that toggling the system menubar repeatedly on macOS can cause the shortcut problem even when launched normally – somehow behavior differs if the preference is set before launch or after.)

Not sure, but this might be the relevant code.

In QuPath v0.2.3 you can use QuPathGUI.launchQuPath(None, Boolean), where the boolean indicates that the app should be launched as if it is an AWT/Swing app.

It might be worth trying both true and false, since I see different blocking/failing behavior on macOS depending upon precisely the thread from which I’m calling the method. See javadocs for Platform.startup for a little more information.

Great! One thing to look out for is whether QuPath can be launched, closed and relaunched. This caused me some trouble (i.e. I thought I had a working solution, but couldn’t restart QuPath later…). Platform.setImplicitExit(boolean) is probably the key to resolving such problems, if they come up.

1 Like