Displaying ImageJ and napari UI simultaneously

Last night I tried coding a script to pop up a napari viewer and the ImageJ user interface side-by-side.

Source code
"""
napari + Fiji
"""

import objc
from Foundation import *
from AppKit import *
from PyObjCTools import AppHelper

import imagej, napari

def wrap(f):
    class AppDelegate (NSObject):
        def init(self):
            self = objc.super(AppDelegate, self).init()
            if self is None:
                return None
            return self

        def runjava_(self, arg):
            f()

        def applicationDidFinishLaunching_(self, aNotification):
            self.performSelectorInBackground_withObject_("runjava:", 0)

    app = NSApplication.sharedApplication()
    delegate = AppDelegate.alloc().init()
    NSApp().setDelegate_(delegate)
    # this is necessary to have keyboard events sent to the UI;
    #   basically this call makes the script act like an OS X application,
    #   with Dock icon and everything
    NSApp.setActivationPolicy_(NSApplicationActivationPolicyRegular)
    AppHelper.runEventLoop()

def start_imagej():
    ij = imagej.init('sc.fiji:fiji:LATEST+net.imagej:imagej-legacy:0.37.0+org.scijava:script-editor:0.5.3', headless=False)
    ij.ui().showUI("swing")
    print('--> ImageJ started')

    print('--> Starting napari')
    with napari.gui_qt():
        viewer = napari.Viewer()

print('--> Starting imagej')
wrap(start_imagej)
print('--> Aaaand done')

Since I have a MacBook, I need to deal with the threading concerns of macOS. Specifically: in order to display the Fiji user interface, Java AWT must be started on the Cocoa event loop. This can be achieved using PyObjC; details here.

However, attempting to invoke the napari viewer from the Cocoa event loop thread crashes the program, with errors like:

2019-12-11 16:07:07.636 python[50393:3589960] pid(50393)/euid(503) is calling TIS/TSM in non-main thread environment, ERROR : This is NOT allowed. Please call TIS/TSM in main thread!!!
2019-12-11 16:07:08.166 python[50393:3589960] Apple AWT Internal Exception: <class 'RuntimeError'>: There is no current event loop in thread 'Dummy-1'.
2019-12-11 16:07:08.166 python[50393:3589960] *** Terminating app due to uncaught exception 'OC_PythonException', reason: '<class 'RuntimeError'>: There is no current event loop in thread 'Dummy-1'.'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff36c2dacd __exceptionPreprocess + 256
	1   libobjc.A.dylib                     0x00007fff6130aa17 objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff36c47629 -[NSException raise] + 9
	3   _objc.cpython-37m-darwin.so         0x00000001068f5a4e PyObjCErr_ToObjCWithGILState + 46
	4   _objc.cpython-37m-darwin.so         0x00000001068cae63 method_stub + 5283
	5   _objc.cpython-37m-darwin.so         0x00000001068f53e0 ffi_closure_unix64_inner + 720
	6   _objc.cpython-37m-darwin.so         0x00000001068f48f6 ffi_closure_unix64 + 70
	7   libsystem_pthread.dylib             0x00007fff62ccd2eb _pthread_body + 126
	8   libsystem_pthread.dylib             0x00007fff62cd0249 _pthread_start + 66
	9   libsystem_pthread.dylib             0x00007fff62ccc40d thread_start + 13
)
libc++abi.dylib: terminating with uncaught exception of type NSException

I can start napari from the main thread (just dedent the “Starting napari” lines one level :wink:), but then: A) the script blocks until I close the napari viewer window; and B) even after closing the window, the ImageJ UI never appears even though the code to start ImageJ does then execute.

Are there any people sufficiently expert in macOS+Python who can advise on the best way forward regarding these threading issues? Have any of you combined the QT and Cocoa/Objective-C frameworks? Is this feasible?

@skalarproduktraum suggested to me that it might not be feasible within the same process, in which case the next step would be to research ways of launching java and python in two separate processes that still share memory somehow.

Ideas for troubleshooting are very welcome.

In the meantime, since I’m rather stuck, I am going to start coding support on the Python side for executing SciJava commands. This could eventually lead to all/most ImageJ2 plugins becoming magically available in the napari UI. :crossed_fingers:

5 Likes

@jni @talley - is there anything we did in the interactive scripting PR that might help here?

1 Like

@ctrueden I had similar issues with imagey, which I used to start a native CPython IPython terminal from within ImageJ. I got it to work by importing the AWT/Swing classes only after starting the QT event loop. I am not 100% sure that this solved but it may have.

And as far as I remember, I did not even need to care about cocoa anymore.

NB: This project is more or less abandoned and I have not touched it in a long time, but maybe there is some useful stuff in there for you.

2 Likes

At the Fiji+Python hackathon in Dresden during the past couple of days, @hanslovsky and I explored various ways of achieving napari + ImageJ simultaneous GUIs on macOS.

Behavior differs between plain Python, IPython and Jupyter. We don’t have a comprehensive progress report to share yet, but here are some findings so far:

A. Starting napari+ImageJ from plain Python

$ python -c '
import napari
with napari.gui_qt():
    napari.Viewer()
'

And then when napari comes up, open the Jupyter Qt console and type:

import imagej
ij = imagej.init(headless=False)
ij.ui().showUI()

And it works because the console in napari is running in the correctly initialized Qt GUI/main thread.

However, if you then touch the Java UI from Python, e.g.:

ij.ui().showDialog('hello')

then Python locks up.

You can get around this in a couple of different ways:

  1. Use Java’s EventQueue to queue the task on the Java event dispatch thread:

    from jnius import PythonJavaClass, java_method, autoclass
    
    class JavaRunnable(PythonJavaClass):
        __javainterfaces__ = ['java/lang/Runnable']
    
        def __init__(self, f):
            super(JavaRunnable, self).__init__()
            self._f = f
    
        @java_method('()V')
        def run(self):
            self._f()
    
    EventQueue = autoclass('java.awt.EventQueue')
    EventQueue.invokeLater(JavaRunnable(lambda: ij.ui().showDialog('hello')))
    

    To make this easier, we are considering adding the EventQueue approach to the scyjava library as scyjava.awt.queue for invokeLater and scyjava.awt.invoke for invokeAndWait; see scijava/scyjava#12.

  2. Use the SciJava ScriptService:

    ij.script().run('.groovy', "#@ ImageJ ij\nij.ui().showDialog('hello')", True)
    
  3. Using ImageJ’s Script Editor.

B. Starting napari+ImageJ from IPython

Launching napari from IPython requires to use the %gui qt magic. Unfortunately, after the napari viewer appears and control returns to IPython, it does not work to subsequently run imagej.init(headless=False)—AWT hangs upon initialization. And the Qt console button is grayed out when launching napari in IPython, so a solution analogous to (A) above is also not an option.

Here is code that successfully starts ImageJ from IPython:

def start_imagej():
    import imagej
    global ij
    ij = imagej.init(headless=False)
    ij.ui().showUI()
    print(ij.getVersion())
from PyQt5 import QtCore
QtCore.QTimer.singleShot(0, start_imagej)

But make sure you have initialized Qt first using %gui qt or at launch via ipython --gui=qt.

ImageJ starts up asynchronously on the Qt GUI/main thread, at which point the ij reference gets populated and becomes usable from IPython. The same caveats as (A) above apply: if you write e.g. ij.ui().showDialog('hello') from IPython, it hangs, but with care, you can do GUI-related things using strategies (1), (2) and (3) above.

You can then also start napari simply with:

import napari
napari.Viewer()

C. Starting napari+ImageJ from Jupyter Notebook

Since Jupyter Notebook is just a different interface over IPython, this solution is pretty similar to (B) above:

Cell 1:

%gui qt

Cell 2:

def start_imagej():
    import imagej
    _ij = imagej.init(headless=False)
    _ij.ui().showUI()
    global ij
    ij = _ij
from PyQt5 import QtCore
QtCore.QTimer.singleShot(0, start_imagej)

Cell 2 returns immediately, having queued ImageJ for asynchronous startup on the Qt GUI/main thread.

Therefore, to ensure cell 3 does not execute before ImageJ is ready, I thought I’d be clever and try this:

import time
print('Waiting for ImageJ to start',)
while not 'ij' in globals():
    print('.',)
    time.sleep(1)
ij.getVersion()

But it does not work: if you run it before ImageJ has appeared, ImageJ startup hangs and the while loop runs forever.

And similar to (A) and (B) above: if you do anything that touches AWT like ij.ui().showDialog(...) from a Jupyter cell, it hangs.

You can then start napari from a subsequent cell using the simple:

import napari
napari.Viewer()

And—unlike with IPython from the CLI—napari’s Qt console button will work. But of course Java AWT operations from there also hang.

You can use the same (1), (2) and (3) tricks described above if you want to touch the Java GUI.

D. Starting ImageJ only from plain Python

The (A) solution above for launching ImageJ does so from inside napari’s Qt console. We also want to support launching and displaying ImageJ by itself.

Here is a plain Python script that starts up Qt and spins up ImageJ:

from PyQt5 import QtCore, QtWidgets

def main():
    app = QtWidgets.QApplication([])
    app.setQuitOnLastWindowClosed( True )

    def start_imagej():
        import imagej
        ij = imagej.init(headless=False)
        print(ij.getVersion())
        ij.launch()
        print("Launched ImageJ")

    QtCore.QTimer.singleShot(0, start_imagej)
    app.exec_()

if __name__ == "__main__":
    main()

Note that the app.exec_() call blocks the main thread, because Qt takes it over as its GUI/main thread. On macOS, the main thread is the only thread that works for Qt to use as its GUI/main thread.

Next steps

All of this is nice if you just want to play around. However, it’s also a hassle:

  • While napari is somehow able to start up directly via napari.Viewer() when IPython or Jupyter is in Qt GUI mode, ImageJ is not and requires the QtCore.QTimer.singleShot mechanism.
  • From plain Python, we are on the main thread and starting the Qt event loop blocks the main thread—see (D) above.

One idea to help mitigate this issue is to give imagej.init three modes: something like gui=headless, gui=blocking and gui=immediate (non-blocking). I think we can make imagej.init smart enough to do the right thing with Qt in different scenarios—but on macOS, invoking imagej.init(gui=immediate) would fail fast with an error that it’s not supported on macOS.

The other major issue is how easy it is to hang the Python process on macOS by making illegal imagej.ui() calls. Perhaps we could do something to guard against this in the pyjnius layer, but more research and experimentation is needed.

What do you think? Are the three gui modes a good way forward? Did we miss something? Does anyone know better ways to use Qt?

7 Likes

@ctrueden this is so great, thank you for doing this work! A world where people are launching napari and IJ instances willy-nilly from an IPython session is an exciting world indeed!

My own experience debugging event loops is very limited, so I only have a couple of very high-level comments right now:

While napari is somehow able to start up directly

We don’t do any magic here. I think this is just that napari is a Qt app so the Qt event loop knows exactly what to do. IJ is not and the Qt event loop gets angry. (The preceding sentence reflects the extent of my technical understanding of event loops. =P)

ij.ui().showDialog('hello') hangs

This is less than ideal. I wonder whether the object returned by ui() can itself be smart and launch things “correctly” with Qt if a Qt instance is detected?

Does anyone know better ways to use Qt?

I think Ahmet Can Solak is our local expert, but he doesn’t appear to be on here. I’ll try to prod him to add to this thread.

Thanks again!

1 Like

Unfortunately, probably not. These are all Java calls and do not (and should not) know anything about QT. It would certainly be possible to make sure that all GUI calls in ImageJ explicitly push onto the EventQueue but that is a lot of work and it is easy to miss things. In my opinion, it would probably best to encourage use of scyjava.awt.invoke and scyjava.awt.queue once available as suggested in (1) above.

Ideally, we would find a way to run QApplication.exec without blocking the thread that it is in but I do not know if that is possible at all. Help from QT experts would be greatly appreciated here.

I think it would be possible to rip apart the ij object and inject wrapped execution into all ui() functions:

Something like this (not quite working as written)
from jnius import PythonJavaClass, java_method, autoclass
class JavaRunnable(PythonJavaClass):
    __javainterfaces__ = ['java/lang/Runnable']

    def __init__(self, f):
        super(JavaRunnable, self).__init__()
        self._f = f

    @java_method('()V')
    def run(self):
        self._f()


class HackedUIService(object):
    def __init__(self, ij):
        super(HackedUIService, self).__init__()
        self._ij = ij
    def __getattr__(self, name):
        value = getattr(ij.ui(), name)
        if str(type(value)) in ['jnius.JavaMethod', 'jnius.JavaMultipleMethod']:
            return lambda: ij.thread().queue(JavaRunnable(lambda: value(kwargs)))
        return value
p_ui = HackedUIService(ij)
ij.ui = lambda: p_ui

However, I think it might create more problems than it solves.

I think this is the solution: being more careful on the Java side. If it works. I’ll do some tests. Many things are already properly guarded by handing off to the EDT—it might just be that showDialog in particular is not doing the right thing, and by fixing it on the Java side, the Python side no longer hangs.

Yeah—from all our research so far, it does not seem possible. I think we will have to live with QT blocking the main thread.

Definitely. Although I get the impression no one active in this area of our community knows too much about it…

Yes, sorry about this. I’ve tried asking a colleague who knows more Qt than I do and they drew a blank. We can try seeing if there is someone else we can find who knows more. Exciting to see this!