How to add plot with same colour as label layer?

Hi there,

First of all, thank you for making napari - it’s become an instrumental piece of my image analysis workflow, and the fact that it’s so customisable makes the tool ever more amazing! So thanks again :slight_smile: You’re enabling further cancer research :smile:

To the point; I currently use napari for viewing 2D image time series. In line with my idea of using napari for the curation of tracking data, I’ve added a matplotlib plot that represents the track length of each label, with label IDs on y axis & frame number in x axis. See below.

The main issue I have is that if I want to check the track for cell ID: 8, for example, the fact that I have two different colourmaps makes the whole process inefficient; because colours in plot & in the image are not the same, I have to hover on the labels layer rather randomly until I find the one of interest. In this case, cell 8 is red/pink in the plot but grey in the image.

How can generate a plot with the same colourmap as the label image? Is this possible?

Other things that I think would be nice to implement (plot-wise) but not sure how to:

  • re-plot track lengths after some editing/curation; my idea here was calculate tracks & completely remove the widget (as I’m not sure if you can change the plot itself without removing it?) but the command viewer.window.remove_dock_widget(mywidget) simply hides the plot-widget and doesn’t remove it. Any ideas?
  • a vertical line showing current frame
  • making the plot-widget responsive to user-clicks: clicking on a specific track on the plot would trigger a function that emphasizes the relevant label on the label_layer → this is only a dream though as I know it’s far fetched xD

Thanks in advance!

Cheers,

Pablo

1 Like

Hi Pablo

That’s great that you’re finding napari useful, and thank you for the kind words above!

Yes, there are a couple ways you could think about doing this. You can extract the rgba color of any given label using layer.get_color(i) where i is the numerical value of the label. Using this method you could make an Nx4 array with the colors of each label using np.array([layer.get_color(i) for i in range(1, 20)]) (note here you might need to do some +1 index adjustment as 0 is background and has no color.

We’ve recently added support for having properties on the labels layer in #1281 that might be of interest to you too, and we’re planning soon to allow for custom colormaps on the labels layer soon, see #716, which would be an alternative way of controlling this.

As for the more custom plotting feature requests, we definitely want to make adding additional interactive plots like you’ve done easier - we have some ideas around that there are in progress in #823, but it may take some time.

You should be able to do this, you might need to keep an additional handle to the plot around somewhere so you can update it though. If you post a minimal example maybe someone here can help you.

This is also something you should be able to do in your matplotlib plot by making a second plot call

For this I’m not sure about how to do this in matplotlib (I do think it will be possible), but there are vispy based plotting widgets, and potentially others that make getting clicks out easier - see #823 for inspiration

Hope this helps!
Best
Nick

1 Like

Hi Pablo!

To reiterate what @sofroniewn said, thank you so much for your kind words! It’s so motivating to hear of others doing cool stuff with napari! :tada: And the above is very cool.

Except for the colour matching, all of your questions are actually matplotlib questions rather than napari questions, so I would encourage you to post your use case with code and a sample image at discourse.matplotlib.org, and please link to your post in this thread, so we can follow. I’ll try to get you started here though.

You can absolutely modify a matplotlib plot after the fact, and indeed this is what you should do because it will generally be much faster! If you have your matplotlib patches objects (which are returned when you make the plot), you can always do get_data and set_data on it — you should have a look at yours and see if you can figure out the structure of the data. Even in the case of changing number of data, you can do set_ylim and set_xlim on your axes, and it’ll be much faster than destroying and creating your plot each time.

Again, here you want to create the line with matplotlib.Axes.vline() and register a callback to change the data with napari. You should be able to do this with the axis event on the viewer viewer.dims.axis.connect(my_callback) Edit: viewer.dims.events.axis.connect(my_callback) (thanks @talley for the correction!). The callback will receive an event as input with two attributes, axis and value. When axis == 0 it means the slice has changed, and the value is the changed slice, so you can use set_data on your vline to move the vertical indicator.

Note that you might need to call figure.canvas.draw() when you update data — I don’t have a good grasp on when .draw() is required vs when it’s triggered automatically, and again, that probably something the good folks at the Matplotlib discourse can help you out with. For reference, you can see in cell 17 on this notebook how I create a plot (loss_canvas = ...) and then define a function to update it (def update_loss: ...), which should give you some ideas about what to try out if things aren’t working.

Actually, this turns out not to be far-fetched at all! =) In this case, you want to tie to the matplotlib event system, which is described in this page. As you can see there, you can register a callback with a click on the canvas, and you’ll get back the x and y coordinates of the click in both canvas coordinates and data coordinates. (They also have a gallery example with more sophisticated picking here, but I’d try it with the basic stuff first.) You should be able to determine which track you’ve clicked on from that and make something happen on the napari side. One option would be to add a shapes layer overlay and display the bounding box of the label number. This should be easy enough by running skimage.measure.regionprops on the layer and pulling out the bbox attribute for the appropriate label. Note though that regionprops returns a list of properties object from label 1…N. To get a quick mapping, you’d have to build a dictionary as follows:

props = measure.regionprops(labels_image)
props_dict = {
    p.label: p.bbox for p in props
}

for example.

Once you’ve created the shapes layer, you can do layer.data = to set the data for the rectangle coordinates.

… So that should get you started, and maybe even finished! :joy: Best of luck and let us know how it goes!

1 Like

You should be able to do this with the axis event on the viewer viewer.dims.axis.connect(my_callback) (though @talley @sofroniewnplease correct me if I have the wrong place for the callback here).

Slight correction to the callback hookup… you need “events” in there:
viewer.dims.events.axis.connect(my_callback)

1 Like

I knew I’d mess it up! :joy: Edited above for posterity. Thank you @talley!

Hi all,

Apologies for my delayed response - I have been busy working on other areas of my image analysis pipeline.

I’ve spent a looong time ‘playing around’ with napari and seeing what I can and can’t do. After much frustration, but thanks to your answers, I managed to get the colour matching right, see below

I’ve also managed to add a vline to my matplolib image, that updates as I scroll through time. Unfortunately, this seems to affect napari’s performance (hence why I’m posting here again and not on discourse.matplotlib.org), as I understand that running this on the same thread is what is making it sluggish? See below for comparison before & after implementation of viewer.dims.events.axis.connect(my_callback)

Normal Slow

Here’s a simplified (i.e. without colour adjustment) version of the update_time function

def update_time(event):
    static_ax.ti_vline.set_xdata([event.value,event.value])
    static_ax.figure.canvas.draw()

I’ve looked into multithreading and made a few attempts with the @thread_worker decorator, but was unsuccessful.

worker = update_time()
worker.start()

Doesn’t work (i.e. vline doesn’t get updated). Most probably because this is not the right way to do it (I really am a noob when it comes to anything Qt/event handling). Would you have any ideas whether it’s even worth doing multithreading here and if so, how would you go on about doing it?

I know of vispy, and that probably there are ways of using it that wouldn’t affect the performance as much, but I don’t really have the time to explore, trial & error, and I’d rather stick with matplotlib for now.

On a side note, I’ve added some buttons using magicgui (really appreciate it when you make it so straightforward). However, when manually resizing the viewer, the ‘default’ dock widget frame/window/canvas seems not ideal. See bottom left of screenshot below.

I want to make the height smaller. I am able to change this using the setGeometry function but again, as soon as the window is manually changed then the widget reverts to its original ‘size’. Any pointers? Or not possible to change?

Again thank you for your previous answers - they were very useful

Cheers,

Pablo

3 Likes

Hi again @pablooriol2! Delays are totally fine, we are all keeping busy! :sweat_smile:

At this point, it would be super useful to see your full code and an example dataset. I’m not sure but it may be that updating the matplotlib plot (canvas.draw()) has to happen on the main thread anyway because it is updating a Qt GUI element. If that’s the case, then the way forward would be to check (a) that we are doing the minimum work possible on the plot, and (b) whether there is a faster way to update only certain elements of the plot. But, for the latter, it would be good to have complete code that we can boil down to something we can time.

Either way, the demo is looking super cool! I’m confident we can get it to where you want to go! :smiley:

Hi @jni,

Thanks for the prompt response!

Here’s a simple but working (jupyter notebook) script:

from matplotlib.backends.backend_qt5agg import FigureCanvas
from matplotlib.colors import ListedColormap, LinearSegmentedColormap
from matplotlib.figure import Figure
import matplotlib.pyplot as plt
import napari

%gui qt

def get_colour(layer,track_IDs):
    
    colors = np.ones((max(track_IDs)+1,4))
    
    for cell in track_IDs:
        colors[cell] = layer.get_color(cell)
    colors[0] = [0,0,0,1]
    newcmp = ListedColormap(colors)
    
    return newcmp

def update_time(event):
    static_ax.ti_vline.set_xdata([event.value,event.value])
    static_ax.figure.canvas.draw()

# data variables
# tracks_img = fov.tracks_curated.mod
# tracks_length = fov.tracks_curated.tracks_length
# cellIDs = fov.tracks_curated.cellIDs

viewer = napari.view_labels(tracks_img,name='Curation',opacity=1)

#generate colourmap
newcmp = get_colour(viewer.layers['Curation'],cellIDs)

cellIDs_text = [str(int(integer)) for integer in cellIDs]

with plt.style.context('dark_background'):
    plt.rcParams['figure.dpi'] = 110
    mpl_widget = FigureCanvas(Figure(figsize=(8.5, 13)))
    static_ax = mpl_widget.figure.subplots()
    static_ax.imshow(tracks_length,cmap=newcmp,aspect='auto')
    static_ax.set(xlim=(0, 120), ylim=len(cellIDs))
    static_ax.ti_vline = static_ax.axvline(0,0,len(cellIDs)+0.5,color='yellow',linewidth=4)
    static_ax.set_xticks(np.arange(0, 121, 6))
    static_ax.set_yticks(np.arange(0, len(cellIDs)+1, 1))
    static_ax.set_yticklabels(cellIDs_text)
    static_ax.tick_params(labelright=True, right=True,labeltop=True,top=True,colors='white')
    mpl_widget.figure.tight_layout()
    
mywidget = viewer.window.add_dock_widget(mpl_widget,name='tracks_length',area='right',shortcut='t')

viewer.dims.events.axis.connect(update_time)

Here’s a zip file with the data variables: Sample_data.zip (530.2 KB)

Let me know if you need anything else and if the code + data works

Thanks again

Pablo

Thanks @pablooriol2, I was able to get the code up and running, and I agree with @jni: the matplotlib call to static_ax.figure.canvas.draw() is what’s killing the performance here. It takes around 125ms on my computer. And I also agree with @jni, since it’s the actual draw event itself that’s slow, it’s not immediately clear to me how this could be sped up with threading directly (at the very best … all you could do with threading here is to keep the GUI slider or and other viewer elements responsive while the next plot is calculating … but you still won’t get faster than ~7-8 FPS if it takes that long to calculate the next draw).

I’m not a matplotlib expert, but I agree it would be worth looking into doing less in that update_time function: since you’re only moving a vertical line, it seems like there should be a faster way to update the plot (but again, I’m not sure). Something like this could be implemented with the vispy.plot module, which would likely be more performant to move a single line (but would then require learning a new plotting API :stuck_out_tongue_winking_eye: … and vispy.plot is still mostly experimental, without a ton of docs)

Hi @talley,

Thank you for this.

Yes, indeed, my feeling is that all this is possible with vispy.plot, but as you say; it involves learning it, meaning it’ll be a time black hole xD

For now though, I’ll disable this feature and do without it since it’s not crucial to the curation of the data, essentially it’s not worth sacrificing such drop in speed performance over a moving bar. Looked nice though :stuck_out_tongue:

BTW, building call buttons with magicgui is proving to be super useful - so thank you once again! One last ask for help; how can I programmatically place my widget of call buttons under my plot with a horizontal layout? If I place all my widgets together (plot + buttons) and then add them to the viewer, all call buttons show up vertically, whereas a horizontal layout is much preferred. See screenshots below.

The current workaround is placing the call buttons in area='bottom' and then manually placing the widget under the plot, which is not ideal. Any pointers would be welcome :slight_smile:

Cheers,

Pablo

Hi Pablo,

Something clicked over time with this example and I remembered “blitting” in Matplotlib can dramatically improve performance of draw calls in programs like this. Also, there is draw_idle which might skip some MPL frames in favour of napari frames, though I don’t know whether this will be the case. I’m not sure whether blitting will help in your case but it should. Some potentially helpful posts:

https://matplotlib.org/3.1.1/api/animation_api.html

You might actually need to dig a bit into the animation code, but if animation is doing it, you should be able to do it. =)

1 Like

Hi @jni,

Thanks for this! Will definitely check it out and try to implement it once I find some time!

Will report back with (hopefully successful) results :slight_smile:

Cheers!

Without digging through all of the code, it is definitely possible to get user events back through a Matplotlib figure, see https://matplotlib.org/users/event_handling.html .

Switching from draw -> draw_idle is the right thing to do. draw means “render the figure now”, draw_idle means “the next time the GUI refreshes re-render” and will not block your callback. I don’t know enough about exactly what is going on under the hood here, but it might help. It is also possible that the Matplotlib figure is getting rendered more than once…

We just merged some improved documentation about how to work with blitting in Matplotlilb https://matplotlib.org/tutorials/advanced/blitting.html . In particular you want something like the class-implementation which should push the mpl frame rate up enough that we won’t be the bottle neck.

Using ax.bars instead of an image may also drastically improve the render time (drawing a rectangle we know the location of is way faster than aggressively up-sampling an image). Is it easy to get the start / step frame of the tracks? If so https://matplotlib.org/gallery/lines_bars_and_markers/broken_barh.html is probaly the right way to go.

2 Likes