Visualizing feature measurements in napari using colormaps as LUTs

We’re currently using napari for visualization of our 3D images and their segmentations. We’d also like to assess the quality of single cell measurements we made in those 3D images. Thus, I’m looking into ways to visualize measurements on segmented objects in napari.
We have one prototype working that generates a new intensity images for each measurement. That looks something like this:

The downside to our current approach is that it’s quite slow and memory intensive (needs to generate a new 3D image for each feature we want to visualize. I’ve been trying to implement a simpler approach to this by using custom colormaps. The idea being: I display my label image using the napari.add_image() function and provide a colormap that maps each label value to the feature measurement. This would easily allow to generate dozens of colormaps for different features that could be checked with minimal computational overhead.

The issue I’m running into is creating colormaps and passing them to napari in a way that works in this case. I generate a colormap using the napari/napari/utils/colormaps/colormap.py Colormap class that scaled from 0 to 1 (instead of e.g. 0 to ~1000 for my label images). To figure out how I need to pass the colormaps, I just gave random colors to each label (like one would to just display a label image). But napari then displays multiple consecutive labels (e.g. labels 35-37) in the same color.

Here’s a specific example:
This image has 748 unique labels, the max label is 761 (a few labels are missing). I generated a napari.Colormap with either 749 colors (0 for background, 748 unique labels) with evenly interspaced colors or 762 (0 for background, 761 unique labels) with evenly interspaced colors.
When I use this colormap to display my label image, multiple labels get assigned the same color (first image: display using this colormap. Second image: Displaying as label image)


Is this a question of specifying the bins? There are more bins than are being displayed, almost like napari would only display ~1/3 of the bins (which would be ~256 bin => 8bit?)
Can colormaps be used for this kind of approach? If so, any idea why my test of just assigning random colors to each object isn’t working?

Eventually, I would want to display something related to feature measurements instead of random colors (see the first example image above). Is there a better way to achieve this or has anyone else tried to get there using colormaps?

1 Like

The full code to try this is a bit messy at the moment, because I tried basing things on how the label_colormap function works in napari. In the end, I have a colormap object:

Colormap(colors=array([[0.        , 0.        , 0.        , 0.        ],
       [0.28011563, 0.61863202, 0.46517712, 1.        ],
       [0.50714034, 0.30618754, 0.26792714, 1.        ],
       ...,
       [0.45735553, 0.2652652 , 0.12380371, 1.        ],
       [0.91726941, 0.65565044, 0.76327294, 1.        ],
       [0.77372795, 0.85242444, 0.71360439, 1.        ]]), name='test', interpolation='zero', controls=array([0.0000000e+00, 1.0000000e-05, 5.1049049e-04, ..., 9.9948951e-01,
       9.9999000e-01, 1.0000000e+00]))

This object has e.g. 749 colors or even 50’000 colors evenly interspaced from controls from 0 to 1. But even specifying way too many colors still has napari showing consecutive labels in the same color.
I view it in the napari viewer using:

viewer = napari.Viewer()
viewer.add_image(lbl_img, colormap=('test', lbl_colormap))

I specifically checked the colors 34-37 in my colormap and they should be very different, so the colormap would specify very different colors to be used (example for a colormap with 762 entries, which I though was the most reasonable, since the label values scale from 0 - 761 in that example):

lbl_colormap.colors[34:38]
array([[0.97180998, 0.27003965, 0.23497851, 1.        ],
       [0.29732901, 0.12059682, 0.12751216, 1.        ],
       [0.35923997, 0.83787304, 0.97641581, 1.        ],
       [0.88306409, 0.54777843, 0.12084796, 1.        ]])

hi @jluethi my initial thoughts are that this is something that we’d really want to enable using the labels layer and colormaps on it directly. We’ve been trying to make some improvements lately to how colormaps are handled there which might help. Maybe @DragaDoncila can take a more detailed look at what you’ve written above, and she might have a good suggestion!!

1 Like

You can use the 'direct' labels_layer.color_mode to map labels to RGBA colors. See this example:

labels_layer.color can be set to a dictionary mapping labels to colors. This is easy with skimage.regionprops_table. You need colors, not properties, so you can use plt.cm.<colormap_name> to get the colors from the values:

rp = pd.DataFrame(
    regionprops_table(
        segmentation, image, properties=('labels', 'my-property')
    )
)

import matplotlib.pyplot as plt

prop = rp['my-property']
prop_scaled = (
    (prop - prop.min()) / (prop.max() - prop.min())
)  # matplotlib cmaps need values in [0, 1]
colors = plt.cm.viridis(prop_scaled)
labels_layer.color_mode = 'direct'
labels_layer.color = dict(zip(rp['label'], colors))

EDIT: updated to avoid using built-in keyword “property”.

2 Likes

@jni Awesome, this works great for the visualization! It’s very nice that it works via a dict using the labels as keys! :slight_smile:
I have a few follow-up questions, based on the example I was able to produce with this.

  1. I have multiple features I’d like to visualise, e.g. Size, Roundness & Elongation here. In this attempt, I just added them as separate label channels, each with their own colormap and named after the feature. This doesn’t scale much past a few different features. Is there a way to select different features for visualization from e.g. a dropdown menu like the colormaps usually are (and provide a list of dicts for this, of course)?
  2. I saw that there were discussions on colorbars last year. What’s the current status there? (e.g. this forum discussion)
  1. Alternatively, if both feature wishes above aren’t in the near-term future: Is there a way to add the actual raw feature measurement to the hover value? e.g. in the screenshot above, I’m hovering over cell 235. Could info be added (e.g. via a label dict) to show this as 235, 730 (with 730 being an actual measurement value for that object)?

Thanks a lot for the fast replies & the great help! :slight_smile:

1 Like

Hi @jluethi ! First, I’m glad you managed to get the colouring to work. I’ll definitely look into why label values were overlapping, that shouldn’t happen even if you don’t directly set the colour.

I think there’s definitely potential for a feature in this idea, where users provide a list of labels and a list of dicts for colors and properties, and we display those as layer options in a dropdown.

This, we can do. Labels layers take a properties dict, the contents of which are displayed in the status bar. See this example for how to specify the properties dict.

1 Like

For this, I would make a plugin, with a dropdown of all the possible properties (perhaps the properties passed to the labels layer) and a matplotlib colormap selector. magicgui makes this super easy. Here’s an example of affinder doing this:

(AffineTransformChoices is an Enum defined earlier in that file — you could make a similar colormap enum.)

Indeterminate… :grimacing:

In addition to @DragaDoncila’s reply, I know @VolkerH had something like this working already last year, not as a hover but as a separate little table that would update as you moved the mouse. And @kevinyamauchi is working on a properties browser plugin here that might be relevant. (e.g. it would be interesting if on hover the table scrolled to and highlighted the relevant row.)

All in all, exciting times for data exploration in napari! :blush:

btw we’d love to see your work and maybe chat in more detail about this application and more in the devcommunity meetings! Next one is 9am Paris time on Wednesday Apr 21. Follow the #dev-meeting channel on our Zulip chat for details!

2 Likes

I think @VolkerH’s code is here although it’s probably not for public consumption and he might yell at me for sharing. :joy: There’s a gif of the functionality in the README:

1 Like

Hi,

@jni I don’t mind you posting that old example, which was an experiment/proof of concept for an object inspector. But it is very old and I am not sure whether the code will work with current napari.

Regarding visualizing propertties as heatmaps: there is the option to create a new intensity image:

but as @jluethi already pointed out that is quite memory intensive and also slow as a new array has to be created.

I like the idea of using the colormap on the label image. I have explored some similar ideas with filtering objects by properties by changing the colormap a while back here Visual filtering: colormaps with transparency/ limiting contrast scaling · Issue #998 · napari/napari · GitHub , but that was also based on an intensity image IIRC. Here, the only change to colormap was that the objects outside of the filter were set to transparent. Due to the automated contrast adjustment of the image layer the behaviour was not quite what I wanted. By changing the colormap for the labels layer (which does not have automated conrast adjusment if I am not mistaken) this could maybe be a feasible approach for quickly filtering objects interactively.

1 Like

Thank you @DragaDoncila, that works well! I was looking at the properties dict before, but hadn’t figured out how to use it. Now that you pointed me to it again, I dug a bit deeper. Actually, the documentation for how to use properties can be a bit misleading in some use cases. It says:

properties : dict {str: array (N,)}, DataFrame
Properties for each label. Each property should be an array of length
N, where N is the number of labels, and the first property corresponds
to background.

I couldn’t figure out how to use a DataFrame here, so I used the dict {str: array (N,)}. In images where there are non-monotonous labels (e.g. the labels are 1, 2, 5, 7, 9), N is the max label value (in that case 9), not the number of labels (in that case 5, respectively 6 with the background). Would be great if that docstring could be improved. I haven’t figured out how the DataFrame works, so can’t describe that. For the dict, it could be something like this:

properties : dict {str: array (N,)}, DataFrame
Properties for each label. Each property should be an array of length
N, where N is the value of the highest label, and the first property corresponds
to background. The index of the array is used to match labels to the property

Where would I best report such a suggestion for a minor docstring update?

(Also, @jni thanks a lot for the suggestions and the invitation to Zulip & the dev-meeting, it was very interesting! I’ll look into putting this in a plugin and will report back later :slight_smile: )

You’re right, that is misleading. You can open an issue over at napari on GitHub or also just raise a PR directly if you’d like!

For future reference, you can actually directly pass the properties dictionary to pandas.DataFrame and it will work. Taken from the example with coins:

label_properties = {
    'row': ['none']
    + ['top'] * 4
    + ['bottom'] * 4,  # background is row: none
    'size': ['none'] + list(coin_sizes),  # background is size: none
}

# convert the dictionary to a DataFrame
df_properties = pandas.DataFrame(label_properties)
print("Properties DataFrame: \n", df_properties)
color = {1: 'white', 2: 'blue', 3: 'green', 4: 'red', 5: 'yellow'}

# add the labels
label_layer = viewer.add_labels(
    label_image,
    name='segmentation',
    properties=df_properties,
    color=color,
)

napari.run()

Output:

Properties DataFrame: 
       row   size
0    none   none
1     top  large
2     top  small
3     top  small
4     top  small
5  bottom  large
6  bottom  large
7  bottom  small
8  bottom  small
2 Likes

More details: I remembered discussing the issue of non-monotonous labels at length, but I didn’t remember what we decided. It turns out that we decided to use a column with the magic name 'index', in which case you can just have your labels there and they will be correctly mapped to each property. See an example here, in the original PR.

In retrospect, perhaps the magic column name should have been 'label', which would directly match skimage regionprops. :confused: We could still add that as an option…

1 Like

Either way, this should also be documented in the docstring!!! :joy:

1 Like

I’ve made an issue here.

2 Likes

Great, thanks a lot @DragaDoncila & @jni for the further explanations and opening this issue for the documentation. A dataframe that takes the 'label' column would indeed be great :slight_smile:

I have now created a basic workflow for this as a napari-plugin: GitHub - jluethi/napari-feature-visualization: Visualizing feature measurements on label images in napari

This plugin applies a colormap on a label image in the viewer using a dataframe with relevant information. It’s still a very rough draft, but looks like that could work :slight_smile:

napari-feature-viz

I have a few questions though, maybe some of you know what direction to point me in (e.g. @jni )? Thanks already in advance!

Here are the points I’m currently struggling with:

  1. In order to visualize features, I need to get access to the dataframe containing this information. For the moment, I just load it from disk every time, but I’d prefer to be able to work with features from memory (e.g. for the next question). Can I somehow directly load it into napari? How do I get this connected to my plugin?
    I’m running napari from a jupyter notebook. I’d prefer to just somehow pass the df to napari & the plugin, but can’t figure out how.

  2. Feature selection: Can I populate a dropdown in the plugin window with the column names of my dataframe? How would I do that? Both for the label column & the feature column.
    (This would probably be much easier if the df is already in memory or require a 2-step process when the df is loaded from disk). But even with a df in memory, I don’t know where to start with this.

  3. Is there a good way to update my contrast limits with the auto-detected ones? E.g. if the user keeps the checkbox on, could the values below be replaced with the newly calculated limits?

  4. I’d like to set properties again like discussed above when using the add_labels function. How do I directly set the label properties in my plugin?
    E.g. I can set the colormap using: label_layer.color = colormap

But:
label_layer.properties = label_properties
(which works when passed to viewer.add_labels( […] properties=label_properties))
Now returns this error:

Traceback (most recent call last):
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/magicgui/widgets/_bases/value_widget.py", line 44, in <lambda>
    lambda *x: self.changed(value=x[0] if x else None)
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/magicgui/events.py", line 655, in __call__
    self._invoke_callback(cb, event)
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/magicgui/events.py", line 676, in _invoke_callback
    _handle_exception(
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/magicgui/events.py", line 140, in _handle_exception
    raise value
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/magicgui/events.py", line 674, in _invoke_callback
    cb(event)
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py", line 187, in _disable_button_and_call
    self.__call__()
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py", line 285, in __call__
    value = self._function(*bound.args, **bound.kwargs)
  File "/Users/joel/Dropbox/Joel/PelkmansLab/Code/napari-feature-visualization/napari_feature_visualization/_dock_widget.py", line 51, in feature_vis
    visualize_feature_on_label_layer(label_layer, DataFrame, Feature, thresholds=(lower_contrast_limit, upper_constrast_limit))
  File "/Users/joel/Dropbox/Joel/PelkmansLab/Code/napari-feature-visualization/napari_feature_visualization/_dock_widget.py", line 112, in visualize_feature_on_label_layer
    label_layer.properties = label_properties
  File "/Users/joel/opt/miniconda3/envs/hiPSC/lib/python3.9/site-packages/napari/layers/labels/labels.py", line 345, in properties
    self._label_index = label_index
UnboundLocalError: local variable 'label_index' referenced before assignment

Any directions are very welcome :slight_smile:

2 Likes

cool stuff!

we don’t really have a great native place for storing pandas dataframes yet… so I would just cache it locally in your module, eg:

from functools import lru_cache

@lru_cache(maxsize=16)
def get_df(path):
    return pd.read_csv(path)

# and then in your function
def feature_vis(DataFrame: pathlib.Path, ...):
    site_df = get_df(DataFrame)

You can add an additional widget and update it when the dataframe changes (this assumes that the columns of the dataframe don’t change after picking a filepath … if they are, then you’ll want to update widget.column_names.choices manually accordingly)

def _init(widget):
    @widget.DataFrame.changed.connect
    def update_df_columns(event):
        # event value will be the new path
        # get_df will give you the cached df
        df = get_df(event.value)
        widget.column_names.choices = list(df.columns)


@magic_factory(column_names={"choices": [""]}, widget_init=_init)
def feature_vis(DataFrame: pathlib.Path, column_names: str):
    ...

I’m not quite following this one yet. what event specifically would you like to have trigger some action, and what action do you want that to be?

I believe that was a bug that was fixed here: fix label properties setter (issue #2477) by alisterburt · Pull Request #2478 · napari/napari · GitHub

1 Like

Thanks a lot @talley for all the explanations!

Ok, thanks, neat idea with the caching :slight_smile: The reading from disk works, but is a bit limiting for some use cases. It means you always have to write a df to disk first, before visualizing the features in napari. If there is any development on this front that would allow for it to be passed directly, would be nice to know to add this to the plugin :slight_smile:

Ok, that sounds useful. My columns will not change in the df. But how do I trigger this event? E.g. can I trigger it when the user selects a path? Or do I need to trigger it e.g. with a button: Once the user specifies a path, they click “Load df”, it gets loaded and then I trigger this function? How could I implement those multiple buttons in the magic_factory setup?

I’m allowing the user to either have contrast limits auto-detected or to manually set them. If the auto-detection is chosen, it would be great if it updates the values shown to the user in the input boxes as well, such that the user can e.g. decrease one value by 10% and rerun the plugin.

Awesome, looking forward to this being in a release and then using it :slight_smile:

That is what this line in my example is doing: @widget.DataFrame.changed.connect. It’s saying: “when the data frame parameter changes in the widget, call this function.”

See more in the documentation here: Quickstart — magicgui

I think the same suggestion applies? If you have a Boolean parameter in your function signature, you can use the same strategy above (…changed.connect) to trigger any action

Ah cool, I’ll try this out then. Thanks @talley!

1 Like