I’m trying to use napari to add bounding box annotations.
I read here https://napari.org/tutorials/applications/annotate_segmentation and found the way to create box annotations with text from labels, however I don’t know how to manually add box annotations.
I want to create the following functions
Hi @hiroalchem, sorry no one has gotten back to you yet on this. Maybe @kevinyamauchi has a suggestion as he wrote that tutorial. I can also take a look when I get the chance!!
Hello @hiroalchem . I am very sorry for the late reply. I lost track of this over the holiday break.
My understanding of your post is that you would like to be able to create manual bounding box annotations and be able to control the text associated with each bounding box. As described in the example you linked above, the text for each shape is generated from the layer properties. The property values for the currently selected shape and the shape the next shape to be added are stored in shapes_layer.current_properties. Thus to either edit the text associated with the currently selected shape or the next shape to be added, we must modify the values in shapes_layer.current_properties. To be a bit more concrete:
# create a shapes layer where the text for each shape is the value of the 'class' property
shapes_layer = viewer.add_shapes(data, properties=properties, text='class')
# set the viewer so that the next shape to be added as the text 'cat'
shapes_layer.current_properties['class'] = np.array(['cat'])
I have pasted an example script to create a GUI for setting/selecting the text when making manual annotations. Please note that there have been changes to napari and magicgui recently and I tested this against napari==0.4.3 and magicgui==0.2.5. There is also a small hack I had to add (commented at the bottom of the script below) to get around a bug we currently have in creating empty layers with text (see this issue).
Apologies again for the late response. Please let me know if this isn’t what you had in mind or if you have any questions!
import napari
from magicgui.widgets import ComboBox, Container
import numpy as np
from skimage import data
# set up the annotation values and text display properties
box_annotations = ['person', 'sky', 'camera']
text_property = 'box_label'
text_color = 'green'
# create the GUI for selecting the values
def create_label_menu(shapes_layer, label_property, labels):
"""Create a label menu widget that can be added to the napari viewer dock
Parameters:
-----------
shapes_layer : napari.layers.Shapes
a napari shapes layer
label_property : str
the name of the shapes property to use the displayed text
labels : List[str]
list of the possible text labels values.
Returns:
--------
label_widget : magicgui.widgets.Container
the container widget with the label combobox
"""
# Create the label selection menu
label_menu = ComboBox(label='text label', choices=labels)
label_widget = Container(widgets=[label_menu])
def update_label_menu(event):
"""This is a callback function that updates the label menu when
the current properties of the Shapes layer change
"""
new_label = str(shapes_layer.current_properties[label_property][0])
if new_label != label_menu.value:
label_menu.value = new_label
shapes_layer.events.current_properties.connect(update_label_menu)
def label_changed(event):
"""This is acallback that update the current properties on the Shapes layer
when the label menu selection changes
"""
selected_label = event.value
current_properties = shapes_layer.current_properties
current_properties[label_property] = np.asarray([selected_label])
shapes_layer.current_properties = current_properties
label_menu.changed.connect(label_changed)
return label_widget
with napari.gui_qt():
viewer = napari.view_image(data.camera())
shapes = viewer.add_shapes(properties={text_property: box_annotations})
shapes.text = 'box_label'
# create the label section gui
label_widget = create_label_menu(
shapes_layer=shapes,
label_property=text_property,
labels=box_annotations
)
# add the label selection gui to the viewer as a dock widget
viewer.window.add_dock_widget(label_widget, area='right')
# set the shapes layer mode to adding rectangles
shapes.mode = 'add_rectangle'
# this is a hack to get around a bug we currently have for creating emtpy layers with text
# see: https://github.com/napari/napari/issues/2115
def on_data(event):
if shapes.text.mode == 'none':
shapes.text = text_property
shapes.text.color = text_color
shapes.events.set_data.connect(on_data)
Thanks for the answer. That’s exactly what I wanted to do! It helped me a lot! Thank you very much.
Hi @kevinyamauchi. Hope you are doing well.
I am trying to use the above code for a project I am working on where we would have to go through a series of images and do the bounding boxes. Per your code, the label widget is connected to the shapes layer and it is changing text and properties. But when I use it in a series of images case, it would end up creating multiple label widgets for each image that one has to keep track of closing once the image is completed annotating. Could you please let me know how to resolve this issue if my problem makes sense?
GitHub - czbiohub/napari-bb-annotations: Napari plugin for bounding box annotations - here is the code.
I changed
shapes = viewer.add_shapes(properties={text_property: box_annotations})
to
shapes = viewer.add_shapes(properties={text_property: box_annotations}, ndim=3)
then the widget has set up the properties correctly.
However, only texts of first slice displayed and when I moved slice, the text remained.
Hmm I wonder if this is related to Text on shapes layer fails when ndim>3 · Issue #2311 · napari/napari · GitHub. If we fix that issue will everything then work for you both @Pranathi.vemuri and @hiroalchem ?
Probably, although I haven’t tried ndim=3 option.
I was thinking of creating a shapes layer with same shape as image stack so I will have the option of using a slider to switch between frames of images and shapes, and also have only one labeling widget. I intend to save the several bounding boxes and labels per each slice of shape in a csv file as well.
@hiroalchem might have more idea as it seems like he tried ndim=3.
@sofroniewn
Thank you for your comment. It seems that the problem will probably be solved by fixing that one.
@Pranathi.vemuri
I am in the process of making something almost exactly like it.
Hey @Pranathi.vemuri ! It’s great to hear from you! Things are going well here. I hope all is well for you.
As I understand your question, you are trying to create a bounding box annotator where annotations can be applied on a per-image basis and you can change the image you are viewing/annotating via the dim sliders. If that’s the case, I think you, @sofroniewn, and @hiroalchem are on the right track. Making your images a stack (e.g., 3D stack of 2D images) and then setting the Shapes layer ndim equal to that of the image (ndim=3 keyword argument) in the example code I pasted above should work. Unfortunately, the text slicing bug @sofroniewn linked is preventing that from working properly.
I have made a PR for a fix for the text on shapes layers here. Please feel free to check it out!
Using that branch, I was able to annotate a stack of images making the following modifications to the example above: (1) passing a 3D image to image layer and (2) setting the shapes layer ndim=3 in the viewer.add_shapes() method. See below for a gif and the code (requires the branch from this PR).
Does this help?
import napari
from magicgui.widgets import ComboBox, Container
import numpy as np
# set up the annotation values and text display properties
box_annotations = ['person', 'sky', 'camera']
text_property = 'box_label'
text_color = 'green'
text_size = 30
# create the GUI for selecting the values
def create_label_menu(shapes_layer, label_property, labels):
"""Create a label menu widget that can be added to the napari viewer dock
Parameters:
-----------
shapes_layer : napari.layers.Shapes
a napari shapes layer
label_property : str
the name of the shapes property to use the displayed text
labels : List[str]
list of the possible text labels values.
Returns:
--------
label_widget : magicgui.widgets.Container
the container widget with the label combobox
"""
# Create the label selection menu
label_menu = ComboBox(label='text label', choices=labels)
label_widget = Container(widgets=[label_menu])
def update_label_menu(event):
"""This is a callback function that updates the label menu when
the current properties of the Shapes layer change
"""
new_label = str(shapes_layer.current_properties[label_property][0])
if new_label != label_menu.value:
label_menu.value = new_label
shapes_layer.events.current_properties.connect(update_label_menu)
def label_changed(event):
"""This is acallback that update the current properties on the Shapes layer
when the label menu selection changes
"""
selected_label = event.value
current_properties = shapes_layer.current_properties
current_properties[label_property] = np.asarray([selected_label])
shapes_layer.current_properties = current_properties
label_menu.changed.connect(label_changed)
return label_widget
with napari.gui_qt():
viewer = napari.view_image(np.random.random((5, 200, 200)))
shapes = viewer.add_shapes(face_color='black', properties={text_property: box_annotations}, ndim=3)
shapes.text = 'box_label'
shapes.text.size = text_size
# create the label section gui
label_widget = create_label_menu(
shapes_layer=shapes,
label_property=text_property,
labels=box_annotations
)
# add the label selection gui to the viewer as a dock widget
viewer.window.add_dock_widget(label_widget, area='right')
# set the shapes layer mode to adding rectangles
shapes.mode = 'add_rectangle'
# this is a hack to get around a bug we currently have for creating emtpy layers with text
# see: https://github.com/napari/napari/issues/2115
def on_data(event):
if shapes.text.mode == 'none':
shapes.text = text_property
shapes.text.color = text_color
shapes.events.set_data.connect(on_data)
Hi @kevinyamauchi Good to hear from you! 
Thanks for your response and for fixing the issue in napari, it totally helps. That is exactly what I need for my plugin and similar to what I was thinking about. I will wait for the fix to be merged in napari from @sofroniewn, others and rewrite the plugin.
Hi @sofroniewn Thanks for merging the PR. just one last question, how do I get the current slice index or frame number from shapes or image layer? I am going over the napari code for shapes and it seems like there is a z_index but I am wondering if that is correct. thank you!
@sofroniewn I just got it from viewer.status like so current_file = list(map(int, re.findall(r'\d+', viewer.status)))[0] . No worries, my question is answered for now!
@kevinyamauchi can you also make a PR to add that example to the examples folder? It is awesome!
@Pranathi.vemuri viewer.dims.current_step or viewer.dims.point should also have what you want too, without having to look at the viewer.status
Thanks, @sofroniewn. Sorry I said last question but I had another doubt come to my mind regarding loading existing shapes i.e bounding boxes and text, being able to edit them. Is that feasible?
For example, I would press a key like e it would enable editing of bounding boxes and text, or press key d and an incorrect bounding box can be removed, press key r and the annotation/label for the bounding box can be changed. Could that be achieved with shapes layer? Could you please let me know and if you have any example, or algorithm on how I could do that, that would be great too!
Thank you!
Definitely! I’d like #2420 to merge first (allows empty layers to be initialized with text) then I’ll make the PR to add the example!
@kevinyamauchi did we add support for editing the contents of the text? I seem to remember a PR where there was a line edit in the layer control panel such that you could then type into the line edit and it would change the text - for example if you run napari/add_shapes_with_text.py at 447d7bfd1f604e75656a201259b0348e2e33fb00 · napari/napari · GitHub then under the display text combo box should the line edit appear? I know in that example we use 'text': '{class}: {likelihood:0.1f}%',, maybe it makes more sense when the text is custom strings?
The text can be edited via the API, but we don’t have a GUI hooked up. Since the text is set by the layer properties, one would need to edit the properties. Something similar to our point annotation tutorial would be a great start (annotating videos with napari - napari). In the context of bounding box annotations, it seems that cycling through label values (as opposed to allowing arbitrary input at runtime) like in that tutorial could work well. @Pranathi.vemuri, is that the behavior you had in mind?
In the first prototype PR, we had some text editing capabilities, but those didn’t make it in the final version. Perhaps a feature for the future!
