Cellfinder-napari - "save training data" saves all cubes, not just re-classified ones

I am using the cellfinder plugin (0.4.15) on napari on macOS - specifically curation of a previously analysed dataset which already has 1000s of cells and non_cells. I have changed / confirmed a handful of labels for both cells and non_cells, terminal output confirms they have been added to the right layer. The problem starts when saving training data:

  1. Can’t select external drives in the save window (also it is not the system window).

  2. A much bigger issue is that it seems to save all cubes, not only the reclassified ones. This is a real problem.

  3. There is no progress bar in the terminal window, it would be nice to have some feedback.

Let me know if I am doing something wrong - unfortunately it can’t be used for generating datasets for re-training (which we still do with the command line tool on a cluster, 0.4.8 and can’t go higher for now).

cheers
Ede

Thanks for raising these @ederancz!

Can I check if you’re using a command-line installed version of napari, or whether this is the bundled version? Also did you install the plugin from the command line, or the plugin manager within napari?

For point 1, this sounds like something with napari specifically. I don’t see this on Ubuntu at least. @jni, @sofroniewn is this a known problem?

For point 2, Can I confirm that you have created new layers for these curated cubes, and that they are selected in the Training data (cells) and the Training data (non cells) boxes in the widget on the right hand side?

If so, I’ll see if I can find a mac to troubleshoot.

For 3 - Yep, the whole plugin needs more graphical feedback, especially for big datasets. I’ve raised an issue to track it. In the mean time, if you’re using napari from the command line, you should be able to see some (limited) progress feedback there.

I’ve not seen or heard about this before, can you maybe give a little more detail on your system? Thanks!

1 Like

Thanks for picking this up @sofroniewn.

From the file menu, all save options bring up the system window, everything works fine.

From the cellfinder/Curation plugin, “Save training data” brings up another window with the name “select output directory” which has limited functionality (and a look unfamiliar to me).

Here is the system info:

napari: 0.4.7
Platform: macOS-10.14.6-x86_64-i386-64bit
System: MacOS 10.14.6
Python: 3.8.8 (default, Apr 13 2021, 12:59:45) [Clang 10.0.0 ]
Qt: 5.15.2
PyQt5: 5.15.4
NumPy: 1.19.5
SciPy: 1.6.3
Dask: 2021.05.0
VisPy: 0.6.6

OpenGL:

  • GL version: 2.1 ATI-2.11.20
  • MAX_TEXTURE_SIZE: 16384

Screens:

  • screen 1: resolution 2560x1440, scale 2.0

Plugins:

  • brainglobe-io: 0.0.2
  • brainreg-standard: 0.0.2
  • cellfinder: 0.0.10
  • console: 0.0.3
  • ndtiffs: 0.1.1
  • svg: 0.1.5
1 Like

That’s caused by this line here and would need to be changed on the cellfinder-napari side.

1 Like

Thanks @adamltyson
Re point 2: If I load an xml file with classifications so I can curate, then I can’t create new training data layers (there is an error) so not sure how to do what you wanted me to confirm. I think I am doing something conceptually wrong, will try to get my head around it tomorrow.

1 Like

Great, thanks Talley. I am tagging @adamltyson for this.

@adamltyson, OK, I’ve got it, works now.
First add training data layers,
then load existing cell_classification and images.

It may be worth adding a description of curation specifically to Training data generation - brainglobe

@talley Could QFileDialog.DontUseNativeDialog cause @ederancz’s problem? I was just using this for a consistent aesthetic across OSs.

I don’t think I’m understanding the problem yet @ederancz, is the problem that the layers have to be added in a specific order for it to work, or something else?

Yes @adamltyson, if the existing classification xml file is loaded first no classification layers can be added.

Something weird is going on then (I can’t reproduce this). So just to be sure, if you do this:

  • Open napari
  • Load cellfinder training data generation plugin (PluginsAdd dock widgetcellfinderCuration)
  • Drag and drop a cellfinder xml file

What happens if you click “Add training data layers”? On my system, two new layers are added Training data (cells) and Training data (non cells), and they are automatically selected in the widget as Training data (cells) and Training data (non cells).

Sorry for causing confusion here @adamltyson
The sequence you wrote down works well even if training data layers are added after XML.

But If after adding XML, you select cell and non-cells from the XML as training data layers (which makes no sense, but I did it), then even after deselecting them (so training data layers in the widget are empty now), when you try to add training data layers it gives the error "Training data layers already exist, no more layers will be added).

I hope this makes sense.

Thanks @ederancz, that makes sense. It’s a bug and I’ve raised it here.

As a workaround, if you end up in this position again, and don’t want to restart the plugin, you can create two new napari points layers, and set these as the training data layers. There’s nothing magic about the button to add new training data layers, all it does is create new layers, names them, sets some display parameters, and automatically sets the selection in the widget.