Create Training Dataset Error

Hi DLC Community,

I am working on a new network on DLC2.8b and I am running the network evaluation, however, for the past two times the program crashed when plotting. It did report back the train and test error. However, I was wondering how necessary is it for the network to complete the plotting for all the frames? It seems to have about 500 left out of the 1768 being processed… I attached the code below.

Thank you in advance for any help!

Running  DLC_resnet50_SPONTANEOUSDec28shuffle1_180000  with # of trainingiterations: 180000
Initializing ResNet
Analyzing data...
1768it [01:25, 20.58it/s]
Done and results stored for snapshot:  snapshot-180000
Results for 180000  training iterations: 95 1 train error: 3.02 pixels. Test error: 4.79  pixels.
With pcutoff of 0.6  train error: 2.95 pixels. Test error: 4.37 pixels
Thereby, the errors are given by the average distances between the labels by DLC and the scorer.
Plotting...
 69%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ                        | 1224/1768 [04:01<01:47,  5.08it/s]
Traceback (most recent call last):
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\deeplabcut\gui\evaluate_network.py", line 340, in evaluate_network
    comparisonbodyparts=self.bodyparts,
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\evaluate.py", line 921, in evaluate_network
    foldername,
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\deeplabcut\pose_estimation_tensorflow\evaluate.py", line 220, in Plotting
    foldername,
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\deeplabcut\utils\visualization.py", line 160, in plot_and_save_labeled_frame
    scaling=scaling,
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\deeplabcut\utils\visualization.py", line 52, in make_labeled_image
    fig, ax = prepare_figure_axes(w, h, scaling)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\deeplabcut\utils\visualization.py", line 185, in prepare_figure_axes
    frameon=False, figsize=(width * scale / dpi, height * scale / dpi), dpi=dpi
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\pyplot.py", line 545, in figure
    **kwargs)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backends\backend_wx.py", line 1969, in new_figure_manager
    return super().new_figure_manager(num, *args, **kwargs)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backend_bases.py", line 3261, in new_figure_manager
    return cls.new_figure_manager_given_figure(num, fig)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backends\backend_wx.py", line 1973, in new_figure_manager_given_figure
    frame = cls._frame_class(num, figure)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backends\backend_wx.py", line 1103, in __init__
    self.toolbar = self._get_toolbar(self.statusbar)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backends\backend_wx.py", line 1131, in _get_toolbar
    toolbar = NavigationToolbar2Wx(self.canvas)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backends\backend_wx.py", line 1377, in __init__
    NavigationToolbar2.__init__(self, canvas)
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backend_bases.py", line 2636, in __init__
    self._init_toolbar()
  File "C:\Users\jusjames\Anaconda3\envs\DLC-GPU\lib\site-packages\matplotlib\backends\backend_wx.py", line 1413, in _init_toolbar
    self.Realize()
wx._core.wxAssertionError: C++ assertion "Assert failure" failed at ..\..\src\msw\toolbar.cpp(938) in wxToolBar::Realize(): Could not add bitmap to toolbar

If you suspect this is an IPython 7.19.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Hello @Justin_James, plotting is not necessary at all (but it does give you extra confidence that the network’s predictions are good); you could pass plotting=False to evaluate_network to ignore it. I had worked on that PR but it apparently requires an update of wxpython, which then breaks the labeling GUI. So use it at your own risk while we prepare a new GUI :grin:

1 Like

Thank you for your response and reasoning!