The Issue with deeplabcut.merge_datasets - The following folder was not manually refined

https://github.com/AlexEMG/DeepLabCut/issues/539

Environment:

  • Mac OS for labelling
  • Google Collab for training the models
  • Mac OS for analysing the video and refining the outliers

Hello! I have successfully trained a model with low pixel error, but there were still some outliers that I tried to clean up.

  1. I analysed a test video (not previously labelled manually or used for training) successfully;
  2. I extracted outlier from this video successfully;
  3. I refined 20 randomly selected frames manually via the GUI;

The problem appeared when I tried to merge the refined dataset.

Input:
>>> deeplabcut.merge_datasets(config_path)

Output:

The following folder was not manually refined,... /Users/season/Desktop/activesampling-tianqiu-2019-11-07/labeled-data/.DS_Store
The following folder was not manually refined,... /Users/season/Desktop/activesampling-tianqiu-2019-11-07/labeled-data/_DS_Store
Please label, or remove the un-corrected folders.

Speculation:

  1. Google Collab?

I used the GUI to label the frames and uploaded onto Google Drive. Then I used Google Collab to train the model. I then download the whole folder to my laptop, and the analysis, outlier extraction, refinement are all performed on my laptop.

Maybe, the transportation between the two environments might have resulted in the error?

  1. Only one video, and it’s a novel video being analysed and refined

As I said, the video I used to analyse is a novel video that is neither included in the manual labelling nor the training processes. Should I analyse all the video within the file and refine them to merge the file?

Let me know if you need me to elaborate further. Thank you!

Sincerely,

Season

This looks like a false/hidden folder; can you show hidden folders and delete this folder? I don’t think this is what your video is called? Otherwise what you describe for steps is correct!

Dear Dr Mackenzie Mathis,

Thank you for responding! I just found out the reason why.DS_Store and _DS_Store exist in the folder. I need to explain why they are added into the folder automatically by the MacOS system, so whoever encountering this in the future would know what to do.

  1. .DS_Store
    “In the Apple macOS operating system, .DS_Store is a file that stores custom attributes of its containing folder, such as the position of icons or the choice of a background image.”

The creation of these two files is the result of downloading the folder from Google Drive. For a file that is unzipped on MacOS, the finder would create a .DS_Store file.

  1. The Hidden File
    The file is hidden, and it does not appear even if the user typed
    Command + Shift + .
    “Since they are not user managed files and are intended to be used internally by Finder, they are not treated like other hidden files and kept hidden despite using the said shortcut.”

Therefore, for anyone who uses Google Drive and Google Collab to train their model, it is crucial to delete the .DS_Store and _DS_Store files. Otherwise, they would interfere with the .merge_datasets function.

Here is a Stackoverflow thread on how to delete them.

After I have deleted both files, the function ran successfully.

  1. Conclusion
    If you are using a macOS system(I am not sure if it is the same for Windows System), it is best to delete the .DS_Store and _DS_Store files, so the program does not confuse the two files with the other folders.

Sincerely,

Season

1 Like

Hello, Dr Mackenzie Mathis, I have solved the problem by deleting .DS_Store, which is created by the Mac system when downloading the zipped folder from Google Drive.

Thank you for the suggestion!

Sincerely,
Season

1 Like