Consistent Offsets when Creating Labelled Videos with DLC

Usage & Issues
#1

Hi there,

I have been using DeepLabCut on human eye videos in order to measure pupil dilation, marking 8 points around the pupil. For most steps of the process, this has worked well. My training data set consisted of 160 frames and I trained up till 625,000 iterations. Pixel error was under 1. I then analyzed the videos.

However, when creating the labelled videos, I noticed a consistent offset in the points labelled on the videos. The labelled points were all slightly shifted on the y-axis, as can be seen in the screenshots below. The 8 points do accurately track the pupil, both its movements and dilations, but there is a clear movement of all the points along the y-axis.

pic2 picn1
picn

I checked to see if this shift along the y-axis was consistent by comparing the csv file with the coordinates for the labels on the frames I had manually labelled for the training set with the corresponding coordinates for those exact frames from the DLC output. What I found was that there was a consistent shift along the y-axis for all of the labeled points. That is, the points were being accurately tracked and identified, but had been moved up by some pixels on the Y-axis. The screenshot of the csv file below shows the difference between my labelled point and the point labelled by DLC. For the x-axis coordinate, there are no issues (a difference of hardly a pixel or two, which is to be expected) however with the y-axes, there is consistently a shift by around 45 to 47 pixels.

offset_measurement
offset_measurement966×441 32.2 KB

This offset is present in all of the videos I have analyzed and labelled. I can’t seem to figure out why this is happening, and was wondering if there is some error I am making or if this is an issue that someone else has run into before?

In case it is relevant, I set cropping parameters for the videos during the extraction step, and the videos were .mp4.

Many thanks!

#2

Hi @zoyamooraj, can you confirm that your labeled images look fine when viewed in the GUI? I’d tend to think that this is purely a visualization bug, as otherwise you’d have gotten errors >> 1 pixel with that shift. I’ll investigate :smiley: Also, what specific version of DLC are you using?

#3

Hey @jeylau, thanks for your response! I’m using version 2.1.8.2. In terms of this offset problem, yes the images looked fine while viewing them in the GUI when I placed the labels (that is, there was no consistent shift in all the points along the y-axis in all the frames).

I did however run into another small problem in regards to labelling in the GUI, which I do not think is related to or the cause of this offset problem, but may be worth mentioning anyway so I’ll briefly outline it. I noticed while labelling with the GUI that sometimes while moving back and forth between different frames I was labelling, and also after saving, closing and re-opening in the GUI, one or two points in a couple of frames would have moved by a few pixels. This was not consistent - it could be any one or two points in any frame. I am not sure if this is common or expected, as it is maybe something which could go unnoticed on larger labelled points (e.g. if a paw in general was being labelled) but since I was always trying to label the exact same contour/point on the pupil, this shift of a few pixels was quite evident.

How I initially labelled it:

precrop
precrop700×406 57.1 KB

How it looked when I returned to it:

postcrop
postcrop693×406 56.6 KB

Point 1 (the dark blue point) and point 4 (the turquoise point) have shifted here of their own accord, as can be seen in the second image.

Like I mentioned above, I don’t believe these two issues to be related, as the main offset issue I initially posted about is consistent as the offset is specific to the y-axis and affects all the points in each frame of all the videos. This secondary labelling issue is where one or two points will arbitrarily shift by a few pixels, and seems to randomly affect a frame or two in the whole set.

I hope I haven’t confused the post too much by introducing this second issue, but thought it might be relevant to mention since you had asked about how it looked in the GUI!

#4

The small shifts you note for some keypoints was fixed back in August with 2.2b8 https://github.com/DeepLabCut/DeepLabCut/commit/cb0e7d1bf97fab7086a24b7a833087a473156597#diff-a0bd07b1fdafebe49028db5eb9a1fc6fecf07e50d3f34013caee7ca0f9c4a2e8. You’ll get the update if you upgrade to DLC 2.1.9.
Coming back to your original issue, do you get that vertical shift with cropping set to False?

#5

Ah, thank you for that info, will upgrade!

Yup the shift occurs even then.

#6

Hi @jeylau, I had wanted to revisit this thread regarding my original problem of the offsets because I’m finding that it remains to be the case as I continue with analysis with DLC. There is always a consistent offset along the y-axis, and is present in all the videos I analyze and label. The tracking is accurate, but just shifted along the y-axis. This occurs both with and without the cropping parameters set.

I was wondering if you had any idea as to what could be causing this.
Many thanks! :slight_smile:

#7

Are the shifts there if you check labels?

The reason I’m asking is that I was having a similar problem and the issue was that the labeling GUI was inconsistent in where the actual label was being placed. i.e., I could place it so that the tip of the arrow was right where I wanted it, but then when I checked labels it was shifted. This resulted in my training data being shifted, and my analysis results being shifted.

Try changing to matplotlib=3.0.3 and numpy=1.16.4 (you might want to set up a different dlc env with these just to make sure you don’t mess up some other features, and only use this dlc env for the labeling GUI).