Consistent Offsets when Creating Labelled Videos with DLC

Hi there,

I have been using DeepLabCut on human eye videos in order to measure pupil dilation, marking 8 points around the pupil. For most steps of the process, this has worked well. My training data set consisted of 160 frames and I trained up till 625,000 iterations. Pixel error was under 1. I then analyzed the videos.

However, when creating the labelled videos, I noticed a consistent offset in the points labelled on the videos. The labelled points were all slightly shifted on the y-axis, as can be seen in the screenshots below. The 8 points do accurately track the pupil, both its movements and dilations, but there is a clear movement of all the points along the y-axis.

pic2 picn1
picn

I checked to see if this shift along the y-axis was consistent by comparing the csv file with the coordinates for the labels on the frames I had manually labelled for the training set with the corresponding coordinates for those exact frames from the DLC output. What I found was that there was a consistent shift along the y-axis for all of the labeled points. That is, the points were being accurately tracked and identified, but had been moved up by some pixels on the Y-axis. The screenshot of the csv file below shows the difference between my labelled point and the point labelled by DLC. For the x-axis coordinate, there are no issues (a difference of hardly a pixel or two, which is to be expected) however with the y-axes, there is consistently a shift by around 45 to 47 pixels.

This offset is present in all of the videos I have analyzed and labelled. I can’t seem to figure out why this is happening, and was wondering if there is some error I am making or if this is an issue that someone else has run into before?

In case it is relevant, I set cropping parameters for the videos during the extraction step, and the videos were .mp4.

Many thanks!

Hi @zoyamooraj, can you confirm that your labeled images look fine when viewed in the GUI? I’d tend to think that this is purely a visualization bug, as otherwise you’d have gotten errors >> 1 pixel with that shift. I’ll investigate :smiley: Also, what specific version of DLC are you using?

Hey @jeylau, thanks for your response! I’m using version 2.1.8.2. In terms of this offset problem, yes the images looked fine while viewing them in the GUI when I placed the labels (that is, there was no consistent shift in all the points along the y-axis in all the frames).

I did however run into another small problem in regards to labelling in the GUI, which I do not think is related to or the cause of this offset problem, but may be worth mentioning anyway so I’ll briefly outline it. I noticed while labelling with the GUI that sometimes while moving back and forth between different frames I was labelling, and also after saving, closing and re-opening in the GUI, one or two points in a couple of frames would have moved by a few pixels. This was not consistent - it could be any one or two points in any frame. I am not sure if this is common or expected, as it is maybe something which could go unnoticed on larger labelled points (e.g. if a paw in general was being labelled) but since I was always trying to label the exact same contour/point on the pupil, this shift of a few pixels was quite evident.

How I initially labelled it:

How it looked when I returned to it:

Point 1 (the dark blue point) and point 4 (the turquoise point) have shifted here of their own accord, as can be seen in the second image.

Like I mentioned above, I don’t believe these two issues to be related, as the main offset issue I initially posted about is consistent as the offset is specific to the y-axis and affects all the points in each frame of all the videos. This secondary labelling issue is where one or two points will arbitrarily shift by a few pixels, and seems to randomly affect a frame or two in the whole set.

I hope I haven’t confused the post too much by introducing this second issue, but thought it might be relevant to mention since you had asked about how it looked in the GUI!

The small shifts you note for some keypoints was fixed back in August with 2.2b8 https://github.com/DeepLabCut/DeepLabCut/commit/cb0e7d1bf97fab7086a24b7a833087a473156597#diff-a0bd07b1fdafebe49028db5eb9a1fc6fecf07e50d3f34013caee7ca0f9c4a2e8. You’ll get the update if you upgrade to DLC 2.1.9.
Coming back to your original issue, do you get that vertical shift with cropping set to False?

Ah, thank you for that info, will upgrade!

Yup the shift occurs even then.