vmTracking enables highly accurate multi-animal pose tracking in crowded environments

vmTracking methodological enhancements and efficiency

vmTracking showed superior performance compared to traditional markerless multi-animal pose tracking methods, consistently improving accuracy across various conditions, including tracking 3 or 5 animals and managing occlusion and crowding. Notably, vmT-DLC improved keypoint matching rates in scenes where traditional markerless multi-animal tracking methods such as maDLC and SLEAP previously showed less than 50% matching accuracy, increasing it to approximately 90%. Conversely, virtual marker tracking using SLEAP’s single-animal tracking approach (vmT-LEAP) displayed limited effectiveness, with performance depending on the specific tracking target and video conditions. In particular, the performance for tracking the 14-point lateralized keypoints was poor, which is likely due to difficulties in tracking the tail. As such, using saDLC in the virtual marker tracking step is considered optimal.

The high accuracy achieved by vmT-DLC appears to be linked to its ability to ensure that all keypoints are consistently predicted, thereby preventing FNs in the tracking process. While this may cause a slight increase in FPs, the overall high accuracy suggests that most predictions are valid. Consequently, vmT-DLC significantly mitigated issues such as missed detections, which are common in conventional markerless multi-animal tracking. Another factor contributing to the high accuracy is the drastic reduction in ID switches and ID mismatches that commonly occur in scenes with occlusion and crowding. These results confirm that vmT-DLC represents a robust solution for addressing occlusion and crowding.

vmTracking requires a two-step process. First, create virtual markers. Second, track with saDLC. Therefore, the high accuracy of vmTracking could also be interpreted as the result of inheriting the corrected maDLC results during virtual marker creation and further training with saDLC. However, increasing the number of training frames with maDLC or SLEAP does not necessarily achieve the accuracy levels reached by vmT-DLC. With maDLC, even when increasing the number of annotation frames, FNs (false negatives) persist at a certain rate without much reduction, posing an obstacle to improving accuracy. This may partly be due to the limitation in DLC version 2.2.3 used in this study, which cannot supplement FN keypoints during refinement annotation. Even if FNs could be corrected in annotated frames, this would be possible in only a subset of frames, suggesting a limited impact on reducing FNs overall. Ultimately, unless FNs can be reduced, it would be impossible to reach the accuracy levels achieved by vmT-DLC in scenes with occlusion or crowding.

Another argument against vmTracking might be that the effort spent on correcting data to create virtual markers could simply be used for correcting the results of markerless tracking directly to obtain accurate tracking outcomes. However, notably, the presence of virtual markers with incorrect IDs or missing markers seemed to have little to no impact on reducing tracking accuracy. It can be assumed that the cost of modification work to create virtual markers is overwhelmingly smaller than that of modification work required to obtain correct tracking results. While there may be a threshold for the acceptable frequency of these less-than-ideal virtual markers, this study suggests that even in scenes with occlusion or crowding, vmT-DLC could likely achieve 90% or more accuracy if 70% to 80% of the virtual markers were assigned correct IDs. This 80% accuracy suggests that even if 10% FNs occurred in markerless tracking, achieving 90% accuracy with the remaining data would suffice, aligning with the accuracy of unmodified maDLC and SLEAP. Although some researchers may consider this level of accuracy sufficient for practical tracking while others may not, it is adequate for creating virtual markers.

Thus, maDLC can achieve high accuracy with minimal annotation frames, even in scenes with occlusion or crowding, and this likely applies to SLEAP as well. This high capability will greatly assist in virtual marker creation by minimizing the training and manual corrections required for creating virtual markers. The requirement for further training and thorough correction to create more accurate virtual markers is considered low. After creating virtual markers, implementing virtual marker tracking with saDLC is expected to significantly address the challenges of missing predicted keypoints and ID switches in scenes with occlusion and crowding, enabling high-accuracy tracking. Overall, vmTracking appears to provide a higher level of accuracy with less effort than markerless multi-animal tracking alone, despite requiring modification work to create virtual markers and an additional tracking process.

Tracking when the number of subjects changes

In maDLC, when the number of target subjects in the video frames changes, keypoints of undetected subjects are not predicted. However, in saDLC, all keypoints are predicted regardless of the number of subjects present. This feature helps to prevent the missed predictions that frequently occur in maDLC during occlusion and crowding, contributing to the enhanced accuracy of vmT-DLC. However, when the actual number of subjects changes, saDLC inevitably results in stray keypoints predicting alternative locations, which can potentially cause confusion in the tracking results. Additionally, there is a fundamental dilemma in how to manage these stray keypoints during refinement for retraining. To avoid the constant effort of selecting and deleting these inevitably occurring stray keypoints, we attempted to apply them to the remaining mice during refinement. Importantly, this process involves labeling incorrect mice, which have virtual markers different from the correct ones that should be used for labeling the stray keypoints. When no target mouse is present, stray keypoints detect a provisional target, but when the correct mouse appears, it is prioritized for tracking. By annotating multiple-colored virtual markers, vmT-DLC can track different markers based on the situation. Consequently, vmT-DLC can handle stray keypoints without significantly compromising accuracy. Virtual marker colors are crucial for individual identification but are not absolute markers. This flexibility may contribute to vmT-DLC’s ability to achieve high accuracy, even if the virtual markers are not strictly accurate.

This validation involved intentionally controlling the entry and exit of subjects within the video frame. It did not address unpredictable movements, such as when only specific body parts (e.g., the head) temporarily left the frame or were entirely obscured by an object. In this study’s validation, there were also instances of dancers overlapping during tracking, making it challenging to predict the occluded dancer’s body parts. Although the tracking results in these cases might not be significantly incorrect, there is room for debate on their precision. Therefore, in these situations—namely, if the target is frequently obscured by physical barriers or frequently enters and exits the camera frame, or if reducing false detections and focusing solely on accuracy within the visible range is prioritized—choosing vmTracking may not be the optimal choice. However, if the study focuses on interaction behaviors in situations where individuals are in close proximity and there is a need to address occlusion and crowding, vmTracking would be the best choice.

Applicability to pose analysis tools

In this study, we conducted an evaluation using Keypoint-MoSeq as an application of pose analysis tools. Consequently, we found considerable agreement between maDLC and vmT-DLC. While the observed agreement might seem relatively low given that the same video was analyzed, the frequency of mismatched combinations in the confusion matrix was generally below 1%. Even in the case of syllable IV and syllable VI, which were among the more frequent mismatched combinations, the similarity dendrogram indicated that these syllables were closely related, suggesting that their action patterns do not clearly represent a discrepancy. Overall, the model generation by Keypoint-MoSeq was barely affected by the results from maDLC or vmT-DLC. The difference in accuracy between maDLC and vmT-DLC may not be a significant issue for Keypoint-MoSeq, which may be able to bridge the differences in tracking accuracy by filtering out keypoint noise and extracting behaviors through unsupervised learning.

However, there were instances where neither data set appeared to fully capture movement patterns, and there were cases where maDLC identified postures that did not align with the actual body (or body axis) orientation. As behavior is extracted from time-series data as a pattern of features, it is not always expected to match the actual behavior in every scene. Yet, if the number of errors increases, discrepancies between the extracted behaviors and actual movements may grow, even if keypoint noise is addressed. Generally, keypoint noise is often detected as high-frequency fluctuations in coordinates; therefore, noise arising solely from FPs is expected to be relatively easy to identify. However, in scenes involving prolonged occlusion or crowding, FN errors are more likely to persist, making it difficult to detect noise based on high-frequency fluctuations. Such scenes, with minimal movement, may impact posture pattern extraction more than movement pattern extraction. While vmT-DLC does not guarantee the absence of incorrect behavioral feature extraction, its characteristic of not producing FN errors may be advantageous for filtering keypoint noise accurately. Consequently, although the effect may be small, vmT-DLC might be able to capture behavioral features more reasonably than maDLC.

Cross-species and setting applicability of vmTracking

Optimizing virtual marker creation

Virtual markers have been shown to serve a role equivalent to physical markers. In addition to not requiring consideration of potential behavioral impacts, as with physical markers, virtual markers also offer the advantage of greater flexibility in their number and placement. This raises the question of how best to configure virtual markers.

While it might seem beneficial to increase the number of virtual markers, doing so may shift them from serving as individual identifiers to acting as indicators of body parts, which could occur even with only a few markers. For instance, if a virtual marker is set to a keypoint corresponding to the head in markerless tracking, most of the virtual markers will generally be located near the head. However, if a marker occasionally appears in a different location, that location might be misidentified as the head. This is an inevitable issue, but it might be that the likelihood of such misidentifications increases as the number of virtual markers increases. As the target is the animal itself, not the virtual marker, it is preferable to use fewer markers to better capture the natural characteristics of the subject.

Limitations

While vmTracking can be considered a method for high-accuracy tracking of multiple animals, it is important to recognize that there are some limitations that can make the application of vmTracking challenging.

First, vmTracking relies on virtual markers for individual identification, which makes applying vmTracking nearly impossible in situations where there are too many targets to be identified by virtual markers alone. However, such situations are mostly encountered in natural environments with wild animal herds, and vmTracking should present no issues in controlled environments like laboratories.

Third, when the subject is completely obscured by a large object, this is essentially equivalent to the subject not being present in the video frame, and as with other tools, tracking is impossible. vmTracking is not a method for tracking something that cannot be seen.

Fourth, while vmTracking using saDLC offers the advantage of eliminating FNs, it predicts even in scenarios where it would be preferable not to, such as in unpredictable situations or when the subject is temporarily out of the video frame. Currently, there is no effective method to address this issue, and the only practical solution involves manually removing the corresponding keypoints.

In summary, the development of vmTracking in this study marks an advancement in the field of markerless multi-animal pose tracking. vmTracking has demonstrated exceptional accuracy and efficiency, particularly in addressing challenges such as occlusion and crowding. The reliance on virtual markers simplifies the tracking process, greatly reducing the need for manual annotation and training, which makes vmTracking more user-friendly and practical for laboratory settings and broader applications. The efficiency of vmTracking allows for high accuracy with relatively few annotation frames, highlighting its potential for animal tracking in complex environments. Further research into the impact of the number, color, size, and position of virtual markers on tracking accuracy will open up new possibilities for refining vmTracking. Overall, vmTracking is a robust alternative to traditional tracking methods and a useful tool in the study of animal behavior, ecology, and related fields, providing an effective and efficient solution to some of the most persistent challenges in multi-animal tracking.

Multi-mouse data set

Behavioral data were collected from C57BL/6J mice housed in groups of 2 to 3 per cage in an environment maintained at a temperature of 24 to 26°C and under a 12-h light-dark cycle. These mice had unrestricted access to food and water. All behavioral recordings were performed during the light phase of the cycle. Cages and housing conditions were refreshed weekly. All experimental procedures were approved by the Institutional Animal Care and Use Committee of Doshisha University (Approval No. A23068).

Data collection occurred in 2 different environments. In the first setting, 5 C57BL/6J mice freely roamed a rectangular open field with a black floor measuring 55 × 60 cm and walls 40-cm high. Recordings in this environment were captured at a resolution of 640 × 640 pixels, a frame rate of 30 fps, and lasted approximately 30 min, with a total of 4 videos recorded. In the second environment, 3 C57BL/6J mice were placed in a circular open field with a white floor measuring 31 cm in diameter and transparent walls 17 cm high. Recordings in this setting were conducted at a resolution of 460 × 460 pixels, a frame rate of 30 fps, and lasted approximately 60 min, with a total of 4 videos recorded.

vmTracking procedure and tracking tools

Regarding SLEAP, we applied the top-down method for multi-animal tracking in the mouse six-keypoint tracking experiment and the fish five-keypoint tracking experiment, while both top-down and bottom-up methods were applied for the mouse lateralized keypoint tracking experiment. For single-animal virtual marker tracking in SLEAP, the single method was employed in the analysis pipeline.

Mouse six-keypoint tracking experiment

After the initial training, performance evaluation and video analysis were conducted. Based on the results, outlier frames were extracted using the jump algorithm, annotated, and then further trained. In maDLC, each additional training session following refinement was counted as an “iteration” (with the first training before refinement labeled as iteration-0 and the training after the first refinement as iteration-1). In the 5-mice tracking experiment, 7 iterations were performed, resulting in a total of 2,097 annotated frames. In the 3-mice tracking experiment, 11 iterations were performed, resulting in a total of 3,886 annotated frames. Data for tracking accuracy evaluation were obtained from these trained data sets. These annotation data generated by maDLC were also imported into SLEAP and used as training data for tracking.

For reference, the number of frames extracted per iteration in maDLC was set to 50 frames per video. However, there were cases where previously extracted frames were re-selected as outlier frames, and some videos were excluded from extraction (for example, in the three-mouse tracking experiment, we did not extract frames from videos in which the number of mice in the arena was manipulated starting from iteration-7 onward). Therefore, a perfect proportional relationship is not found between the number of iterations and the number of annotated frames. In this study, the number of iterations is used as an intermediate point for comparing results based on the number of annotated frames, and the effect of the iteration count itself is not taken into consideration.

Creating virtual marker videos

In vmTracking, it is necessary to conduct an independent markerless tracking project to assign virtual markers for individual identification. However, in this study, as there was an existing markerless tracking project for comparison with the vmTracking results, instead of creating a separate project for generating virtual markers, we used the aforementioned markerless tracking project for virtual marker creation. The virtual markers were created based on the intermediate data obtained during the additional training aimed at refining the markerless tracking results. Note that corrections, such as ID switches, were made during the virtual marker creation, but these annotations were independent of the annotations used to improve the multi-animal tracking results. No additional training for markerless tracking was conducted using the corrected results for virtual marker creation.

Furthermore, to simplify the creation of virtual marker videos, we also conducted an independent maDLC project to track only the 2 keypoints intended to be designated as virtual markers. However, during the initial video analysis step, tracklet stitching failed in all videos, and we were unable to obtain coordinate data, leading to the project’s discontinuation at that stage.

Virtual marker tracking with single-animal tracking methods

For the analysis of scenarios where the number of mice in the video frame changes, frames from these scenarios were added to the initial 1,167 annotation frames for additional training. In saDLC, the keypoints specified in the configuration file are almost always predicted without exception, so that no missing keypoints occur. Consequently, when the number of tracked individuals in the arena decreases, stray keypoints appeared as they lost their tracking targets. To deliberately manage these stray keypoints, the remaining mice were annotated with stray keypoints to learn new tracking rules. Specifically, in frames where only 2 mice were present in the arena, labeling was performed such that IDs 2 and 3 overlapped. However, in frames where only 1 mouse was present, all labels were assigned to that single mouse, an additional 517 frames were added (for a total of 1,684 frames), and data for tracking accuracy evaluation were obtained from these trained data sets.

Mouse lateralized keypoint tracking experiment

In the lateralized keypoint tracking experiment, we conducted both a physical marker tracking experiment and a vmTracking experiment. To eliminate the influence of factors such as differences in the annotation data used for training across conditions, the number of annotated frames, and the number of iterations, the same annotation data set was prepared. Additionally, no refinement or additional training was performed, and the results after a single round of training were compared. When using DLC, although there was a difference in the training network—DLCRnet for maDLC and EfficientNet_b0 for saDLC—the maximum number of training iterations was unified at 200,000.

In the physical marker videos, there were instances where the physical markers detached from the mice. From the sections of the video where the markers remained attached, twelve 3-min video clips (i.e., each consisting of 5,400 frames) were extracted. Using the physical markers as a guide, we annotated 14 keypoints per mouse on 180 frames per clip, totaling 2,160 frames, and trained the data using the same annotation data set with maDLC and saDLC. In the physical marker tracking experiment, tracking results were obtained by analyzing the video clips used for training.

In the vmTracking experiment, we used the markerless video and corresponding virtual marker video from the three-mice, six-keypoint tracking experiment. Additionally, we tracked the markerless video using idtracker.ai (described later) and created another virtual marker video (ID virtual marker video) that was distinct from the one created with maDLC. From each of these 3 types of videos, twenty-four 3-min video clips (i.e., each consisting of 5,400 frames) were extracted from the same scenes.

Using the virtual marker videos, we annotated 180 frames per video clip, totaling 4,320 frames. This annotation data set was then divided into 2 independent data sets, each containing 12 clips. The 2 data sets were trained independently, with the markerless video trained using maDLC, and the virtual marker and ID virtual marker videos trained using saDLC. After training, tracking results were obtained by analyzing the video clips from the data set that was not used in training, thereby minimizing the effects of overfitting on the training data. Additionally, the annotation data were imported into SLEAP, where the markerless video was trained using the top-down and bottom-up methods, and the virtual marker and ID virtual marker videos were trained using the single method. As with DLC, videos that were different from the training data set were analyzed. The results were used as the data for tracking accuracy evaluation.

Fish tracking experiment

A maDLC project was created for tracking 10 fish, with 50 frames extracted from each of 3 videos using k-means clustering and 5 evenly spaced points annotated along the midline from the head to the tail tip. These data were imported into SLEAP for training and analysis, and corrections were made based on these data to create a virtual marker video. The second and fourth keypoints from the head side were used as virtual markers, with node size set to 2 and the color palette set to “alphabet.”

A saDLC project was created for virtual marker tracking with these virtual marker videos. Five keypoints per fish were set, totaling 50 keypoints assigned across 10 fish. Annotation was performed with consistent IDs assigned across frames based on virtual marker color, followed by training, analysis, and refinement, resulting in a total of 852 annotated frames (no tracking accuracy evaluation data were obtained at this stage). Next, to compare maDLC and vmTracking under similar conditions, this annotation data set was applied to each project, followed by training and analysis to obtain results for tracking accuracy evaluation. In this study, a single video consisting of 2,259 frames (with 410 annotated frames) was used for evaluation.

Dance tracking experiment

A saDLC project was created for virtual marker tracking with the six-point virtual marker video. Each dancer was assigned 19 keypoints, totaling 133 keypoints across 7 dancers. Annotation was performed with consistent IDs assigned across frames based on virtual marker color, followed by training, analysis, and refinement, resulting in a total of 150 annotated frames (as with the fish tracking, no tracking accuracy evaluation data were obtained at this stage). Then, to compare maDLC and vmTracking for the 2 types of virtual marker videos under similar conditions, this annotation data set was applied to each project, followed by training and analysis to obtain results for tracking accuracy evaluation.

Creating tracking videos excluding virtual markers

idtracker.ai

Two markerless videos used in the three-mice tracking experiment were tracked using idtracker.ai. To eliminate the influence of black areas in the background, a region of interest was set to match the field in the markerless videos. The parameters for tracking were set to 3 for Number of animals, with Dark animals selected for a Bold intensity threshold of 100 and Bold area threshold of 1,000. After correcting ID switches in the tracking results, the data were output as virtual marker videos, from which twenty-four 3-min video clips were extracted. For tracking accuracy evaluation using centroids, the uncorrected data were used.

Physical marker

Mice tracking performance evaluation

Keypoint evaluation

These metrics were calculated as percentages relative to the number of target points (equivalent to the GT count) and recorded as Tgt match, FN, each type of FP, and ID switches. Additionally, as accuracy metrics focused on the predicted data, we provided Pred match (percentage of Match counts relative to the number of predicted data points) and Bodypart match (the percentage of the sum of Match counts and ID mismatch FP counts relative to the number of predicted data points). RMSE, which is a metric based on the number of predicted data points, was also calculated based on the distance to the corresponding GT keypoint.

In this study, Tgt match was given the highest priority. However, when an ID switch occurred, even if accurate tracking was performed with the switched ID afterward, it may still be classified as a body part mismatch FP, potentially leading to underestimated tracking accuracy. Therefore, to capture matches in cases of accurate tracking after an ID switch, we included a body part match, which was defined as a match that does not require correct ID assignment. While it would have been possible to evaluate the results after making manual corrections, this evaluation method was chosen to more accurately assess the raw outcomes of multi-animal tracking and vmTracking. Notably, because body part match is based on the predicted keypoints, it serves as a more lenient matching criterion compared to the stricter Tgt match. This approach aims to provide accuracy assessments across various criteria.

Centroid evaluation

For centroid evaluation, the centroid derived from posture-tracking keypoints and the centroid obtained from idtracker.ai were compared based on GT data. In the 14-keypoint lateralized tracking experiment, the centroid was calculated using 11 predicted keypoints, excluding the tail. If only 1 predicted keypoint was available, then that single point was used as the centroid. A direct comparison with the GT centroid was not performed. This decision was made because the centroid derived from posture tracking keypoints serves as a representative value of the posture tracking data and cannot be said to have the same meaning as the idtracker.ai centroid, making a direct comparison inappropriate. Instead, the obtained centroid was compared with the 3 central points along the GT keypoints’ midline to assess whether it approximately matched the correct mouse.

Specifically, if the obtained centroid was within 20 pixels of any of these 3 GT keypoints, it was classified as a Tgt match. Considering that the centroid derived from posture tracking and the centroid obtained from idtracker.ai are not equivalent, a larger value was set for the threshold compared to the 10-pixel threshold used in keypoint evaluation. Although the centroids were not directly compared with the GT centroid, as the GT serves as the reference, it was reasoned that the centroid derived from multiple keypoints of posture tracking might be closer to the GT keypoints, thus warranting a larger threshold. If it exceeded this threshold, it was classified as a FP, and if no centroid was obtained (indicating that all keypoints used for centroid calculation were FNs), it was classified as an FN. FPs were further categorized as Deviating FPs and ID mismatch FPs. Unlike keypoint evaluations, body part mismatches could not be assessed; therefore, Bodypart mismatch FPs were excluded from the centroid evaluation. Similarly, only matches based on predicted keypoints (Pred match) were counted, with Bodypart matches excluded. Additionally, since consecutive frame GT data were not used in the lateralized keypoint tracking experiment, ID switches were not counted. RMSE was also excluded from the evaluation, as establishing a common GT centroid was challenging.

Accuracy of virtual markers

Keypoint-MoSeq

To assess whether differences in accuracy between maDLC and vmT-DLC affect the results of pose analysis tools, we applied data from these methods to Keypoint-MoSeq, which is a machine learning tool that automatically identifies behavioral modules (syllables) from keypoint data and filters out noise of keypoints to enhance syllable identification. Nonetheless differences in tracking accuracy may still influence syllable identification.

We used results from a three-mice tracking experiment with 6 keypoints for evaluation using Keypoint-MoSeq. First, ID switches were corrected (excluding instances where IDs quickly reverted) to ensure ID consistency within the data set because ID switches could have a significant impact on Keypoint-MoSeq’s module identification. Then, a model was created using maDLC and vmT-DLC results derived from one of these videos. For model training, we ran 50 iterations of AR-HMM fitting and 500 iterations of full model fitting with vmT-DLC data, followed by 200 iterations with maDLC data. The κ value for full model fitting was set to 1e4 (10,000). This model was then applied to data from both maDLC and vmT-DLC, and results were assessed. Only syllables occurring at a frequency of 1% or higher were output, covering a movement duration of 1 s in the plot (0.5 s before and after each syllable).

Based on the resulting syllable plots and similarity dendrogram, we grouped 3 low-movement syllables into a single category. Additionally, syllables with an occurrence rate below 1% (excluded from plots and the dendrogram) were grouped together. Using a confusion matrix of syllables from maDLC and vmT-DLC data, syllable frequencies from both tracking methods were visualized, and Cohen’s kappa coefficient was calculated as an agreement measure. For mismatched syllable combinations with frequencies exceeding 1% in the confusion matrix, cases that remained mismatched for 2 s (60 frames) or more (as brief mismatches might be noise-related) were visually checked against the actual tracking footage to verify validity.

Evaluation of stray keypoints

Fish tracking evaluation

Of the 3 videos, a single video consisting of 2,259 frames was used for the evaluation. The evaluation was conducted based on frame-to-frame variations and outliers observed throughout a series of videos. Based on setting equidistant keypoints along the body axis, it was assumed that if the distances between adjacent keypoints were not uniform, the tracking was not accurate. To this end, the variance of distances between adjacent keypoints was calculated as a measure of tracking accuracy, with greater variance interpreted as lower accuracy. The variance for each fish per frame was calculated, the median variance identified, and the frequency of outliers assessed. To consistently assess the frequency of outliers, outlier thresholds were determined using the interquartile range method based on integrated data from maDLC and virtual marker tracking. For each condition, these thresholds were applied, and the number of frames with variance considered outliers for each fish was counted.

Schooling behavior, in which fish often swim in the same direction, was also evaluated. An angle based on 2 anterior keypoints for each fish was calculated to represent the head orientation, and the cosine similarity for head direction among all 10 fish was computed for each frame to examine synchrony. Furthermore, it is rare for the direction of the head to change abruptly between frames, such changes likely indicate a tracking error. Therefore, the angle of head direction was calculated using the 2 anterior keypoints for each fish, with a focus on the synchronization of these predictions. The difference in angles between frames was measured, and the changes exceeding 90° were counted as abnormal directional changes. Such frames were interpreted as lower tracking accuracy.

As a sampling-based evaluation using annotation data for training as the GT, the proportions of Tgt matches, FNs, each type of FP, Pred matches, and Bodypart matches were calculated. Additionally, a correlation analysis of cosine similarity was performed based on the number of frames where data were available in maDLC (i.e., frames without missing cosine similarity data).

Dance tracking evaluation

In this study, we evaluated the detection of similar poses during 2 s of similar movements and 2 s of static poses among 7 dancers. For each dancer, we calculated distances between all keypoints to create a standardized distance matrix. The RMSD of the standardized distance matrices between dancers was then computed as a measure of similarity; the more similar the poses, the smaller the RMSD. To determine the similarity of each frame, we calculated the mean RMSD across all dancer pairs per frame. Furthermore, as a sampling-based evaluation using annotation data for training as the GT (150 frames), the proportions of Tgt match, FNs, each type of FP, Pred match, and Bodypart match were calculated.

Statistical analysis

Outliers in tracking data, often due to tracking errors, are unavoidable. To mitigate the influence of these outliers, all statistical analyses in this study used nonparametric tests. For comparisons between 2 paired conditions, the Wilcoxon signed-rank test was used. For comparisons among 3 or more conditions, the Friedman test was applied for paired data and the Kruskal–Wallis test was used for unpaired data, with both followed by Bonferroni correction. Spearman’s rank correlation was used for correlation assessments.

More specifically, metric-by-metric comparisons in the mouse tracking experiments used the Wilcoxon signed-rank test for physical marker data, while other comparisons employed the Friedman test with Bonferroni correction. To examine the relationship between virtual marker accuracy and Tgt match, Spearman’s rank correlation was used. In the OC scene of the three-mice tracking experiment, the accuracy across varying annotation frame counts was evaluated using the Friedman test with Bonferroni correction. Comparisons of accuracy based on annotation frame counts relative to the vmT-DLC level were performed with paired comparisons using the Wilcoxon signed-rank test, applying Bonferroni correction with a comparison count of 12.

For overlapping keypoint evaluations as the number of tracked mice varied, comparisons of d1-3 were conducted within each condition based on differences in the number of tracked mice using the Friedman test (i.e., 5 comparisons across 3 conditions). Additionally, comparisons across tracking conditions for each d1-3 were made using the Kruskal–Wallis test (i.e., 3 comparisons across 5 conditions), with Bonferroni correction applied to the total comparison count of 45. Due to the small sample size, we also calculated Cliff’s delta as an effect size to aid in interpretation. For the Keypoint-MoSeq analysis, Cohen’s kappa coefficient was calculated to assess syllable agreement between maDLC and vmT-DLC results.

In the fish tracking experiments, the Wilcoxon signed-rank test assessed the median variance in keypoint distances for each fish, the frequency of this variance as an outlier, and the frequency of head orientation changes greater than 90° between frames. Spearman’s rank correlation analysis was conducted to assess the relationship between cosine similarity and ground truth. In the dance tracking experiment, RMSD values based on standardized distance matrices among 7 dancers were evaluated using the Friedman test with Bonferroni correction.

Note that bar graphs were used in the figures when n ≤ 20. In these figures, each plot represented individual measurements, bars indicated the mean, and error bars (if present) indicated the 95% confidence interval. Box plots were used when n > 20. In these plots, the bottom and top of the box represent the first quartile (Q1) and third quartile (Q3), with the line inside indicating the median. The ends of the whiskers represent the maximum and minimum values excluding outliers, and measurements beyond 1.5 times the interquartile range (Q3 to Q1) from Q1 or Q3 are plotted as outliers in circles. Additionally, the mean value is marked with a star symbol.