Skip to content

[NeurIPS'21] Unified tracking framework with a single appearance model. It supports Single Object Tracking (SOT), Video Object Segmentation (VOS), Multi-Object Tracking (MOT), Multi-Object Tracking and Segmentation (MOTS), Pose Tracking, Video Instance Segmentation (VIS), and class-agnostic MOT (e.g. TAO dataset).

License

Notifications You must be signed in to change notification settings

Zhongdao/UniTrack

Repository files navigation


[NeurIPS 2021] Do different tracking tasks require different appearance model?

[ArXiv] [Project Page]

UniTrack is a simple and Unified framework for addressing multiple tracking tasks.

Being a fundamental problem in computer vision, tracking has been fragmented into a multitude of different experimental setups. As a consequence, the literature has fragmented too, and now the novel approaches proposed by the community are usually specialized to fit only one specific setup. To understand to what extent this specialization is actually necessary, we present UniTrack, a solution to address multiple different tracking tasks within the same framework. All tasks share the same appearance model. UniTrack

Demo

Multi-Object Tracking demo for 80 COCO classes (YOLOX + UniTrack)

In this demo we run the YOLOX detector and perform MOT for the 80 COCO classes. Try the demo by:

python demo/mot_demo.py --classes cls1 cls2 ... clsN

where cls1 to clsN represent the indices of classes you would like to detect and track. See here for the index list. By default all 80 classes are detected and tracked.

Single-Object Tracking demo for custom videos

python demo/sot_demo.py --config ./config/imagenet_resnet18_s3.yaml --input /path/to/your/video

In this demo, you are asked to annotate the target to be tracked, by drawing a rectangle in the first frame of the video. Then the algorithm tracks the target in following timesteps without object detection.

Tasks & Framework

tasksframework

Tasks

We classify existing tracking tasks along four axes: (1) Single or multiple targets; (2) Users specify targets or automatic detectors specify targets; (3) Observation formats (bounding box/mask/pose); (2) Class-agnostic or class-specific (i.e. human/vehicles). We mainly experiment on 5 tasks: SOT, VOS, MOT, MOTS, and PoseTrack. Task setups are summarized in the above figure.

Appearance model

An appearance model is the only learnable component in UniTrack. It should provide universal visual representation, and is usually pre-trained on large-scale dataset in supervised or unsupervised manners. Typical examples include ImageNet pre-trained ResNets (supervised), and recent self-supervised models such as MoCo and SimCLR (unsupervised).

Propagation and Association

Propagation and Association are the two core primitives used in UniTrack to address a wide variety of tracking tasks (currently 7, but more can be added), Both use the features extracted by the pre-trained appearance model. For propagation, we adopt exiting methods such as cross correlation, DCF, and mask propation. For association we employ a simple algorithm as in JDE and develop a novel reconstruction-based similairty metric that allows to compare objects across shapes and sizes.

Getting started

  1. Installation: Please check out docs/INSTALL.md
  2. Data preparation: Please check out docs/DATA.md
  3. Appearance model preparation: Please check out docs/MODELZOO.md
  4. Run evaluation on all datasets: Please check out docs/RUN.md

Results

Below we show results of UniTrack with a simple ImageNet Pre-trained ResNet-18 as the appearance model. More results can be found in RESULTS.md.

Single Object Tracking (SOT) on OTB-2015

Video Object Segmentation (VOS) on DAVIS-2017 val split

Multiple Object Tracking (MOT) on MOT-16 test set private detector track (Detections from FairMOT)

Multiple Object Tracking and Segmentation (MOTS) on MOTS challenge test set (Detections from COSTA_st)

Pose Tracking on PoseTrack-2018 val split (Detections from LightTrack)

Acknowledgement

A part of code is borrowed from

VideoWalk by Allan A. Jabri

SOT code by Zhipeng Zhang

Citation

@article{wang2021different,
  author    = {Wang, Zhongdao and Zhao, Hengshuang and Li, Ya-Li and Wang, Shengjin and Torr, Philip and Bertinetto, Luca},
  title     = {Do different tracking tasks require different appearance models?},
  journal   = {Thirty-Fifth Conference on Neural Infromation Processing Systems},
  year      = {2021},
}

About

[NeurIPS'21] Unified tracking framework with a single appearance model. It supports Single Object Tracking (SOT), Video Object Segmentation (VOS), Multi-Object Tracking (MOT), Multi-Object Tracking and Segmentation (MOTS), Pose Tracking, Video Instance Segmentation (VIS), and class-agnostic MOT (e.g. TAO dataset).

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published