Associative memory extends event-camera object tracking to long-duration sequences

Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset (arxiv 2403.05839) introduces both a benchmark for long-duration event-camera tracking and a tracker architecture that maintains track through occlusion and re-appearance over sequences measured in minutes rather than seconds.

The short-track problem

Most event-camera tracking benchmarks contain sequences that are seconds long, during which the target remains continuously visible. In practice, real tracking scenarios involve partial and full occlusions, targets leaving and re-entering the field of view, and periods of low contrast where the event camera generates few target events.

In these conditions, a tracker that does not maintain a target model across the gap will lose track and fail to re-acquire the target when it reappears. Standard event-camera trackers based on nearest-neighbor matching or short-window correlation perform well on continuously visible targets but fail to re-acquire after even brief occlusion.

Associative memory for track persistence

The paper augments a standard event-camera tracker with a memory component that stores target appearance templates over time. When the tracker loses a target candidate (event-based target signature falls below a confidence threshold), the track is suspended but the template is retained. When new candidate tracks appear, the associative memory is queried to check whether any current candidate matches a recently suspended track.

This re-identification capability allows the tracker to re-acquire targets after occlusion — the stored template provides a reference appearance model that the current candidate is compared against, even if the comparison is across a substantial time gap.

Benchmark statistics

The benchmark dataset contains sequences with average duration of 3.7 minutes per track, compared to 8–30 seconds in prior benchmarks. Targets experience an average of 4.2 occlusion events per sequence. Standard trackers that perform well on the short-sequence benchmarks experience significant accuracy degradation on the long-term benchmark; the associative memory tracker maintains competitive accuracy.

Relevance to persistent area surveillance

For ground-based C-UAS sensors monitoring a site over extended periods, long-term tracking is the relevant capability. A perimeter sensor that loses and fails to re-acquire a target after it passes behind a building or tree line is not useful for tracking targets through complex environments.

For a seeker payload in the terminal phase, the track duration is measured in seconds rather than minutes — long-term re-identification is not the relevant capability. But the principle of maintaining a target model through periods of low confidence is directly applicable: a seeker that loses the target signature in momentary clutter or glare needs to maintain a track estimate through the gap and re-acquire rather than declaring track loss.

The associative memory architecture — storing a template, suspending the active track, querying on reappearance — is the right conceptual framework for this seeker behavior, adapted to the millisecond timescales of terminal guidance rather than the second-to-minute timescales of surveillance.

---

Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset. arxiv 2403.05839