MTevent: a multi-task event-camera dataset covering 6D pose and moving object detection

MTevent: A Multi-Task Event Camera Dataset for 6D Pose Estimation and Moving Object Detection (arxiv 2505.11282) introduces a dataset that addresses a gap in the event-camera research ecosystem: most datasets are single-task, optimized for either detection, tracking, or pose estimation, but real applications require all three simultaneously and benefit from joint training.

Why multi-task matters

A terminal seeker needs to solve detection and pose simultaneously. Detection determines whether a target is present; 6D pose — three translational degrees and three rotational degrees of freedom — determines where the target is in 3D space and how it is oriented. For the guidance problem, the quantity that matters is relative position and velocity, which requires both a target detection and a well-estimated 3D position.

Models trained on single-task datasets often fail to generalize when deployed in a scenario that combines tasks. A detection network trained on object presence without 3D information learns features that are useful for presence but not for localization. Joint training on a multi-task dataset allows shared feature representations to develop that are useful for both.

Dataset composition

MTevent contains synchronized event streams and frame images from multiple sensor configurations, with ground truth annotations for:

• 6D pose of rigid objects (position and orientation in camera frame)
• Bounding boxes for detected moving objects
• Object velocity in the scene

The simultaneous availability of all three ground truth modalities allows researchers to train and evaluate models that produce all three outputs jointly, rather than evaluating detection and pose estimation as separate problems with separate models.

The moving object portion of the dataset contains a variety of target classes at different scales and speeds, with explicit representation of fast-moving targets — up to 10 m/s within the camera field — to ensure the dataset captures the high-motion regime where event cameras have the most advantage.

Joint training results

The paper demonstrates that models jointly trained on all three tasks outperform single-task models on each individual task, even though the jointly-trained models share capacity across tasks. The improvement is largest for the 6D pose task, where the moving object detection labels provide additional context about which regions of the image are active targets and which are static background.

This is consistent with the broader multi-task learning literature: when tasks share underlying structure (both require detecting edges and motion boundaries), training one task provides useful gradients for the other.

Implications for seeker design

For a seeker payload, the multi-task result suggests that a single model producing detection, track, and relative-pose output jointly may be more parameter-efficient and more accurate than three separate specialized models. This matters in an embedded context where parameter count is a hard constraint.

The practical limit is that MTevent’s moving target speeds (up to 10 m/s) are well below typical terminal engagement closing speeds (50–200 m/s). The joint training benefit likely persists at higher speeds, but the dataset does not validate this directly. Collecting a version of this dataset at seeker-relevant closing speeds is a gap worth addressing.

---

MTevent: A Multi-Task Event Camera Dataset for 6D Pose Estimation and Moving Object Detection. arxiv 2505.11282