Masked autoencoder pretraining advances event-based point-cloud action recognition

A 2025 paper adapts masked autoencoder pretraining to event-camera point clouds, enabling self-supervised learning that dramatically reduces the labeled data requirement — useful for domains where collecting labeled data is difficult.

Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras (arxiv 2501.01040) applies the masked autoencoder (MAE) pretraining paradigm — which has driven significant progress in image and video understanding — to event-camera data represented as 3D point clouds.

The labeled data problem

Event-camera algorithms generally need labeled training data: event streams with associated annotations indicating what targets are present and where. Collecting this data is expensive, particularly for the scenarios most relevant to defense sensing — events collected from field hardware, outdoors, with real targets.

Self-supervised pretraining offers an alternative: train the model on large amounts of unlabeled event data first, then fine-tune with a small amount of labeled data. If the pretraining task forces the model to learn useful representations of the data structure, the fine-tuning step needs far less labeled data than training from scratch.

Masked autoencoders are currently the most successful self-supervised pretraining approach for images and video: mask out a random portion of the input, train the model to reconstruct the masked portion, and the learned representations are useful for downstream tasks.

Event point cloud representation

The paper represents event streams as 3D point clouds in (x, y, t) space — x and y are spatial coordinates, t is timestamp. Each event is a point at its (x, y, t) coordinates with polarity as a point attribute. This representation preserves the temporal precision of events without forcing them into frame-like grids.

The MAE pretraining task is: mask a random subset of points from the point cloud, reconstruct the masked points from the unmasked points. The model learns to predict where events will occur given partial event stream observations — which requires understanding motion patterns and sensor characteristics.

Fine-tuning with limited labels

After pretraining on unlabeled event streams from a diverse collection of scenes, the model is fine-tuned on the target task (action recognition) with a small labeled dataset. The paper demonstrates that fine-tuning with only 10% of the labeled data achieves accuracy comparable to training from scratch with the full labeled dataset.

This is a 10x reduction in the labeled data requirement — a significant practical advantage for scenarios where labeling is expensive.

Transfer to new target classes

The pretraining benefit is largest when fine-tuning on target classes not represented in the pretraining data. The pretraining task teaches the model to understand event-camera data structure in general, which transfers across target classes. Fine-tuning then adapts the learned representation to the specific classes of interest.

For a seeker application, this suggests a pretraining strategy: collect large amounts of unlabeled event data during development testing — flights, field tests, sensor noise characterization — and pretrain on all of it, even the recordings where no target is present. Then fine-tune with a smaller labeled dataset of target detections and tracks.

The approach does not eliminate the need for labeled data entirely, but it reduces the labeled fraction needed to reach a given accuracy, which is practically valuable when labeled terminal-guidance data is scarce.

Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras. arxiv 2501.01040