State space models reshape how event cameras process temporal data — a CVPR 2024 result

Event cameras produce a stream of asynchronous timestamped events rather than frames. That is their advantage. But it also makes them awkward to feed into the neural architectures that dominate vision: convolutional networks want spatial grids, transformers want token sequences of fixed length, recurrent networks accumulate hidden state across time steps. None of those assumptions match a sensor that might fire tens of thousands of events per millisecond during a fast maneuver and almost nothing during a static scene.

A CVPR 2024 paper from Zubić and colleagues proposes a different backbone: structured state space models (SSMs), specifically S4 and Mamba. These architectures were developed for long-sequence modeling in natural language and audio, where the key challenge is efficiently propagating information across many time steps without the quadratic scaling cost of attention. It turns out those properties transfer usefully to event streams.

The core argument

State space models represent a sequence as a continuous-time dynamical system, discretized at the input time steps. For event cameras, those time steps are irregular — events arrive when pixels change, not at a fixed clock rate. SSMs handle irregular time steps gracefully because the discretization is parameterized by the time delta between inputs, not by a fixed cadence. A recurrent network processing the same stream has to either ignore the time deltas or encode them as auxiliary inputs, which is a kludge. A transformer ignoring them loses the physics of the problem entirely.

Zubić et al. demonstrate this on gesture recognition and object recognition benchmarks. The SSM backbone produces latency-competitive results while requiring substantially less computation than transformer-based approaches on equivalent event data. The sequential processing structure also makes the model well-suited to streaming inference — you don’t need to accumulate a window of events before processing begins.

Why this matters for us

Terminal guidance has a latency profile that most computer vision research ignores. A camera on a frame-rate schedule imposes a minimum detection latency equal to one frame period. An event camera removes that constraint at the sensor level — but only if the processing pipeline can keep pace. An SSM operating on the raw event stream can in principle produce an updated state estimate after every event, with the latency determined by arithmetic throughput rather than frame timing.

That property is directly useful for fire control. The engagement geometry changes on a timescale of single-digit milliseconds in the terminal phase. A state estimator that updates on event arrival rather than on frame arrival can, in theory, track angular rate and acceleration at the sensor’s native time resolution. The question is whether an SSM can be made small enough to run embedded.

Deployment considerations

SSMs have a favorable property for embedded implementation: the recurrent form of the discretized state equation is a simple matrix multiply and addition per time step. For small state dimensions, that is well within reach of an FPGA or a SoC with a vector unit. The selective-state (Mamba) variant adds a data-dependent selection mechanism that complicates the hardware slightly, but the core operation remains simpler than a transformer attention block.

The authors don’t address embedded deployment directly — their target is GPU-class hardware — but the architecture points in the right direction. Work that takes the SSM formulation and quantizes it for fixed-point FPGA execution would be a meaningful step toward what the terminal guidance problem actually needs.

The broader takeaway from this paper is that event-camera networks benefit from architectures that respect the physics of the sensor rather than mapping events onto frame-based representations as a preprocessing step. SSMs are a more natural fit than most of what came before.