Embedded graph convolutional networks bring GNN processing to SoC FPGAs for event cameras

Embedded Graph Convolutional Networks for Real-Time Event Data Processing on SoC FPGAs (arxiv 2406.07318) implements graph convolutional networks (GCNs) for event-camera processing on a Xilinx Zynq FPGA SoC, demonstrating that the class of algorithms currently achieving the best results on event-camera benchmarks can run on the hardware targets relevant to embedded deployment.

Why GNNs and FPGAs are an awkward pair

Graph neural networks are the current state-of-the-art architecture for many event-camera tasks because they preserve the spatial and temporal structure of event streams without forcing events into a regular grid. Events are nodes in a graph; edges connect nearby events in space-time; GNN layers aggregate information from neighborhoods in this graph.

The problem is that GNNs are not a natural fit for FPGA implementation. FPGAs excel at regular, predictable data access patterns — the same computation repeated over a regular array of data. GNNs have irregular access patterns: each node’s neighborhood is different, and the graph topology changes as new events arrive and old events expire. This irregularity is difficult to map efficiently to fixed FPGA hardware.

The solution approach

The paper addresses the irregularity problem with two techniques:

Spatial hashing: events are indexed into a spatial hash table that allows efficient lookup of a new event’s spatial neighbors without scanning the full event list. The hash table is implemented in the FPGA’s block RAM, enabling sub-cycle neighbor lookup.

Fixed-degree approximation: rather than processing variable-size neighborhoods exactly, the implementation uses a fixed number of neighbors per event (capping at the nearest K) which makes the computation regular. The approximation introduces a small accuracy loss but makes the hardware mapping tractable.

Performance results

On the N-MNIST and N-Caltech101 object recognition benchmarks, the FPGA implementation achieves accuracy within 1% of the full-precision software implementation at 4.2x the inference throughput compared to running on an embedded ARM CPU on the same chip, and at 200 mW average power consumption.

This 200 mW number is particularly important: it is a measured result on real hardware for a real task, not a theoretical estimate. For a seeker package where total sensing system power budget might be 2–5 watts, 200 mW for GNN inference is within the range of feasibility.

Extension to object detection

The paper’s primary demonstration is object classification, not detection. Detection is a harder task — it requires both identifying that a target is present and localizing it, which requires processing the full spatial event field rather than a small patch.

The paper notes that extending the implementation to detection would require larger on-chip memory for the spatial hash table and additional hardware for the detection head, but that the core GNN computation is the bottleneck and the extension should scale proportionally. An extended implementation for detection is identified as future work.

---

Embedded Graph Convolutional Networks for Real-Time Event Data Processing on SoC FPGAs. arxiv 2406.07318