Streaming track processing: stateful pipelines for real-time fusion

Real-time track fusion is, at its core, a stateful problem. A radar return, a SIGINT intercept, or a UAV detection arriving on the wire means nothing in isolation – its value comes entirely from being matched against the accumulated history of every object the system already believes exists. Deciding whether an observation extends an existing track or creates a new one requires the engine to remember everything it has seen. That single requirement – durable, mutable, per-track memory updated at the rate sensors produce data – is what separates streaming track processing from the simple stateless transforms most data pipelines are built from. This article examines how stateful stream processing is engineered for real-time fusion: the state model, event-time windowing, exactly-once semantics, partitioning for scale, and the operational failure modes that bring these pipelines down.

Why track fusion is inherently stateful

A stateless pipeline applies a pure function to each event and forgets it. That model works for enrichment, format conversion, or filtering, but it cannot fuse tracks. The fusion question – "does this observation belong to a track I am already maintaining?" – is answerable only with reference to state: the current position and velocity estimate of every active track, its uncertainty (covariance), its last-update time, and its association history.

Concretely, the engine maintains one record per track, keyed by a stable track ID. That record holds the Kalman (or particle) filter state, the timestamp of the last observation applied, a short history of contributing sensor reports, and the track's classification and confidence. When a new observation arrives, the engine reads the candidate track states, decides which track (if any) the observation belongs to using a track correlation algorithm, mutates the matched track's filter state in place, and emits an update. The state store is read and written on the hot path of every single observation – which is why its design dominates the performance and reliability of the whole system.

The state store: where track memory lives

In a production streaming engine, track state is not held in plain application memory. It lives in a managed, fault-tolerant state store that the framework can checkpoint and restore. Apache Flink backs keyed state with an embedded RocksDB instance per task; Kafka Streams materializes state into local RocksDB stores fronted by a compacted changelog topic that allows full reconstruction after a crash. Either way, the contract is the same: the engine gives you a keyed map – track ID to track state – that is local, fast, and durable across restarts.

The key design decisions for the state store are size and access pattern. Track state must be compact: a filter mean vector, a covariance matrix, a handful of metadata fields, and a bounded ring buffer of recent observation references – not the full observation history. Unbounded per-track history is the single most common cause of state blowup. Access is overwhelmingly read-modify-write on a single key per observation, so the store is tuned for point lookups and in-place updates rather than scans. Range scans, when needed for spatial gating, are kept out of the hot path by maintaining a secondary spatial index updated asynchronously.

State serialization deserves explicit attention. Because the framework writes and reads track state to disk on every checkpoint and recovery, the serializer for the track-state class is a hot-path component, not a detail. A reflective, schema-on-read serializer that walks object graphs per record will dominate CPU at high observation rates; a hand-written or code-generated serializer that lays out the fixed-size filter state as a flat byte buffer is often an order of magnitude faster. The same discipline pays off in checkpoint size – compact, fixed-width state encodings shrink snapshots and shorten the recovery window after a node failure.

Bounding state growth

Every track that is created but never expired occupies state forever. A pipeline without disciplined expiry will see its state store grow monotonically until checkpoints slow down and the engine falls behind real time. Three expiry mechanisms work together: time-to-live (drop a track that has received no observation within N seconds), miss-count limits (drop a track that has been predicted-but-not-updated through M consecutive expected windows), and area-of-interest culling (drop tracks that leave the operational region). Expiry is not housekeeping that can be deferred – it is a correctness and stability requirement, and it must run on the same event-time clock as the rest of the pipeline so it behaves identically during live processing and replay.

Event time, watermarks, and windowing

Sensor feeds do not arrive in order, and they do not arrive on time. A radar plot observed at 09:47:03.120 may reach the fusion engine 400 ms later than a SIGINT intercept of the same object observed at 09:47:03.080, simply because the two feeds traverse different networks and processing stages. If the engine correlated by the time events arrived (processing time), it would routinely fail to associate observations that genuinely describe the same object at the same instant.

Streaming fusion therefore keys on event time – the timestamp at which the sensor observed the object – and uses watermarks to reason about completeness. A watermark is the engine's estimate that no further events with an event time earlier than the watermark will arrive. Correlation windows close when the watermark passes their end, plus a configured allowed-lateness grace period that holds the window open just long enough for stragglers. Observations later than the grace period are not silently dropped; they are routed to a side output so analysts can audit how much data missed its window and tune the grace period accordingly.

Choosing the grace period is a direct latency-versus-completeness trade. A longer grace period catches more late reports and produces more complete correlation, but every track update inherits that delay before it reaches the operator. For tactical ground tracks a grace period of a few seconds is typical; for air tracks where sub-second latency is mandatory, the grace period shrinks to tens or low hundreds of milliseconds, accepting that some late reports will be handled as track corrections rather than in-window correlations.

Exactly-once semantics for track integrity

In a fusion pipeline, delivery semantics are not an academic concern – they determine whether the operational picture is correct. Consider at-least-once delivery, where a failure can cause an observation to be replayed. If the same radar return is applied to a Kalman filter twice, the filter treats it as two independent measurements and becomes artificially confident, shrinking its covariance and biasing the estimate toward a single noisy reading. The track looks more certain while being more wrong – the worst possible failure for a system commanders act on.

Exactly-once semantics eliminate this by guaranteeing each observation affects track state precisely one time, even across crashes and restarts. The mechanism is atomic checkpointing: the engine periodically snapshots the state store and the input offsets it has consumed, committing both together. On recovery it restores the snapshot and resumes consuming from the committed offsets, so observations that were already folded into state are never re-applied. Flink implements this with its distributed checkpoint barriers; Kafka Streams uses transactional writes that tie state-store changelog updates and output-topic offsets into a single transaction.

Exactly-once is not free. Checkpoints must complete faster than the interval between them, or the pipeline accumulates un-checkpointed state and eventually stalls. Checkpoint duration scales with state size – which is the second reason aggressive track expiry matters. A pipeline that holds 50,000 stale tracks checkpoints slowly; the same pipeline holding only the few thousand genuinely active tracks checkpoints in milliseconds. Bounding state is what keeps exactly-once affordable at operational tempo. For pipelines that also need a replayable, tamper-evident record of every state change, the checkpointed log pairs naturally with an event-sourced audit trail.

Partitioning and scaling stateful operators

A single task cannot fuse the entire battlespace at high sensor rates, so the stream is partitioned and processed in parallel. The defining constraint of stateful fusion is that two observations of the same physical object must be routed to the same partition – otherwise they land in separate state stores, never meet, and the object spawns parallel tracks that no operator can reconcile.

The partition key must therefore preserve correlation locality. Keying by sensor ID fails immediately, because the whole point is to correlate across sensors. Keying by a coarse geographic cell works well: all observations within a region land on one task that holds the state for objects in that region. The challenge is boundary handling – an object crossing a cell boundary must be handed off between partitions without dropping or duplicating its track. Practical systems use overlapping cells or a dedicated boundary-reconciliation stage to manage handoff, and they size cells so that no single cell becomes a hotspot that overwhelms one task while others idle.

Because state is local to a partition, rescaling a stateful pipeline is not as simple as adding workers. The engine must redistribute keyed state when it rebalances partitions across nodes – Flink does this by reading checkpointed state and reassigning key groups; Kafka Streams replays changelog topics to rebuild local stores on the new instance. Both are bounded by state size, which is, once again, why a disciplined state budget underpins every other property of the system. The same partitioned-log backbone that carries observations into the fusion engine is described in more depth in our note on message queue architecture for defense data pipelines.

From track deltas to the operational picture

The fusion engine should publish change, not state. Each time a track is created, updated, or dropped, the engine emits a delta event onto a downstream topic that the common operating picture and other consumers subscribe to. Publishing deltas rather than full-state snapshots keeps the display responsive even when the active track count reaches tens of thousands, because the consumer applies small incremental changes instead of re-rendering the world on every tick.

Each delta carries a monotonically increasing version per track so consumers can detect and correct out-of-order delivery – applying an older update after a newer one would teleport a track backward. The event schema is the formal contract between fusion and its consumers; freezing and versioning it lets the COP, analytic tools, and archives evolve independently of the fusion core. End to end, a well-tuned stateful pipeline holds sensor-observation-to-COP latency in the single-digit seconds for ground tracks and under a second for air tracks, with the dominant variable being the watermark grace period rather than the compute inside the operator.

One operational property is worth stating plainly: a stateful streaming engine is only as trustworthy as its replay behavior. Because association decisions and expiry both run on the event-time clock, a recorded sensor session can be fed back through the identical pipeline to reproduce the exact track picture an operator saw – provided every operator (association, windowing, expiry) is deterministic given its state and input. That determinism is what makes the system testable and accreditable: an engineer can change an association threshold, replay a known scenario, and compare the resulting tracks against a baseline rather than waiting for the behavior to recur in the field. Treat non-determinism – wall-clock reads, unordered map iteration, floating-point reductions whose order depends on thread scheduling – as defects, because each one breaks replay and, with it, the ability to verify the fusion engine behaves the same way twice.