Fusing intelligence from SIGINT, IMINT, HUMINT, and OSINT into a single coherent picture sounds straightforward in concept. In practice, it is one of the hardest engineering problems in defense data infrastructure. Each discipline produces data at a different velocity, in a different format, with a different confidence model — and a system that merely aggregates these feeds without truly correlating them produces a noisier picture than a well-managed single-INT feed, not a better one. Serious data integration for defense intelligence requires an architecture designed from first principles around the properties of each source type, not a generic data warehouse with INT adapters bolted on.
This article walks through the full architecture of a real-time multi-source fusion system — from the stream ingestion layer that absorbs heterogeneous feeds, through entity resolution and the correlation engine, to confidence scoring and the publication of an enriched intelligence picture to C2 systems. Each section addresses the specific design decisions that separate production-grade fusion from prototype systems that look correct in a demo but fail at operational scale.
The multi-source intelligence fusion problem — why fusing SIGINT/IMINT/HUMINT/OSINT is harder than it looks
Three structural problems make multi-source fusion qualitatively harder than processing any individual INT at scale. Ignoring them — treating fusion as just a data pipeline problem — is the most common architectural mistake, and it produces a system that cannot be corrected once data is flowing at production volume.
Format heterogeneity is the most visible problem. An ELINT intercept record carries emitter frequency, pulse repetition interval, scan type, bearing, and time-difference-of-arrival data. An IMINT detection carries a bounding box, a pixel-space confidence score, a ground sample distance, and a UTM coordinate with associated CEP. A HUMINT narrative extract carries free text, an analyst's source grade, a named location, and a temporal window measured in hours or days. These three records may all refer to the same real-world object — a radar vehicle — but they share no common field, no common coordinate system convention, and no common identifier. Normalizing them into a single canonical schema requires purpose-built adapters for each source type and a canonical entity model that can accommodate all of them without collapsing their semantics. The multi-INT sensor fusion military data model design covers that canonical layer in detail.
Confidence mismatch is subtler but causes more analytical errors. Each INT applies its own confidence model. SIGINT geolocation carries a CEP (Circular Error Probable) in meters — a geometric accuracy estimate. IMINT detection carries a classifier probability — the neural network's output logit for the object class. HUMINT carries an admiralty reliability grade — a six-point human judgment scale. These three numbers mean fundamentally different things. A naive fusion system that averages them produces a number that is mathematically valid but analytically meaningless. A correct confidence model must first translate each source-native confidence into a common probabilistic framework, then aggregate across sources using a Bayesian or evidence-combination scheme (such as Dempster-Shafer) that correctly handles dependent and independent evidence.
Temporal skew is the problem that breaks real-time fusion architectures most visibly. Different sources have vastly different latencies from event occurrence to data arrival in the pipeline. A SIGINT intercept may arrive in the pipeline within seconds of the transmission. IMINT from a tasked satellite may require 60–90 minutes of imagery processing before a detection record is produced. A HUMINT report describing yesterday's events may be ingested on day D+2. A fusion engine that correlates records by pipeline arrival timestamp will systematically mis-associate events — it will fail to link the SIGINT intercept and the IMINT confirmation because they arrived 90 minutes apart, even though they describe the same event. The architecture must separate event time (when the event occurred in the world) from ingestion time (when the message arrived in the pipeline) and perform all temporal correlation on event time, using ingestion time only for pipeline performance monitoring.
Stream ingestion architecture — Kafka vs Pulsar for intelligence feeds, schema registry for heterogeneous message types, backpressure handling
The stream ingestion layer must handle messages arriving at rates ranging from a few HUMINT reports per hour to millions of SIGINT records per hour from a single collection system, all flowing into the same fusion pipeline. The design must absorb burst traffic from high-velocity sources without applying that backpressure to low-velocity, high-priority sources — a HUMINT report that arrives in a burst alongside a SIGINT flood must not be delayed because the pipeline is saturated.
The primary architectural choice at this layer is between Apache Kafka and Apache Pulsar. Both are distributed, fault-tolerant, and battle-tested at the scale of defense-grade systems. Kafka has a larger ecosystem and tighter integration with stream processing frameworks — Kafka Streams and Apache Flink both have first-class Kafka source connectors. Pulsar offers native multi-tenancy, per-namespace topic isolation, and a separated compute-storage architecture. For a single-tenant fusion platform operating within one classification domain, Kafka is the pragmatic choice. For a platform that must isolate feeds by classification level, support disconnected forward-deployed nodes that sync asynchronously to a rear-echelon cluster, or serve multiple tenant organizations, Pulsar's architecture provides cleaner isolation primitives without ad-hoc workarounds.
Regardless of broker choice, the topic structure must enforce one topic per source type, not one topic per INT discipline:
A schema registry is mandatory for the normalized output topics. Without enforced schemas, a single malformed message from any adapter can corrupt a downstream consumer's deserialization context and stall the entire fusion pipeline. The registry enforces Avro or Protobuf schemas on every write to fusion topics and rejects messages that fail validation before they reach the broker. Schema compatibility rules — BACKWARD, FORWARD, or FULL — must be set per topic based on how consumers evolve. For the normalized observation schema, BACKWARD compatibility is appropriate: new schema versions can add optional fields, and old consumers that do not understand new fields will continue to function correctly.
Backpressure handling requires per-source-type rate limits with priority classification. High-priority sources — SIGINT and IMINT from live collection systems — must never be rate-limited. Low-priority sources — bulk OSINT ingestion, historical HUMINT loading — should carry explicit priority weights that allow the broker to throttle their consumers when pipeline throughput is constrained. In Kafka, this is implemented via consumer group quotas and partition assignment strategies. In Pulsar, per-namespace rate limits and dispatch quotas provide equivalent control. The intelligence data pipeline military architecture patterns cover queue management and consumer group design in depth.
Entity resolution and deduplication — blocking strategies for entity candidate generation, ML-based similarity scoring, graph-based entity merging
Entity resolution is the component that transforms a stream of independent observations — each describing a detected object or reported event — into a set of unified entity records where all observations about the same real-world object have been merged. It is the hardest component to get right and the one most likely to degrade silently: a system with poor entity resolution does not crash; it produces a picture with too many entities (fragmented tracks) or too few (false merges), and these errors compound over time as fusion builds on corrupted entity state.
The naive approach — compare every incoming observation against every existing entity — is computationally infeasible at any realistic scale. An ELINT collection system producing 10,000 intercept records per minute against a corpus of 50,000 active entity tracks would require 500 million comparisons per minute before any similarity scoring. Blocking strategies reduce this to a tractable candidate set by exploiting the structure of intelligence data:
- Geo-cell blocking — index every entity by its estimated location using an H3 hexagonal grid at resolution level 9 (approximately 0.1 km² cells). An incoming observation generates candidate pairs only against entities in the same cell and its six immediate neighbors. This single blocking key eliminates over 99.9% of candidate pairs for spatial data while achieving high recall for genuinely co-located observations.
- Temporal blocking — restrict candidates to entities whose valid-time interval overlaps a configurable window around the observation's event time. For a moving vehicle, the window must account for maximum plausible speed between last observation and current observation location — a tank cannot cover 50 km in 5 minutes. Temporal blocking without geo-cell blocking produces enormous candidate sets for any globally active entity; the two work together as a compound key.
- Hard-identifier blocking — group all records sharing a deterministic identifier (IMSI, license plate OCR output, biometric hash, aircraft Mode-S transponder code) into a single candidate cluster regardless of location or time. Hard identifiers bypass all other blocking and are treated as definitive merge signals pending a deduplication check on the identifier itself.
- INT-type affinity blocking — certain cross-INT pairs are structurally unlikely to match and should not be scored. A maritime SIGINT emission observation should not be scored against a land vehicle IMINT track. Affinity rules, defined per entity type, eliminate structurally implausible candidate pairs before scoring.
Similarity scoring of candidate pairs uses a gradient boosting model (XGBoost or LightGBM) trained on labeled match/non-match pairs from historical intelligence data. Feature vectors include: spatial distance normalized by positional uncertainty, temporal distance normalized by entity type velocity envelope, attribute overlap score for typed attributes (entity class, affiliation, equipment type), and INT-type compatibility weight. The model outputs a match probability in [0, 1]. Pairs above a high threshold (typically 0.85–0.92) are automatically merged; pairs in a middle band are routed to analyst review; pairs below a low threshold are treated as separate entities.
Graph-based entity merging handles the multi-way case — where three or more observations must all be resolved to the same entity. A graph is maintained where nodes are observation records and edges are merge decisions with associated confidence scores. Connected components of the graph represent candidate entity clusters. The golden record for each cluster is computed by applying the defined merge policy to all contributing observations. When a new high-confidence edge is added that connects two previously disjoint components, the components are merged and their golden records are combined. The graph representation also enables split operations: removing a high-confidence edge and recomputing connected components is the mechanism for correcting a false merge.
Correlation engine design — window-based correlation (temporal proximity), spatial correlation (geo-cell bucketing), semantic correlation (entity attribute matching)
Where entity resolution asks "are these observations about the same object?", the correlation engine asks "do these events involving different objects form a meaningful pattern?". A vehicle detected near a logistics depot, followed 20 minutes later by the same vehicle at a fuel point, followed by a SIGINT intercept of a radio call consistent with a unit position report from the fuel point — these three events involve the same vehicle but also describe a situational pattern (resupply activity) that is analytically more significant than any single observation. The correlation engine produces that higher-order intelligence object.
Correlation operates across three planes simultaneously:
Temporal correlation groups events into windows by their event timestamp and looks for co-occurrence patterns within each window. The window definition is configurable per rule: a rule detecting simultaneous jamming onset and friendly communication disruption may use a 30-second window; a rule detecting a vehicle at a way-point followed by another vehicle at the same way-point may use a 2-hour window. Sliding windows (overlapping by half the window duration) ensure that events at window boundaries are not missed. Apache Flink's event-time windowing with watermarks is the standard implementation choice because it handles out-of-order event arrival correctly — critical for a pipeline where IMINT and HUMINT have very different ingestion latencies.
Spatial correlation groups entities by their estimated location using geo-cell bucketing at a resolution calibrated to the correlation rule's spatial scale. H3 resolution 8 (approximately 0.7 km²) is appropriate for detecting vehicles operating in the same zone; H3 resolution 6 (approximately 36 km²) for detecting activity patterns across a region. Unlike entity resolution geo-cell blocking (which needs high recall), correlation spatial bucketing needs rule-defined precision: a correlation rule detecting co-location of a command vehicle with a logistics vehicle should use a tighter cell than a rule detecting area-wide supply activity. Rules encode their own spatial resolution parameter rather than inheriting a global setting.
Semantic correlation matches entities based on attribute values rather than spatial or temporal proximity. An entity classified as an air-defense radar that begins emitting, correlated with an entity classified as a combat aircraft entering the radar's coverage zone — the correlation is driven by attribute semantics (radar type vs aircraft type and trajectory), not by their being co-located. Semantic rules encode attribute predicates over entity types: entity.type == EQUIPMENT AND entity.attributes.role == AIR_DEFENSE AND entity.activity_state == EMITTING. These predicate expressions are evaluated by the stream processor against each entity update event; when all predicates in a rule fire within the window, the correlation fires.
Correlation rules are stored as declarative definitions in a rule registry, not as hardcoded logic. This enables intelligence analysts to author new rules without engineering involvement, and allows rule changes to be applied without redeploying the correlation engine:
Each fired correlation produces a Situation entity — a new intelligence object at the JDL Level 2 layer that references the contributing entity UUIDs, carries the rule ID and rule confidence weight, and is published to the enriched entity stream alongside the standard entity updates.
Confidence scoring and provenance — source reliability × information credibility matrix (NATO admiralty system), confidence propagation across fused entities
Every entity in the fused picture carries a confidence score, but confidence in a multi-source fusion system is not a single number — it is a composite that must be decomposed along several dimensions to be analytically useful. A score of 0.73 that was produced by three corroborating SIGINT records means something very different from a score of 0.73 that came from one unreliable HUMINT report and nothing else. The architecture must preserve the decomposition, not just the aggregate.
The confidence model layers three inputs. First, source reliability weight derived from the NATO admiralty source reliability grade: A=1.0, B=0.8, C=0.6, D=0.4, E=0.2, F=0.1. This weight scales the contribution of each source observation to the fusion computation. Second, information credibility weight derived from the admiralty credibility grade: 1=1.0, 2=0.8, 3=0.6, 4=0.4, 5=0.2, 6=0.1. This weight is applied per-observation independently of source reliability, because a reliable source can report uncertain information. Third, inter-source agreement score — the fraction of independently corroborating sources that agree on the key attribute value, weighted by their source reliability. A position value corroborated by three A-rated sources with B2 information grades scores very differently from one corroborated by two E-rated sources with D4 grades.
The composite confidence for an attribute is:
Confidence propagation across derived entities follows a dilution rule: a Situation entity derived by the correlation engine from two contributing entity golden records cannot have higher confidence than the lower-confidence contributing entity. Confidence can only increase when additional independent corroborating observations are added to the contributing entities. This prevents the correlation engine from artificially inflating confidence by linking low-quality entities.
Provenance links are stored as a chain: every golden record references the observation UUIDs that contributed to it; every observation references its source ID, collection system, and admiralty grades. The chain is immutable — it is never updated in place. When a contributing source is later assessed as unreliable or deceptive, the chain provides the complete set of entities that must be re-evaluated with that source's contributions excluded. Source retraction is a first-class operation, not an emergency procedure.
Intelligence picture publication — enriched entity publication to C2 systems via CoT/NIEM, latency targets for time-sensitive intelligence
The enriched entity stream is the internal representation. Publication converts that representation into the wire formats that C2 systems and downstream consumers actually understand. For most defense deployments, this means Cursor on Target (CoT) for track data and NIEM-based message formats for structured intelligence reports.
CoT publication maps each enriched entity update to a CoT event. The mapping must handle three update types: position updates (new coordinates with updated uncertainty), attribute updates (confidence change, activity state change, affiliation assessment change), and entity lifecycle events (entity creation, retirement, and split/merge events). Position updates are straightforward — they translate to a CoT point event with the entity's estimated position and the position confidence encoded in the CoT error field. Attribute updates translate to CoT remarks or detail element updates on the existing entity. Lifecycle events — particularly merge and split — require special handling: a merge event retires the lower-confidence entity's CoT track and updates the surviving entity's CoT detail with provenance information; a split event creates a new CoT track for the newly separated entity.
The CoT stale time field is the natural mechanism for TTL management at the C2 receiver. Each published CoT event sets stale_time to entity_publication_time + entity_TTL. The receiving C2 system (ATAK, CURSOR, or similar) removes the entity from its display when stale_time passes without a refresh. The fusion system must ensure that entities with active observations receive CoT refresh events at an interval shorter than their TTL — typically at 50% of TTL — to prevent valid entities from disappearing from the picture during normal operation.
Latency targets determine how the publication layer must be implemented. For time-sensitive targeting workflows, end-to-end latency from sensor event to C2 display must be under 30 seconds. Achieving this requires: async, non-blocking publication workers; parallelized CoT serialization across entity partitions; prioritized output queues that give precedence to high-confidence moving tracks over static entity updates; and direct TCP multicast delivery to C2 recipients rather than REST API polling. For general situational awareness at the operational level, 60–120 second latency is typically acceptable, enabling simpler synchronous batch publication patterns.
NIEM publication serves intelligence report consumers — systems that need structured, machine-readable intelligence assessments rather than track data. A Situation entity output by the correlation engine is published as a NIEM Intelligence Message with the contributing entity UUIDs as references, the situation class as the subject, and the confidence and provenance data in the NIEM message metadata block. NIEM consumers include multi-domain intelligence sharing platforms, interoperability gateways to partner nation systems, and archival systems that require structured provenance for audit compliance.
Handling temporal uncertainty and stale intelligence — TTL management for intelligence entities, decay curves for position/activity confidence, re-evaluation triggers
Intelligence is perishable. A position report for a vehicle track that was accurate 20 minutes ago is less accurate now — the vehicle may have moved. An activity assessment based on a single SIGINT intercept 4 hours ago may no longer reflect current behavior. A fusion system that does not model this degradation and propagate it to the published picture will cause analysts to make decisions based on stale information with high confidence scores that were earned by older observations.
TTL management begins with TTL assignment at entity creation. TTL values are entity-type and confidence-tier specific:
- High-confidence moving vehicle track (confidence ≥ 0.80, mobility class = WHEELED/TRACKED): TTL 8–15 minutes
- Medium-confidence moving track (confidence 0.50–0.79): TTL 4–8 minutes
- Low-confidence tentative track (confidence < 0.50): TTL 2–4 minutes
- High-confidence fixed facility (confidence ≥ 0.80, mobility class = STATIC): TTL 24–72 hours
- Personnel entity (activity-based, not position-based): TTL 30–120 minutes depending on last activity type
- Situation entity (derived correlation): TTL equal to the shortest TTL of any contributing entity, capped at 60 minutes
Confidence decay curves model how position and activity confidence degrade between confirming observations. Rather than a binary active/stale state, the decay curve provides a continuous confidence function over time since last observation:
Re-evaluation triggers fire when a new observation arrives that is spatially and temporally consistent with a stale entity. Rather than creating a new entity for the new observation, the trigger routes it through the entity resolution pipeline against the stale entity as a candidate. If the match confidence exceeds the merge threshold, the new observation refreshes the stale entity: the TTL clock resets, the confidence is recomputed from the new observation (possibly upward from the decayed value, if the new observation has strong confidence), and the entity transitions back from STALE to ACTIVE status. The refresh is logged as an update event in the entity's audit trail.
The decay model must also be communicated to C2 consumers, not just applied internally. Publishing a high-confidence position to CoT without signaling that the position is 18 minutes old causes the C2 display to show a sharp, confident track icon that the analyst may interpret as a real-time observation. Two mechanisms address this: the CoT stale_time field (already discussed) ensures the entity disappears from display when it is definitively stale; and an age-since-last-observation field published in the CoT detail element gives the analyst real-time visibility into how old the underlying observation is, even for entities that have not yet reached their stale time. Some C2 systems use this field to modulate the visual confidence representation — shifting track icon opacity or size as a function of observation age.
Together, the seven architectural components described in this article — heterogeneous stream ingestion, schema-enforced normalization, blocking-plus-ML entity resolution, multi-plane correlation engine, layered confidence scoring, CoT/NIEM publication, and TTL decay management — constitute a complete real-time multi-source intelligence fusion platform. None of these components is independently novel. What makes the architecture production-grade is the explicit design of the interfaces between them: the canonical observation schema that normalization produces and entity resolution consumes; the entity graph format that entity resolution produces and the correlation engine consumes; the enriched entity stream that both entity resolution and the correlation engine write and the publication layer reads. A system where these interfaces are implicit or undocumented — where each component assumes specific behavior from its upstream — is fragile at exactly the points where intelligence data is most variable in character.