Satellite imagery ingestion pipeline for defense: from raw scene to tasked analysis

A satellite scene is not intelligence. It is raw data: a compressed raster of pixel values encoded in a proprietary format, tagged with a coordinate reference system, attached to a sensor telemetry file, and wrapped in a classification container that determines who can touch it. Between that raw delivery and a targeting analyst receiving a usable orthorectified image on their workstation lies an ingestion pipeline -- a series of automated processing stages that most GEOINT programs underestimate until it breaks at 0200 on an urgent collection requirement. This article dissects each stage of a defense satellite imagery ingestion pipeline: how scene tasking interfaces with satellite operator APIs and national technical means (NTM) request systems, what the preprocessing stack does to raw imagery before it enters the catalog, why format choices matter operationally, how spatial catalogs enable fast scene retrieval across millions of archived scenes, and how routing logic connects newly ingested imagery to the exploitation tools and analyst queues that need it.

The satellite imagery pipeline: scope and operational requirements

A defense imagery ingestion pipeline spans three functional domains: acquisition (getting the scene from the satellite or provider into the pipeline), processing (converting the raw scene into a calibrated, georeferenced, cataloged product), and exploitation routing (delivering the right scene to the right tool or analyst at the right time). Each domain has distinct latency, throughput, and reliability requirements that shape architectural choices. For routine collection supporting long-cycle intelligence production, end-to-end latency from scene acquisition to catalog availability of several hours is acceptable. For time-sensitive intelligence (TSI) requirements -- battle damage assessment, force tracking, or dynamic targeting -- the same pipeline must compress that latency to under 30 minutes, and in some architectures, under 10.

Operational requirements impose constraints that commercial imagery pipelines do not face. Classification handling means that scenes at different security levels cannot share processing infrastructure without accreditation, or must be processed in isolated enclaves with strict data transfer controls between levels. Chain-of-custody logging -- recording who ordered a collection, which processing algorithms were applied, which analyst received the product, and what exploitation actions were taken -- is mandatory for finished intelligence with legal and operational accountability. Availability requirements for imagery pipelines that support active operations are typically higher than for peacetime production systems, requiring redundant processing nodes, automated failover, and degraded-mode operation plans for the case where the primary commercial downlink path or cloud processing environment is unavailable.

The pipeline must also be multi-source by design. A defense theater does not rely on a single satellite constellation. Commercial providers (Maxar WorldView Legion, Planet SuperDoves, Airbus Pleiades Neo, Satellogic), synthetic aperture radar (SAR) providers (Capella Space, ICEYE, Umbra), and coalition or NTM imagery all arrive through different delivery mechanisms with different format conventions, metadata schemas, and licensing constraints. The ingestion pipeline abstracts these differences behind a common internal schema so that downstream processing, cataloging, and exploitation tools operate on a normalized representation regardless of source.

Scene ordering and tasking: integrating with satellite operator APIs and NTM request systems

Satellite tasking begins with a requirement: a geographic area of interest (AOI), a collection window, a resolution requirement, and an intelligence priority. In a defense organization, requirements are formalized as standing requirements (STANREQs) or ad-hoc task orders managed in a requirements tracking system. The ingestion pipeline's tasking module reads active requirements and translates them into collection orders submitted to each relevant satellite operator or NTM broker. For commercial providers, this means calling a REST tasking API: submitting an AOI polygon, a collection window, a product level specification, and authentication credentials. The Maxar SecureWatch API, Planet Orders API, and Airbus Intelligence Access API all follow broadly similar patterns -- POST an order, poll for status, and download the scene package from a signed URL when collection is confirmed.

NTM integration follows a different pattern governed by classified request protocols. Rather than a commercial REST API, NTM requests flow through controlled dissemination systems using message formats such as STANAG 4559 (NATO standard for imagery request and delivery) or US IC-specific protocols. The ingestion pipeline's NTM interface module handles authentication against the relevant brokering system, submits requests in the required schema, monitors for delivery notifications, and retrieves scene packages through the authorized transfer path. The key architectural principle is that NTM and commercial tasking must be handled by separate interface modules with isolated credential stores, even if they feed into the same downstream processing queue, because their classification, handling, and audit requirements differ.

Order management requires a local state machine to track the lifecycle of each collection request: submitted, acknowledged by the provider, collected (satellite pass occurred), downlinked, processed by the provider, and delivered to the pipeline's ingest endpoint. Provider-side processing failures, cloud cover at collection time, and satellite tasking conflicts all require handling logic -- rescheduling, escalation to a higher-priority alternative provider, or flagging the requirement as unfilled for manual review. The tasking module should maintain a historical record of all requests and outcomes to support collection effectiveness reporting and provider performance analysis.

Raw image preprocessing: orthorectification, atmospheric correction, and cloud masking

A scene delivered by a commercial provider at Level 1B (radiometrically corrected, sensor geometry) is not ready for exploitation or cataloging in a defense imagery system. Before it enters the spatial catalog, it must be orthorectified -- geometrically corrected to remove sensor attitude errors and terrain displacement -- and radiometrically normalized to a consistent surface reflectance scale. These steps are not optional refinements; they are prerequisites for overlaying imagery against vector maps, performing change detection against prior collections, and measuring objects accurately enough for military intelligence purposes.

Orthorectification uses the rational polynomial coefficients (RPCs) bundled with the scene and a digital elevation model (DEM) to project each pixel from sensor geometry into a map projection. SRTM 1-arc-second (approximately 30 m horizontal resolution) is the baseline DEM for most theater-level pipelines; for high-resolution collections (0.3--0.5 m ground sample distance) where sub-meter geolocation accuracy matters, a theater-specific high-resolution DEM derived from stereo satellite collections or airborne lidar is required. The RPC-based model achieves 3--8 m circular error probable (CEP) without ground control; adding a sparse set of GPS-surveyed ground control points (GCPs) to refine the RPC solution improves accuracy to 1--2 m CEP for post-processed products. For missions where absolute geolocation accuracy is critical -- target coordinate mensuration, for instance -- the pipeline must integrate a GCP database and apply the RPC refinement step automatically.

Atmospheric correction converts top-of-atmosphere (TOA) radiance to surface reflectance, removing the effects of molecular scattering, aerosol absorption, and solar illumination geometry. This step is essential for multispectral change detection: two scenes collected under different atmospheric conditions will show apparent radiometric differences in every band even if the ground surface has not changed, producing false alarms. Radiative transfer models such as MODTRAN or 6S compute correction coefficients given atmospheric parameters (aerosol optical depth, water vapor, ozone column) obtained from coincident MODIS retrievals or model analysis fields. Cloud and cloud-shadow masking uses a quality assessment algorithm (FMask, S2cloudless, or a trained CNN) to label each pixel as clear, cloud, shadow, or snow/ice. The cloud mask is stored as a companion band alongside the processed scene and propagates through all downstream processing -- change detection algorithms, for example, must exclude cloud-masked pixels from their statistics.

Format landscape: NITF, GeoTIFF, JPEG 2000, and their defense use cases

Defense imagery pipelines must manage multiple coexisting formats because no single format satisfies all use cases within a defense organization. NITF 2.1 (National Imagery Transmission Format) is the authoritative container for intelligence imagery in US and allied systems. It carries the image data alongside structured metadata fields that no other format natively supports: security classification and handling markings in the file header, PIAIMC (Profile for Imagery Access) technical extension records describing sensor parameters and collection geometry, SENSRB (Sensor Data Records) for precise sensor telemetry, and IGEOLO corner coordinates and map projection information. NITF's structure also permits multiple image segments within a single file, allowing a panchromatic band, multispectral stack, and pan-sharpened product to coexist in one container with a shared metadata header.

GeoTIFF -- and specifically Cloud-Optimized GeoTIFF (COG) -- is the workhorse format for web-based visualization, GEOINT platform visualization layers, and AI/ML processing workflows. COG files organize internal tile and overview structure so that HTTP range requests can fetch only the portion of the image visible at the current map zoom level, enabling a web map service to stream imagery from object storage without pre-generating tile pyramids. For AI model inference -- change detection, object detection, feature extraction -- GeoTIFF with GDAL-readable metadata is the standard input format for Python-based geospatial ML frameworks. The pipeline generates COG derivatives from the NITF master as a parallel output step, writing them to the object storage layer accessible to web services and ML inference nodes.

JPEG 2000 occupies a specific niche in defense imagery: it is the compression format embedded within NITF files for high-resolution products where lossless or visually lossless compression at 4:1 to 8:1 ratios is required, and it is the format used in many legacy alliance imagery exchange standards. JPEG 2000's wavelet-based compression outperforms JPEG at high compression ratios while preserving fine-detail features critical for exploitation (vehicle identification, facility analysis, activity pattern recognition). The pipeline should be able to read and write JPEG 2000 streams both as standalone files and as image segment data within NITF containers, using a compliant library such as OpenJPEG or Kakadu. For defense data fusion pipelines handling multi-source imagery, normalizing all sources to a consistent internal format before catalog indexing eliminates format-specific handling in downstream tools.

Catalog indexing: spatial and temporal indexing for fast scene retrieval

The spatial catalog is the operational memory of the imagery pipeline. Every processed scene must be indexed before it is useful: an orthorectified NITF sitting in object storage that no catalog knows about is effectively invisible to analysts and exploitation tools. The SpatioTemporal Asset Catalog (STAC) specification has become the standard schema for defense and commercial imagery catalogs because it defines a common JSON structure for scene metadata -- footprint geometry, acquisition datetime, sensor and platform identifiers, band descriptions, asset links -- that is readable by a growing ecosystem of catalog clients, search APIs, and visualization tools without custom integration work.

Under the STAC API, a PostGIS-backed PostgreSQL database stores Item records and their GeoJSON footprint geometries. Spatial queries -- "all scenes intersecting this polygon, collected in the last 14 days, with less than 15% cloud cover, at 0.5 m or better resolution" -- execute as PostGIS spatial intersection queries with composite indexes on the footprint geometry column, the acquisition datetime column, and the cloud cover and resolution numeric fields. For a catalog holding 10 million scene records, this query structure returns results in under 500 ms if indexes are maintained and query plans are optimized. The pipeline's indexing step inserts each new STAC Item record immediately after the NITF is written and its metadata is validated, so the scene is queryable within seconds of processing completion.

Temporal indexing matters as much as spatial indexing for change detection workflows. Analysts and automated services frequently query for "all prior collections of this AOI" to establish baseline imagery for change detection or activity pattern analysis. An index on the acquisition datetime column with a B-tree structure supports range queries (all collections between date A and date B) efficiently, but the most useful temporal access pattern -- "all scenes intersecting footprint X, ordered by acquisition date" -- requires a joint spatial-temporal query that benefits from a covering index combining the geometry and datetime columns. The same spatial indexing principles used in sensor data normalization pipelines apply here: the schema must be designed for the query patterns the exploitation tools actually issue, not for schema normalization alone.

Routing to exploitation: queuing imagery to appropriate analytic tools and analyst workstations

A newly cataloged scene is a candidate for delivery to multiple downstream consumers simultaneously: an automated change detection service, an AI-based object detection model, a human imagery analyst, and a report generation system. The routing engine is the component that matches each new scene against registered requirements and determines which consumers receive it, in what priority order, and via what delivery mechanism. The routing model used in most defense imagery systems is based on named area of interest (NAI) subscriptions combined with standing requirements (STANREQs) that specify filter criteria -- minimum resolution, maximum cloud cover, collection date window, sensor type -- and a destination system or analyst queue.

When the indexing step writes a new STAC Item, the routing engine evaluates it against all active subscriptions. Subscriptions are implemented as spatial queries against the NAI polygon library: if the scene footprint intersects a registered NAI, the engine applies the subscription's filter criteria. A scene that passes all criteria generates a delivery notification to the designated destination. For AI exploitation services, the notification carries the scene's NITF storage URI and is published to a work queue (RabbitMQ, AWS SQS, or an equivalent message broker) consumed by the service's worker processes. For analyst workstations, the notification updates the analyst's task queue in the imagery exploitation system (SOCET GXP, RemoteView, or FADE/MIST) with a new task record pointing to the scene. For time-critical intelligence requirements, the routing engine applies a priority boost that pre-empts lower-priority items already in the exploitation queue.

Cross-classification routing requires particular care. A scene collected at a higher classification level than the analyst's baseline accreditation cannot be routed to their standard workstation queue; it must be routed to a workstation in the appropriately accredited enclave. The routing engine must query the analyst's clearance and accreditation record in the identity management system before dispatching any delivery notification. Automated AI services that process imagery at multiple classification levels must be accredited at the highest level of data they process, and their output products must carry the source classification markings of the imagery they consumed. Pipeline designers who defer these controls to "be added later" invariably discover that retrofitting classification-aware routing into an existing message-passing architecture is more expensive and disruptive than building it in from the start.

Pipeline performance: throughput, latency, and storage requirements at operational scale

A medium-scale defense imagery pipeline supporting a single theater of operations typically processes 50--150 satellite scenes per day from multiple commercial and government sources. At 0.5 m resolution, a standard commercial collection swath covers 15--30 km wide and 100--200 km long, producing orthorectified scenes of 1--4 GB each as GeoTIFF and 2--8 GB as uncompressed NITF. Daily ingest volume at this scale runs 150--600 GB of new scene data, plus preprocessing intermediates that may double or triple the working storage requirement during active processing. A high-resolution full-theater surge -- comprehensive coverage across a large contested area -- can push daily ingest volumes to several terabytes, requiring preprocessing clusters that scale horizontally to meet latency SLAs.

Processing latency is the performance constraint that most directly affects operational utility. For TSI workflows, the target is under 30 minutes from scene delivery to catalog availability; for routine production, under 4 hours is acceptable. The orthorectification step is the most computationally intensive stage: a full-resolution 0.3 m panchromatic scene with RPC refinement and DEM projection takes 5--20 minutes on a single modern compute node. Parallelizing across scene tiles and running multiple scenes concurrently on a cluster of 8--16 nodes achieves TSI latency targets for typical scene volumes. Atmospheric correction is computationally lighter (1--3 minutes per scene) but requires access to coincident atmospheric parameter data from NWP model analysis or satellite-derived aerosol products, which introduces a data dependency that can delay processing if the ancillary data pipeline is not pre-populated.

Storage architecture follows a tiered access model aligned with exploitation patterns. Active working storage (NVMe-backed block or high-performance object storage) holds the most recent 30--60 days of orthorectified scenes at full resolution, supporting sub-second catalog queries and rapid scene retrieval for active exploitation. The 6--18 month active archive tier uses object storage (S3-compatible) with retrieval latency of seconds to minutes, adequate for historical analysis and change detection baselines. Long-term retention beyond 18 months moves to cold object storage or tape, with retrieval latency of hours -- acceptable for historical record obligations but not for active exploitation. The STAC catalog database always holds complete metadata for all tiers; the storage URI in each catalog record points to the appropriate tier, and the retrieval layer handles tier-transparent access so that exploitation tools do not need to know which storage tier a requested scene resides in.