Time-series databases for defense telemetry: ingest, retention, query

A modern armored vehicle emits hundreds of sensor readings per second – engine RPM, oil pressure, gearbox temperature, suspension load, fuel flow, battery voltage, GPS position, and link quality on every radio attached to it. A fleet of 200 vehicles produces tens of millions of data points per minute. A naval vessel with full machinery monitoring produces more. Storing this telemetry in a relational database was never viable at scale; the write patterns, query shapes, and retention requirements are fundamentally different from transactional workloads. Time-series databases (TSDBs) exist precisely to solve this problem, and the engineering decisions made when deploying one – schema design, batching parameters, downsampling policy, tag cardinality – determine whether the system can sustain operational tempo or collapses under load within days of deployment. This article covers the full lifecycle: from sensor network ingest through retention tiers and query patterns for defense analytics.

Why time-series databases exist

A time-series database is a storage engine purpose-built for append-heavy, timestamp-ordered data where queries nearly always involve a time range, and where the primary value of the data is in its temporal relationship – how a metric changes over time, not how any individual reading relates to another entity.

The core technical distinction from relational databases is storage layout. TSDBs use columnar storage with automatic partitioning by time (called shards, chunks, or buckets depending on the product). All readings for a given metric within a time window are stored physically adjacent on disk. This means a time-range aggregation query – "give me the mean engine temperature for platform P over the past 24 hours" – reads a contiguous block of disk, not scattered heap pages. At 10,000 sensor writes per second, a TSDB can sustain this workload with single-digit millisecond write latency on commodity NVMe storage. A relational database would saturate its I/O subsystem under the same write rate because each insert must update multiple indexes.

Compression is the other critical advantage. Floating-point sensor readings from adjacent timestamps are often nearly identical – temperature changes by fractions of a degree between samples. TSDBs use delta encoding, XOR compression (for IEEE 754 doubles), and run-length encoding to achieve compression ratios of 10:1 or better on typical telemetry streams. A raw telemetry stream that would occupy 1 TB in a relational database stores in 80–120 GB in a TSDB.

Schema design: measurements, tags, and fields

The schema decisions made before first write are the hardest to reverse. A poorly designed tag set causes series cardinality explosion – a condition where the number of distinct time-series in the database grows unboundedly, consuming index memory and degrading all queries. This is the most common production failure mode for TSDB deployments and it is entirely avoidable with correct design.

Measurements

A measurement (called a metric family in some products) is a logical grouping of related fields that are always written together at the same timestamp. Sensible measurement boundaries for defense platform telemetry include: engine_telemetry (RPM, oil pressure, coolant temperature, fuel flow rate), nav_position (latitude, longitude, altitude, heading, speed over ground), power_systems (battery voltage, alternator output, load current per bus), and radio_link_quality (signal strength, noise floor, packet error rate, retransmission count). Fields within the same measurement share a timestamp and are stored together, so grouping by write cadence and operational cohesion produces the most efficient layout.

Tags versus fields

Tags are indexed metadata that identify the series. Every unique combination of tag values creates a distinct series in the index. Fields are the numeric values written at each timestamp – they are not indexed, only stored. The design rule is: if you need to filter or group by a dimension in a query predicate, it must be a tag. If you only need to read the value, it should be a field.

For military platform telemetry, appropriate tags are: platform_id (a stable short identifier, e.g. "VH-041"), platform_type (e.g. "M1A2", "BTR-4", "Mi-8"), unit_id (battalion or squadron identifier), sensor_class (e.g. "engine", "nav", "comms"), and base or grid_zone for coarse location context. These are low-cardinality: a fleet of 500 vehicles with 20 unit assignments and 5 platform types produces at most 50,000 distinct series – well within the comfortable operating range of any production TSDB.

What must NOT be tags: raw GPS coordinates, event UUIDs, free-text fault codes, or any other field with cardinality proportional to the number of data points. Placing a raw latitude as a tag creates one new series for every data point – the index grows without bound and query performance degrades within hours. High-cardinality identifiers belong in fields or in a separate relational metadata store that joins to the TSDB series by the stable low-cardinality tags.

High-rate ingest: batching, buffering, and out-of-order writes

Defense sensors – particularly those on vehicle platforms or UAVs – do not write to the TSDB directly. An edge collector runs on the platform or on the gateway that aggregates data from a vehicle section, buffers readings locally, and flushes batches to the TSDB over the tactical network. The ingest architecture has three parameters that determine whether it can absorb the sustained write load without dropping data or saturating the network.

Batch size. Writing one point at a time produces one network round-trip per sensor reading – completely unsustainable at high rates. Batch sizes of 1,000–5,000 points per write request reduce network overhead by three orders of magnitude. The tradeoff is write latency: with a 1,000-point batch and a 100 Hz sensor, data is delayed by up to 10 seconds before the batch flushes. For operational monitoring where 10–30 second latency is acceptable, large batches are optimal. For near-real-time alerting, smaller batches with a time-based flush interval (e.g., flush every 2 seconds regardless of batch fullness) are more appropriate.

Write-ahead buffering. Tactical radio links experience outages. When the link is down, the edge collector must buffer unsent data locally – to persistent storage, not only memory, so data survives a process restart or power cycle. Buffer sizing should be calculated from the maximum expected link outage duration multiplied by the peak write rate: a 10-minute outage with a 5,000-point-per-second sensor stream requires 3 million points of buffer capacity, roughly 150 MB at typical floating-point field widths. Collectors that use only in-memory buffers will lose data on every link interruption.

Out-of-order write acceptance. When buffered data arrives after the link is restored, it carries timestamps from the past. TSDBs vary significantly in their tolerance for out-of-order writes: some reject writes with timestamps older than a configurable window; others accept any historical write but at a performance cost. The acceptance window must be configured to match the maximum expected link outage for the platform type. For vehicle platforms on tactical radio, 300–600 seconds is typical. For satellite-relay platforms where a link outage during a pass gap can last 90 minutes, the window must be 5,400 seconds or more.

Retention policies and downsampling

Raw telemetry at full resolution is expensive to store long-term and rarely necessary beyond a short operational window. A three-tier retention policy covers virtually all defense telemetry requirements without unnecessary storage cost.

Tier 1 – Raw. Full-resolution data (10–100 Hz depending on sensor type) retained for 7–30 days. This window supports real-time monitoring, immediate post-incident analysis, and threshold-alert review. For a 500-platform fleet with 50 sensors per platform writing at 10 Hz, full-resolution storage consumes approximately 2–4 TB per day at compressed TSDB storage – manageable for 30-day retention with commodity NAS hardware.

Tier 2 – 1-minute aggregates. Computed by continuous query or downsampling task: mean, min, max, and count for each field over each 1-minute window. Retained for 6–12 months. This resolution supports trend analysis, predictive maintenance feature engineering, and anomaly detection at fleet scale. Storage at 1-minute resolution is approximately 600× smaller than the raw tier.

Tier 3 – 1-hour aggregates. Retained for 3–7 years for long-term fleet health analysis, lifecycle planning, and sustainment program review. At this resolution, a 7-year history for a 500-platform fleet occupies well under 100 GB – trivially inexpensive relative to the operational value of the historical record.

Downsampling tasks must be scheduled with a deliberate offset from the window boundary. A task scheduled to run at exactly the minute boundary will frequently read an incomplete window – the last few seconds of data may not yet have flushed from the ingest pipeline. A 30-second offset ensures the window is complete before aggregation. This detail eliminates an entire class of subtle edge artifacts that appear as anomalous dips or spikes at regular intervals in downsampled data.

Query patterns for defense telemetry analytics

Five query patterns account for the majority of operational and analytical use against a defense telemetry TSDB. A production deployment must handle all five with index-only execution – no full-series scans – at the data volumes expected after 6–12 months of accumulation.

Last-value queries. "What is the current engine temperature of vehicle VH-041?" This is the most frequent query in an operational monitoring dashboard. TSDBs optimized for this pattern maintain a last-value index per series so the query returns in constant time regardless of how much historical data exists. Without this optimization, last-value queries degrade to a reverse-time-scan over the raw series – acceptable at low cardinality, unacceptable at fleet scale.

Time-range aggregations. "What was the mean fuel consumption rate for all M1A2 platforms in Unit-7 over the past 72 hours?" This is the core analytics query: a tag filter selecting the relevant series, a time-range scan, and an aggregation function applied across both the time dimension and the filtered series. Execution time should be measured in tens of milliseconds at 12-month data volumes for a correctly indexed schema; hundreds of milliseconds indicate a cardinality problem.

Threshold crossing detection. "When did gearbox oil temperature on any vehicle in the fleet first exceed 120°C in the past 30 days?" This query requires scanning over a time range with a predicate on a field value. Some TSDBs execute this efficiently with columnar min/max metadata that prunes chunks with no values exceeding the threshold; others require a full field scan. Understanding which execution strategy the chosen product uses is critical for sizing alerting system latency budgets.

Multi-series comparison. "Show me fuel consumption for all vehicles of type BTR-4 over the past week, grouped by unit." This is the fleet benchmarking query used by maintenance planners to identify outlier platforms. It requires the tag index to efficiently enumerate all series matching the platform_type filter, then aggregates each. The result is one time-series per unit – a small number of output series regardless of fleet size, if the tag schema is correct.

Derivative queries. "What is the rate of change of vibration amplitude on the port engine of VH-041 over the past 6 hours?" Rate-of-change is the core feature for mechanical anomaly detection – a sudden increase in the derivative of a vibration or temperature series often precedes component failure by hours or days. This feeds directly into the message queue pipeline that delivers anomaly events to the maintenance operations center. Most TSDBs support derivative computation as a native function; those that do not require an application-layer computation over a dense time-range query result, which is viable but adds latency.

Integration with the broader data platform

A TSDB does not operate in isolation. In a defense data platform, it is one component in a pipeline that also includes edge collectors, message queues for real-time event routing, a data lake for long-term multi-format storage, and analytics and monitoring frontends. The integration contract between the TSDB and these components must be defined explicitly.

For downstream consumers that need near-real-time telemetry – alerting systems, operational dashboards, fusion engines – the recommended pattern is for the edge collector to publish each batch to both the TSDB (for persistence and historical query) and a message queue topic (for low-latency event routing to consumers). This decouples the TSDB from the real-time delivery path: consumers receive events within seconds of sensor capture, without polling the TSDB, and the TSDB serves only historical and aggregation queries for which its storage layout is optimized.

For integration with the defense data lake, the TSDB's downsampled tiers are the appropriate source: export 1-minute aggregate snapshots as Parquet or CSV on a scheduled basis and land them in the lake's ingestion zone. Exporting raw data from the TSDB to the lake is redundant if the edge collector already writes raw data to both destinations – but the TSDB remains the authoritative source for time-range queries over recent data because its storage format is orders of magnitude faster for this pattern than the lake's object-store-backed columnar files.