Power-constrained AI inference at the tactical edge

Every AI inference system that leaves a data center and enters a tactical environment hits the same wall: the hardware that delivers the best model accuracy consumes power the platform cannot supply and generates heat the enclosure cannot shed. SWaP-C — Size, Weight, Power, and Cost — is not a secondary engineering concern at the tactical edge; it is the primary design constraint from which all other decisions derive. This article covers the full engineering chain from SWaP-C budget definition through hardware selection, quantization strategy, power profiling methodology, runtime selection, thermal management, and mission-specific power budget design for military AI deployments.

SWaP-C defined: why weight and power matter more than raw TOPS at the edge

The tactical edge constraint hierarchy puts TOPS last, not first. Before a system designer can care about how many tera-operations per second a chip can execute, three prior questions must be answered: does the chip fit in the available volume, does its mass push the platform over its payload limit, and does its power draw exceed what the platform's battery or alternator can supply? Only after all three answers are no does throughput become relevant.

Size constrains physical integration. A Jetson AGX Orin module measures 100 mm × 87 mm and requires a carrier board that adds another 15–20 mm in the z-axis. A soldier's dismounted computing kit, a small-caliber turret controller, or a loitering munition's guidance bay may offer only a 60 × 60 × 25 mm volume for additional compute. In that case, a 34 mm × 26 mm Hailo-8 M.2 accelerator fits where the AGX Orin does not.

Weight compounds across a system. A 300 g inference module added to a dismounted soldier's kit joins a load that may already weigh 35–45 kg. Military ergonomics programs use 30% of body weight as a maximum sustained load threshold; every gram of inference hardware competes with ammunition, water, and communications gear for allocation below that limit. On a UAV, added weight reduces flight time by increasing the power required to maintain altitude, creating a double penalty: more weight draws more battery power for propulsion and less battery power remains for the payload.

Power is the most unforgiving constraint because exceeding it is not just a performance problem — it can terminate the mission. A ground vehicle's alternator has a fixed output; a UAV battery has a fixed capacity. When the AI inference subsystem, communications radio, sensor suite, and vehicle systems collectively exceed available power, the power distribution unit sheds loads in priority order. AI inference is rarely the highest priority load. The system that was designed to run continuously runs intermittently or not at all.

Cost determines expendability doctrine, which in turn constrains hardware selection. A $3,000 Jetson AGX Orin developer kit cannot be installed on a one-way loitering munition intended for expendable use. The cost constraint is not purely economic — it also affects logistics, because expensive compute modules require secure supply chains, controlled storage, and accountability procedures that slow battlefield resupply cycles.

The correct figure of merit for tactical edge AI hardware is therefore useful inference throughput per watt per cubic centimeter per dollar — a multi-dimensional efficiency that no single TOPS number captures. For a detailed comparison of how the leading platforms stack up on this composite metric, our article on edge AI hardware selection for defense works through the trade space systematically.

AI accelerator hardware landscape: Jetson, Hailo, Coral, Qualcomm — peak TOPS vs sustained throughput vs power envelope comparison

Four silicon families cover the practical range of tactical edge deployments. Understanding where each sits on the TOPS/W curve and what architectural trade-offs each represents is prerequisite to any platform selection decision.

The Jetson family's strength is its software ecosystem: full Linux, CUDA, TensorRT, and a large body of open-source inference tooling. The Hailo-8's strength is raw power efficiency at the accelerator level — its dataflow architecture tiles the neural network graph across processor clusters and executes it in a pipelined fashion that minimizes DRAM access, which is the dominant power cost in conventional GPU inference. The Coral Edge TPU is the simplest integration path for teams already using TFLite, but its 4 TOPS ceiling and strict graph compilation requirements limit it to smaller models. Qualcomm's QCS8550 SoC combines CPU, GPU, and a Hexagon NPU on a single die, delivering excellent TOPS/W in a smartphone-derived package that is increasingly appearing in handheld military devices. A thorough side-by-side evaluation of these platforms appears in our edge AI hardware comparison article.

Sustained throughput diverges significantly from peak TOPS under real operating conditions. Peak TOPS figures are measured at 100% datapath utilization with idealized memory access patterns. Real inference workloads on YOLOv8 or RT-DETR architectures often sustain 50–70% of peak TOPS because the model's operator mix — convolutional layers, batch normalization, activation functions, and the detection head's multi-scale output processing — does not keep all execution units busy simultaneously. Benchmark any candidate platform with your actual model before committing to it in a platform design.

Quantization strategies: INT8, INT4, FP16 — accuracy vs power trade-offs, PTQ vs QAT, per-channel vs per-tensor

Quantization is the single most impactful software technique for reducing inference power at the tactical edge. Reducing numerical precision from FP32 to INT8 shrinks the model's memory footprint by 4x and cuts DRAM bandwidth demand proportionally — and since DRAM access is often the dominant power draw in inference workloads, this translates directly to lower power consumption independent of the compute savings.

FP16 (16-bit floating point) is the least aggressive option. It halves memory footprint relative to FP32 and is natively supported by all modern AI accelerators, including Jetson GPU tensor cores. Accuracy loss is negligible — typically less than 0.2 mAP on standard detection benchmarks — because the reduced dynamic range of FP16 rarely clips meaningful weight values in a well-trained model. FP16 is the right default for the first deployment of a new model when accuracy risk tolerance is low and the power saving from INT8 is not required to meet the SWaP-C budget.

INT8 is the production standard for power-constrained military edge deployments. Hardware with INT8 execution units — Jetson Orin, Hailo-8, Coral Edge TPU — delivers 2–4x throughput improvement over FP32 at equivalent model capacity, with typical accuracy loss of 0.5–2 mAP. The accuracy impact is strongly dependent on calibration quality. Post-training quantization (PTQ) calibrates INT8 scale factors by running the FP32 model on a calibration dataset and recording the distribution of activations at each layer. The choice of calibration dataset is critical: calibrate on images from the deployment sensor and altitude, not on a generic public aerial dataset, or the activation distributions will be mismatched and accuracy loss will be higher than the benchmark numbers suggest.

Quantization-aware training (QAT) inserts simulated quantization noise into the training graph so the optimizer adjusts weights to be robust to INT8 rounding. QAT consistently outperforms PTQ by 1–3 mAP, with the largest gains on small models where per-channel weight variance is higher. The cost is a 10–50 epoch fine-tuning run — typically 6–24 hours on a training GPU — and access to the original training dataset. For programs where accuracy has a hard floor tied to a probability-of-detection requirement, QAT is the correct approach. For rapid prototyping or post-handoff situations where the training dataset is unavailable, calibrated PTQ with per-channel scale factors is the practical alternative.

Per-channel vs per-tensor quantization is a calibration choice that significantly affects accuracy on convolutional models. Per-tensor quantization assigns a single scale factor to the entire weight tensor of a layer; per-channel assigns an independent scale factor to each output channel. The difference matters because convolutional filters in mature detection models have substantially different L2 norms across channels — the network has learned to concentrate information in some filters and suppress it in others. Forcing a single scale factor across all channels creates large rounding errors in the high-norm filters. Per-channel quantization preserves accuracy at the cost of a marginally larger calibration table. All production TensorRT and PyTorch quantization tooling supports per-channel weight quantization; there is no engineering reason to choose per-tensor for weight quantization in 2026.

INT4 quantization packs two values per byte and achieves up to 8x compression relative to FP32, but accuracy loss of 3–8 mAP on detection benchmarks makes it unsuitable for most tactical applications without QAT. Its primary use case is very large language model inference where DRAM bandwidth, not compute, is the binding constraint. For defense ISR and target detection applications running YOLOv8-class models, INT8 is the practical operating point.

# TensorRT INT8 calibration — representative snippet
import tensorrt as trt

class Int8Calibrator(trt.IInt8EntropyCalibrator2):
    def __init__(self, calibration_images, cache_file):
        super().__init__()
        self.cache_file = cache_file
        self.dataset = calibration_images   # 500+ deployment-domain images
        self.index = 0

    def get_batch(self, names):
        if self.index >= len(self.dataset):
            return None
        batch = preprocess(self.dataset[self.index])
        self.index += 1
        return [cuda.memcpy_htod_async(d_input, batch, stream)]

    def get_calibration_cache(self):
        if os.path.exists(self.cache_file):
            with open(self.cache_file, 'rb') as f:
                return f.read()
        return None

# Build INT8 engine
config.set_flag(trt.BuilderFlag.INT8)
config.int8_calibrator = Int8Calibrator(calib_images, 'calib.cache')
# Per-channel quantization is default in TRT 9+

Power profiling methodology — measurement instrumentation, dynamic vs static power, thermal throttling in MILSPEC environments

Software-reported power estimates from vendor APIs are useful for trend monitoring but insufficient for system-level power budget validation. The only reliable source of truth is inline current measurement on the hardware power rails supplying the AI accelerator. The gap between software-reported and measured power can reach 15–25% on Jetson platforms during sustained inference, because the thermal management firmware adjusts clock frequencies in ways that are not always reflected in the API's instantaneous readings.

The standard instrumentation approach for embedded AI platforms uses INA3221 or INA226 current sense amplifiers placed in series with each power rail's supply line, reading via I2C at 100–200 Hz. On Jetson Orin, NVIDIA exposes the on-board INA3221 sensors through sysfs:

# Read Jetson Orin power rails via sysfs (sample every 100 ms)
RAILS=(
  "/sys/bus/i2c/drivers/ina3221/1-0040/hwmon/hwmon1"  # VDD_GPU_SOC
  "/sys/bus/i2c/drivers/ina3221/1-0040/hwmon/hwmon2"  # VDD_CPU_CV
  "/sys/bus/i2c/drivers/ina3221/1-0041/hwmon/hwmon3"  # VIN_SYS_5V0
)
while true; do
  ts=$(date +%s%N)
  for rail in "${RAILS[@]}"; do
    pwr=$(cat "$rail/power1_input" 2>/dev/null)  # microwatts
    echo "$ts,$rail,$pwr"
  done
  sleep 0.1
done >> /var/log/power_trace.csv

Static (idle) power is the baseline draw with the model loaded in accelerator memory and the inference pipeline running but no frames being processed. For Jetson Orin Nano in 7 W mode, idle power is approximately 1.5–2.0 W. This is the floor below which the system never drops while the inference service is active. In a duty-cycle model, idle power multiplied by idle time sets the minimum energy cost of keeping the inference capability ready.

Dynamic (active inference) power is the incremental draw above idle during active frame processing. The difference between idle and active power represents the energy cost of each inference cycle. For a Jetson Orin Nano running YOLOv8s INT8 at 15 fps, dynamic power above idle is approximately 4–6 W, for a total of 5.5–8 W during inference bursts.

Thermal throttling is the most dangerous power profiling scenario to omit from a MILSPEC test program. All modern AI accelerators have firmware that reduces clock frequencies when die temperature approaches the rated junction maximum. On Jetson platforms, this is governed by the thermal management framework and begins throttling at 5–10°C below the junction limit. In a MILSPEC environment where the enclosure can reach +70°C ambient, a chip rated to 85°C junction temperature has only a 15°C thermal budget from ambient to junction. If the thermal resistance from die to ambient exceeds 15°C/W and the chip is dissipating 10 W, the junction temperature cannot stabilize below the junction limit — throttling begins immediately and persists indefinitely. The practical consequence is that the inference throughput specified at room temperature is not the throughput available on a hot vehicle in a desert environment. Power profiling must be conducted at the maximum expected ambient temperature with the hardware installed in its production enclosure.

Operating system and runtime selection for power efficiency — bare-metal vs RTOS vs Linux, TensorRT vs ONNX Runtime vs TFLite

The choice of operating environment directly affects idle power. A full Linux distribution with a desktop environment, background services, and a logging stack draws significantly more power at idle than a minimal Linux with only the inference service and its dependencies. Bare-metal or RTOS deployments can reduce idle power further still by eliminating the OS scheduler and kernel subsystems entirely, but they sacrifice the toolchain compatibility that makes deploying and updating models practical in an operational program.

For Jetson platforms, the recommended path is minimal Linux (Ubuntu Server or Yocto-derived BSP) configured for target-specific boot. Disable systemd services not required for inference operation: remove the display manager, the NetworkManager daemon if the platform uses a fixed network configuration, any cloud telemetry agents, and Bluetooth if the hardware supports it. The nvpmodel service should be retained and set to the lowest power mode that meets the inference latency requirement at mission frame rate. A properly stripped Jetson Orin Nano running only the inference service, an SSH daemon, and a minimal logging process has idle power of approximately 1.2–1.8 W versus 2.5–3.5 W for the default developer image.

Runtime selection determines how the quantized model graph is compiled to hardware instructions at load time or ahead of time. TensorRT is the highest-performing runtime for NVIDIA Jetson hardware. It fuses adjacent operator kernels, selects the fastest algorithm variant for each layer given the target precision and batch size, and exploits the Jetson GPU's tensor core capabilities fully. A YOLOv8s model compiled to a TensorRT INT8 engine file runs at 30–45 fps at 7–9 W on Jetson Orin Nano; the same model in a generic ONNX Runtime session without TensorRT backend runs at 12–18 fps at 10–14 W. The power difference is not accidental — TensorRT's kernel fusion reduces the number of DRAM access round-trips per inference, and DRAM access is the dominant power cost.

ONNX Runtime with TensorRT Execution Provider is the recommended option for programs that need to target both Jetson hardware and non-NVIDIA accelerators from the same codebase. The ONNX Runtime API abstracts the hardware backend, and the TensorRT EP handles the compilation and execution on Jetson while a different EP (DirectML, CUDA generic, CPU) handles execution on other platforms. The portability benefit comes at a marginal performance cost relative to using TensorRT directly, typically 5–10% throughput reduction.

TFLite with the Edge TPU delegate is the correct runtime for Coral platforms. It compiles the model graph to the Edge TPU's on-chip SRAM at load time; any operations that cannot be mapped to the Edge TPU execute on the host CPU. The fraction of the model that falls back to CPU is the primary driver of both latency and power efficiency — a model with 10% CPU fallback may consume 2–3x the power of a fully on-chip model because the host CPU draws significantly more power per operation than the Edge TPU's dedicated hardware. Use the edgetpu_compiler's compilation log to identify operations that are not mapped to hardware and modify the model architecture to replace them with Edge TPU-compatible equivalents before deploying to production.

Thermal management in MILSPEC enclosures — heat spreaders, conduction cooling, -40°C to +85°C operational range

MILSPEC enclosures for tactical electronics typically specify an operational temperature range of -40°C to +85°C per MIL-STD-810 method 501/502. The lower bound and upper bound create opposite thermal engineering challenges. At -40°C, silicon devices may not start reliably without pre-conditioning; at +85°C ambient, passive heat rejection is marginal for any AI accelerator dissipating more than 3–4 W.

Conduction cooling is the dominant heat rejection mechanism in sealed MILSPEC enclosures where convective airflow is excluded by the IP protection rating. The thermal path runs from the AI accelerator's die through the package, through a thermal interface material (TIM), through a heat spreader plate, through the enclosure wall, and finally to the external environment via natural convection and radiation. Each material interface adds thermal resistance, and the sum of all resistances determines the temperature rise from die junction to ambient air.

Thermal resistance budget — Hailo-8 in sealed Al enclosure
==========================================================
Junction → case (Hailo-8 package):      2.0 °C/W
Case → TIM (phase-change pad, 1 mm):    0.5 °C/W
TIM → heatspreader (Al 6061, 3 mm):     0.3 °C/W
Heatspreader → enclosure wall:          1.0 °C/W
Enclosure wall → ambient (150 cm² Al):  4.5 °C/W
                                        ─────────
Total R_th (j→a):                       8.3 °C/W

At P_diss = 5 W, ΔT = 8.3 × 5 = 41.5 °C
At T_ambient = 71°C (MIL):  T_junction = 71 + 41.5 = 112.5 °C
Hailo-8 rated max junction: 125 °C → margin 12.5 °C ✓

Heat spreader material selection matters significantly. Aluminum 6061 has a thermal conductivity of approximately 167 W/m·K and is the standard aerospace structural alloy. Copper (385 W/m·K) provides 2.3x better thermal conductance and is used where the heat spreader must bridge a larger area between the chip package and the enclosure wall. Pyrolytic graphite sheets (700–1500 W/m·K in-plane) are used in the most demanding applications where aluminum cannot meet the thermal budget, but they are brittle and require protection from vibration and shock loads — a significant concern in armored vehicle or air-dropped applications. For most tactical AI enclosures operating with Hailo-8 or Coral-class accelerators, a 3–5 mm aluminum plate between the module and the enclosure wall is sufficient if the enclosure has at least 100–150 cm² of exterior surface area for natural convection.

Cold start at -40°C requires that the board be powered with heaters before applying compute load. COTS Jetson modules are rated to -25°C (industrial variant) for storage and operation; some military variants extend to -40°C. Below the rated lower limit, internal capacitors may not charge correctly and flash storage may not initialize. A thermostat-controlled resistive heater drawing 3–5 W from the platform's battery — activating automatically when enclosure temperature drops below -20°C — is a simpler and more reliable solution than waiting for semiconductor suppliers to specify operation at -40°C. The heater should be placed between the enclosure wall and the compute board to ensure the board temperature reaches the minimum rated operating temperature before main power is applied.

Thermal testing must be conducted with the production enclosure and the production mounting configuration. Benchtop testing with an open carrier board in a thermal chamber will show lower junction temperatures than the production configuration because the forced convection from the chamber fan helps cool the board in ways the sealed enclosure cannot replicate. Always test in the worst-case configuration — sealed enclosure, maximum inference duty cycle, maximum ambient temperature — and document the steady-state junction temperature at that condition as part of the system qualification record.

Mission-specific power budget design — duty cycle modeling, inference frequency vs battery life, power modes linked to threat state

A system designed for peak inference performance throughout an entire mission will either run out of battery before mission completion or carry a larger battery that adds weight and cost. Neither outcome is optimal. The correct design approach links inference frequency to operational state, running the AI accelerator at high duty cycle only when the tactical situation warrants it and reducing duty cycle — or entering a low-power idle state — during phases where full inference is unnecessary.

Duty cycle modeling begins with a mission profile: a time-ordered sequence of operational phases with estimated durations and associated threat states. A ground vehicle mission might have a transit phase (60 minutes, low threat, moving along a known-safe route), an approach phase (20 minutes, elevated threat, entering unknown territory), and a surveillance phase (40 minutes, high threat, stationary observation). Each phase has a different inference requirement: 2 fps during transit, 10 fps during approach, 15 fps during surveillance. The power draw in each phase is the sum of idle power plus (active inference power × inference duty cycle fraction).

The table illustrates that duty-cycle management reduces AI inference energy consumption by 42% relative to running at maximum frame rate continuously — from 20 Wh to 11.5 Wh for a 160-minute mission on a Hailo-8 class platform. On a system with a 40 Wh AI subsystem battery allocation, the managed approach extends battery endurance from 120 minutes to 210 minutes, which can be the difference between a platform that completes the mission and one that loses AI capability in the final phase.

Power mode transitions should be driven by the platform's operational state machine rather than by manual crew input, because crew cognitive load in high-threat environments makes manual management unreliable. State machine inputs include vehicle speed (above 15 km/h suggests transit mode), weapon system status (armed suggests elevated threat), GPS geofencing against known threat areas, and explicit commander override commands. The AI inference manager receives state transitions as events and adjusts the inference loop's frame rate accordingly, either by changing the sleep interval between frame captures or by configuring the accelerator's operating frequency through the power mode API. The complete design approach for deploying these systems in field conditions is covered in our article on onboard AI inference for UAV platforms, which applies the same duty cycle principles in the airborne context.

Reserve margin must be built into every power budget. A 20% reserve is a minimum — tactical missions routinely extend beyond their planned duration, ambient temperatures in summer desert environments exceed planning assumptions, and software updates may increase model size or inference frequency between planning and execution. A power budget that is 100% allocated at planning time will be overdrawn in the field. Size the battery or power allocation to cover 120% of the duty-cycle modeled energy requirement, and document the reserve policy in the system design specification.