Thermal imagery classification at the edge: IR sensor processing for defense

Thermal infrared imagery gives operators what visible-band cameras cannot: persistent detection capability through darkness, smoke, dust, and light camouflage. A vehicle engine warm from recent operation glows against a cold field at 3 AM. A dismount crossing open ground radiates a clear silhouette at 2 km in conditions that blind any EO camera. The physics of thermal emission is inherently discriminating -- targets that generate or retain heat stand out from their surroundings in ways that no amount of optical camouflage can fully suppress. Deploying a classification model to process this imagery at the edge, on the sensor platform rather than a remote server, is what turns a thermal camera into an autonomous detection layer. This article covers the sensor physics, preprocessing requirements, CNN adaptations for single-channel thermal input, the specific challenges of small target detection, multi-sensor fusion, quantization constraints, and the evaluation frameworks used to validate military thermal AI.

Thermal sensor physics and why IR differs from visible-band imagery

Thermal infrared cameras do not capture reflected ambient light. They measure the long-wave infrared radiation (LWIR, 8–14 µm) or mid-wave infrared radiation (MWIR, 3–5 µm) that objects emit as a function of their surface temperature and emissivity. The Stefan-Boltzmann law governs the total radiated power, but in practice the relevant quantity for target detection is the differential radiance between the target and the local background -- the apparent temperature contrast. A human body at 37°C radiates approximately 100 W of infrared power continuously. A tank engine compartment after 30 minutes of operation can sustain a contrast of 20–40 K above ambient. These thermal signatures exist regardless of illumination conditions, making LWIR and MWIR sensors the primary passive detection modality for military ISR at night.

The single-channel nature of thermal imagery is the most consequential difference from visible-band input for AI pipeline design. An RGB camera produces three correlated channels that encode color, and the vast majority of pretrained CNN weights -- ResNet, MobileNet, EfficientNet, and their derivatives -- were trained on three-channel ImageNet data. Applying these models directly to a single-channel thermal frame by simply treating it as one of three identical input planes produces suboptimal results: the first convolutional layer's learned filters are designed to extract color edges and chromatic gradients that have no meaning in thermal data. The model must be adapted or retrained to extract the relevant features in thermal imagery, which are primarily thermal contrast gradients, emissivity boundaries between materials, and temporal radiance changes caused by motion or heat transfer.

Uncooled microbolometer detectors, used in most tactical and airborne edge thermal sensors, operate at ambient temperature and measure the change in electrical resistance of each pixel element as it absorbs infrared radiation. They are significantly less sensitive than cooled photon-detector arrays (InSb or HgCdTe), with noise-equivalent temperature difference (NETD) values of 30–80 mK versus 5–20 mK for cooled sensors, but their low power consumption (under 2 W for the detector alone), lack of a cryogenic cooling system, and sub-$3,000 module cost make them the dominant platform for UAS and ground-vehicle-mounted edge AI deployments. The lower NETD directly sets the minimum detectable temperature contrast, which in turn sets the maximum detection range for a given target type and atmospheric condition.

Pre-processing: non-uniformity correction, bad pixel replacement, and contrast enhancement

Raw microbolometer output cannot be fed directly into a classification model. Three preprocessing stages are mandatory for consistent model performance: non-uniformity correction (NUC), bad pixel replacement (BPR), and adaptive contrast enhancement. Each stage compensates for a different category of sensor artifact.

Non-uniformity correction addresses the fixed-pattern noise inherent to all focal plane arrays. Each pixel in the FPA has a slightly different gain and offset characteristic due to manufacturing variations, and these characteristics drift slowly with detector temperature. Without NUC, the raw image contains a spatially structured noise pattern -- often described as a "waffle" or "plaid" artifact -- that is stable enough to be partially learned by a classification model as a spurious feature, elevating false-positive rates for objects whose outline happens to align with the NUC artifact pattern. The standard factory calibration procedure applies two-point NUC by imaging two uniform-temperature blackbody sources and computing per-pixel gain and offset correction coefficients. In field operation, scene-based NUC supplements the factory calibration: when the sensor pans or the scene contains large uniform regions, a running estimate of the flat-field offset is accumulated and subtracted from subsequent frames, correcting for the thermal drift that develops over tens of minutes of operation.

Bad pixel replacement handles detector elements that respond incorrectly -- permanently stuck at a fixed value, excessively noisy, or unresponsive to temperature changes. The manufacturer supplies a bad pixel map with each sensor module. Before any other processing, bad pixel values are replaced using bilinear interpolation from immediate neighbours. For a typical 640x512 uncooled FPA, bad pixel counts of 0.1–0.5% (320–1,600 pixels) are within specification and do not materially degrade detection performance after BPR. Contrast enhancement using contrast-limited adaptive histogram equalization (CLAHE) is the final preprocessing step: it boosts local contrast within spatially limited tile regions (typically 8x8 pixel tiles) without amplifying noise globally, making subtle thermal gradients at target boundaries visible to both human operators and convolutional feature extractors.

CNN architectures for thermal classification: single-channel adaptations

Adapting a pretrained CNN for single-channel thermal input is a well-studied problem with three established approaches, each with a different accuracy-training-cost tradeoff. The simplest approach replicates the single thermal channel into three identical planes, allowing the pretrained model to be used without any architectural modification. This works moderately well for coarse classification tasks but performs poorly when thermal-specific texture features matter, because the first-layer filters are constrained to extract redundant representations across three identical planes rather than specializing to thermal contrast.

The second and generally preferred approach replaces the first convolutional layer with a new single-channel layer while retaining all subsequent pretrained weights. The new first layer is initialized from the mean of the three pretrained input channel weights, which provides a sensible prior for the first-layer filter responses and significantly reduces the amount of thermal-domain training data needed to fine-tune to acceptable accuracy. Experiments on military thermal datasets consistently show 2–5% accuracy improvement over the channel-replication approach when fine-tuned on the same dataset. The third approach trains entirely from scratch on a thermal-specific dataset, which achieves the highest domain-specific accuracy but requires substantially more labeled thermal data (typically 50,000 or more annotated examples across all target classes) to avoid underfitting. For military thermal classification, where labeled data is expensive to acquire and often export-controlled, transfer learning from the second approach is the practical standard.

Architecture scale matters significantly for edge deployment. A full ResNet-50 adapted for thermal input achieves high classification accuracy but requires 25 MB of model weights and 4 GFLOPS per inference pass at 640x512 resolution -- infeasible for a real-time 30 Hz pipeline on a Jetson Orin Nano at under 15 W. MobileNetV3-Small adapted for single-channel input reduces this to 2.5 MB and 56 MFLOPS, fitting comfortably within the compute budget while retaining competitive accuracy on standard thermal classification benchmarks. The same architecture constraints that govern automatic target recognition at the edge apply here: model selection is a joint optimization over accuracy, latency, memory footprint, and power draw rather than accuracy alone.

Small target detection in thermal imagery: low-SNR environment challenges

When a person or vehicle is observed at long range -- beyond 2 km for a typical uncooled sensor with a medium focal length -- the target occupies only 2–8 pixels in the FPA. At this scale, standard object detection frameworks that rely on spatial feature hierarchies (YOLO-family detectors, Faster R-CNN) cease to function reliably. The receptive field of deeper convolutional layers spans hundreds of pixels; a 4-pixel target contributes almost no discriminating signal to those layers. Effective small target detection requires an entirely different detection strategy that operates at the pixel and patch scale before any spatial downsampling.

The dominant approach for small target detection in LWIR imagery is the infrared search-and-track (IRST) pipeline, which combines background suppression with temporal accumulation. Background suppression uses a top-hat morphological transform with a structuring element slightly larger than the expected target footprint to subtract extended background structure while preserving point-like or small-footprint targets. The residual after background subtraction contains candidate target blobs at signal-to-clutter ratios as low as 1.5 dB, below which reliable single-frame detection is physically impossible given the sensor NETD. Temporal accumulation -- integrating evidence across 5–20 frames along predicted trajectory -- extends the effective detection threshold by exploiting target motion coherence, allowing detection of targets that would be invisible in any individual frame. Track-before-detect frameworks formalize this accumulation: they maintain a set of candidate track hypotheses and score each hypothesis against incoming frame data without ever declaring a single-frame detection, committing to a confirmed track only when accumulated evidence exceeds a threshold.

Fusing thermal detections with EO and radar track data

No single sensor modality is sufficient for reliable classification and tracking across all operational conditions. Thermal sensors degrade in rain and heavy fog (water vapor absorbs LWIR efficiently), lose target contrast when background and target temperatures equalize (midday thermal crossover), and cannot resolve fine-grained visual features that enable platform-type identification. Visible-band EO sensors fail at night and in smoke. Radar provides long-range detection and velocity measurement but lacks the spatial resolution to classify target type at relevant tactical ranges without additional reference data. Multi-sensor fusion at the track level combines the complementary strengths of each modality into a more robust and informative track than any sensor provides alone.

Track-level fusion, rather than pixel-level image fusion, is the architecture used in deployed military ISR systems because it is computationally tractable on edge hardware and robust to sensor availability gaps. Each sensor produces an independent detection stream that is converted to a common track representation: a state vector (position, velocity, heading), a covariance matrix encoding estimation uncertainty, a class label with confidence, and a sensor-type tag. A multi-sensor association gate matches new detections from each sensor to existing tracks using Mahalanobis distance, gating out detections that are statistically inconsistent with any existing track state. When a thermal detection and an EO detection are associated to the same track within the same time window, the fused state applies a covariance-weighted combination: the sensor with smaller position uncertainty contributes proportionally more to the fused estimate. The class label fusion uses a naive Bayes combination of per-sensor class probabilities, conditional on independence between sensor modalities -- an assumption that holds well enough in practice when sensors differ in physical measurement principle. The result is described in detail in our coverage of multimodal fusion AI for ISR sensor streams.

Radar contributes most to the fusion architecture through its velocity measurement precision: Doppler-capable radars can resolve target radial velocity to within 0.1 m/s, which is far tighter than the velocity estimate derivable from thermal track displacement across frames. This velocity prior substantially improves track association during occlusion periods when neither the thermal nor EO sensor has a valid detection, by allowing the tracker to predict the next detection location from the radar-derived velocity. On platforms where a radar is not available, IMU-aided track prediction using platform motion compensation from the carrier's inertial navigation system partially substitutes for external velocity measurement when target kinematics are relatively predictable.

Edge deployment: quantization constraints on thermal models

Deploying a thermal classification model on a UAV payload or vehicle-mounted edge compute board imposes hard constraints on model size, inference latency, and power consumption that do not apply in cloud or server environments. The standard deployment path is INT8 post-training quantization followed by compilation to a hardware-optimized inference runtime such as TensorRT (for NVIDIA Jetson), ARMNN (for Cortex-A class processors), or TFLite with XNNPACK (for general ARM hardware). INT8 quantization replaces 32-bit floating-point weights and activations with 8-bit integers, reducing model size by 4x and inference time by 2–4x on hardware with INT8 accelerator support, with a typical accuracy penalty of 1–3% on classification accuracy metrics.

Thermal models are somewhat more tolerant of quantization than RGB models because the input dynamic range is narrower: after preprocessing, a thermal frame occupies a comparatively uniform range of pixel values without the extreme outlier values that RGB images can exhibit (direct specular reflections, saturated highlights). This narrower input distribution means the quantization scale factors computed during calibration are more accurate, and the accuracy loss from INT8 quantization for thermal classification is often at the low end of the typical range. For models that use attention mechanisms -- such as small Vision Transformer variants -- INT8 quantization degrades softmax and normalization operations more severely than it affects convolutions, requiring mixed-precision quantization (INT8 for convolutions, FP16 for attention layers) to preserve accuracy within 2% of the full-precision baseline. The same quantization workflow used for EO-camera ISR drone payloads applies to thermal payloads with the preprocessing stage adapted for single-channel NUC output.

Memory bandwidth is a frequently underestimated bottleneck for thermal edge deployment. A 640x512 thermal frame at 16-bit depth is 655 KB per frame. At 30 Hz, the raw data rate is approximately 20 MB/s. On a Jetson Orin Nano with a unified memory architecture, competing demands from the capture driver, preprocessing pipeline, model inference, and track management can saturate the available memory bandwidth before compute capacity is exhausted. Profiling memory bandwidth utilization, not just GPU occupancy, is necessary for identifying the real throughput ceiling and determining whether frame-rate reduction, spatial subsampling, or asynchronous pipeline staging is the appropriate mitigation.

Evaluation datasets and performance metrics for military thermal AI

Validating a thermal classification model for military use requires datasets and metrics that reflect operational conditions rather than the controlled laboratory settings common in academic benchmarks. The relevant performance dimensions are probability of detection (Pd) at a target false alarm rate (FAR), not the top-1 accuracy familiar from ImageNet evaluation. A model that achieves 95% top-1 accuracy on a balanced test set may still have a FAR of 10 per hour on a live thermal stream if the training set underrepresents the diversity of real background clutter conditions.

Publicly available datasets for military thermal AI include the KAIST multispectral pedestrian dataset (paired LWIR and RGB, 95,000 annotated frames, pedestrian-focused), the FLIR ADAS dataset (driving scenarios, not optimized for military targets), and the LITIV dataset (indoor and outdoor multi-modal sequences). For vehicle and weapon classification specifically, operational military datasets are classified and not publicly available; research groups typically use synthetically generated thermal imagery from physics-based simulation tools (DI-Guy, CAMEO-SIM, or VIRSuite) to augment limited real-data collections. Simulation-to-real transfer fidelity -- the degree to which models trained on synthetic thermal data generalize to real sensor output -- is an active research area; current best practice combines synthetic pretraining with fine-tuning on a smaller real-data set using domain adaptation techniques such as adversarial feature alignment.

The standard reporting metrics for military thermal detection systems are the receiver operating characteristic (ROC) curve with Pd plotted against FAR expressed in false alarms per unit time on a live stream, the F2 score (which weights recall twice as heavily as precision, appropriate for threat detection where missed detections are more costly than false alarms), and range-dependent Pd curves that characterize detection probability as a function of target-to-sensor range. For track-level fusion systems, additional metrics track the track initiation latency (frames from first target appearance to confirmed track), track continuity under occlusion, and the accuracy of fused position estimates compared to ground truth from GPS-instrumented test vehicles.