Synthetic Data for Defense AI: Training Models Without Classified Datasets

Training high-performance computer vision models for defense applications requires large, diverse, and precisely annotated datasets. The challenge is that operationally relevant training data — imagery of military vehicles, weapons systems, personnel, and tactical environments — is frequently classified, access-controlled, or simply does not exist in sufficient volume and diversity for deep learning. A model trained on a few hundred images of a specific vehicle type will underperform dramatically compared to one trained on tens of thousands of examples covering multiple lighting conditions, seasonal environments, partial occlusion, and sensor modalities.

Synthetic data generation addresses this bottleneck by creating photorealistic training imagery computationally, with automatic annotation, at a scale that real-world collection cannot match. The field has matured significantly: modern game engines running on GPU clusters can generate tens of thousands of precisely annotated training images per hour, complete with ground-truth bounding boxes, segmentation masks, depth maps, and sensor-specific rendering. The critical engineering challenge is not generating synthetic data — it is generating synthetic data diverse and realistic enough that models trained on it transfer effectively to real sensor imagery.

Why Real Defense Data Is Insufficient

The data scarcity problem in defense AI has multiple structural causes. Classification restrictions mean that the most operationally relevant imagery — footage of adversary equipment, tactical engagements, and sensitive geographic areas — cannot be widely distributed to training pipelines even within a defense organization. Legal and operational constraints limit the collection of training data from exercises. The annotation burden is severe: a single EO sensor dataset from a week-long exercise may contain thousands of hours of video, but extracting meaningful labeled samples requires expert annotators who understand military vehicle taxonomy, behavior patterns, and operational context.

Equipment rarity compounds the problem. The specific vehicle and equipment types that a target detection model must recognize are often produced in small quantities, not commonly visible in open-source imagery, and too sensitive to photograph for training purposes. A model that needs to recognize a specific armored fighting vehicle variant may have access to fewer than 50 real-world training examples — far below the thousands required for robust detection across the range of operational conditions.

Sensor modality gaps present a further challenge. Defense detection models frequently need to operate across EO, IR, SAR, and hyperspectral sensors, but training datasets in non-EO modalities are particularly sparse. Generating real LWIR or SAR imagery of military vehicles at scale, with controlled ground truth, is operationally impractical. Synthetic generation fills this gap directly: the same scene can be rendered simultaneously in EO, LWIR, and SAR-approximate modalities from the same 3D asset, providing matched multi-modal training pairs that would be impossible to collect operationally.

Game Engine Pipelines: Unreal Engine 5 and CARLA

Unreal Engine 5 has become the dominant platform for high-fidelity defense synthetic data generation. Its Nanite virtualized geometry system supports sub-centimeter geometric detail in vehicle and terrain meshes, while the Lumen global illumination system produces physically accurate lighting that adapts correctly to time-of-day, weather, and atmospheric conditions. For defense applications, the key UE5 capabilities are: procedural terrain generation using the Landscape system with realistic elevation data imported from SRTM or military topographic sources; foliage and vegetation scattering at mission-scale areas; dynamic weather and lighting that randomizes sun angle, cloud cover, haze, and precipitation across training batches; and programmatic scene control via Python scripting that allows fully automated generation of training scenarios without manual scene setup.

A production synthetic data pipeline for vehicle detection typically operates as follows: a library of high-fidelity 3D vehicle models (built from reference photographs, technical drawings, or CAD data) is combined with procedurally generated terrain environments. Python scripts randomize vehicle position, orientation, scale variation, and grouping. Lighting conditions, weather parameters, and camera altitude/angle are varied independently. For each generated frame, the engine exports both the rendered image and its corresponding annotation file — bounding boxes, segmentation masks, and instance labels — in YOLO, COCO, or Pascal VOC format, depending on the training framework. A single GPU workstation can generate approximately 2,000–5,000 annotated frames per hour; a modest 8-GPU rendering cluster produces 16,000–40,000 frames per hour, enabling a training dataset of one million images to be generated in under a week.

CARLA, the open-source autonomous driving simulator built on Unreal Engine, provides an alternative starting point for ground-vehicle scenarios in urban and semi-structured environments. Its mature Python API, pre-built urban maps, and sensor simulation library (including LiDAR, radar, and camera models with configurable noise) make it well-suited for IED detection, checkpoint monitoring, and convoy tracking applications where structured road networks are present.

Domain Randomization: Making Synthetic Data Generalizable

Domain randomization is the core technique that makes synthetic-to-real transfer work. The underlying principle is that if a model is trained on synthetic data with sufficient variation in all visual parameters that differ between the synthetic and real domains — lighting, textures, backgrounds, noise, sensor characteristics — the model will learn features robust enough to generalize to real imagery, because no single synthetic configuration is privileged.

In practice, domain randomization for defense computer vision randomizes: texture appearance of target vehicles (weathering level, camouflage pattern, dust, mud, thermal signature variation for IR models); background environment (terrain type, vegetation density, urbanization, road surface); lighting conditions (time of day, sun azimuth and elevation, sky state from clear to heavy overcast, artificial illumination for night scenarios); sensor parameters (focal length, altitude, gimbal angle, blur, compression artifacts, noise level); and target configuration (vehicle orientation, grouping, partial occlusion by terrain and vegetation, loading state for trucks and APCs).

Research has quantified the randomization coverage required for reliable sim-to-real transfer. Insufficient randomization — training with fixed backgrounds or single lighting conditions — produces models that perform well on the synthetic test set but fail on real imagery. Excessive randomization beyond the plausible distribution of real conditions can also degrade performance by forcing the model to generalize across configurations that never occur operationally. The practical approach is guided randomization: distributions informed by the expected operational environment (desert vs. European mixed terrain vs. urban), target sensor parameters, and seasonal conditions relevant to the deployment theater.

GAN and Diffusion Model Augmentation

Generative Adversarial Networks and diffusion models provide a complementary augmentation pathway that operates at the pixel level rather than the scene level. Where game engine pipelines generate full synthetic scenes, GANs and diffusion models can modify existing imagery — both synthetic and the limited real imagery available — to produce additional training variants.

CycleGAN-based domain transfer is used to convert photorealistic synthetic EO imagery into LWIR-approximate representations, bridging the sensor modality gap without requiring separate LWIR rendering of all scenes. The approach trains a CycleGAN on paired or unpaired EO/LWIR image sets and then applies the learned transformation to the full synthetic EO dataset, producing pseudo-LWIR training data at scale. While not identical to real LWIR imagery, CycleGAN-generated pseudo-LWIR provides sufficient domain coverage to bootstrap IR detection models that would otherwise lack training data entirely.

Diffusion model-based augmentation addresses the texture and appearance diversity problem. A diffusion model fine-tuned on real vehicle imagery can generate new texture variations of synthetic vehicles — applying realistic camouflage patterns, weathering, and environment-appropriate coloration — without requiring manual 3D texture painting. The SDXL architecture adapted for industrial applications has shown particular promise for generating diverse military vehicle texture variants from textual conditioning prompts describing camouflage patterns, operational wear, and environmental conditions.

Sim-to-Real Gap: Validation and Closing Techniques

The sim-to-real gap quantifies the performance degradation observed when a model trained entirely on synthetic data is evaluated on real imagery. For well-executed synthetic pipelines with comprehensive domain randomization, this gap typically manifests as a 5–20 percentage point reduction in mean average precision (mAP) on real imagery compared to a model trained on an equivalent number of real annotated images. In many defense applications, this performance level is operationally acceptable, particularly when real training data is simply unavailable.

Several techniques reduce the sim-to-real gap below acceptable thresholds. Fine-tuning with a small real dataset (as few as 100–500 carefully annotated real images) after initial synthetic training dramatically reduces the gap: the synthetic pre-training provides a strong feature initialization, and the small real fine-tuning set adapts those features to the real domain without the large annotation burden of training from scratch on real data. This hybrid approach — large-scale synthetic pre-training plus small-scale real fine-tuning — is the current best practice for defense object detection when real data access is constrained.

Neural rendering approaches, particularly NeRF (Neural Radiance Fields) and its successors (Instant-NGP, 3D Gaussian Splatting), offer a novel path to closing the synthetic gap. NeRF models trained on a small number of real photographs of a target vehicle can synthesize novel viewpoints, lighting conditions, and partial occlusion states that were not present in the original photographs, effectively expanding a dataset of 50 real images into thousands of synthetic variants while preserving real-world appearance fidelity. This approach bypasses the need for high-quality 3D artist assets entirely.

Classification and Handling of Synthetic Training Datasets

An important but often overlooked consideration in defense synthetic data programs is the classification status of the generated datasets themselves. Synthetic imagery of non-existent scenarios using generic vehicle models is generally unclassified. However, synthetic imagery generated from classified vehicle models, realistic maps of sensitive geographic areas, or operational scenarios derived from classified intelligence may inherit classification requirements. Programs must establish data governance procedures that define classification rules for synthetic datasets based on their input asset provenance and scenario content, maintaining the security benefits of synthetic data while managing the classification burden that would otherwise block model distribution to edge deployment hardware.

The operational chain for a mature synthetic data program runs: 3D asset library (classification-reviewed) → procedural scene generation (automated, GPU cluster) → annotation export (YOLO/COCO format) → quality validation (automated detection confidence checks, human spot inspection) → model training (YOLOv8/v9 or DINO-based detector) → real-data fine-tuning (if available) → performance validation on held-out real imagery → TensorRT deployment package for edge hardware. Each step has associated security controls, and the entire pipeline can be executed within a classified enclave if required by the sensitivity of the 3D assets used.