Secrets Management in Defense CI/CD Pipelines: Vault, HSM, and Key Rotation

Every defense software system depends on secrets: TLS certificates that authenticate service identities, API keys that authorize access to external services, database passwords that protect operational data, encryption keys that protect classified information, and code-signing keys that verify firmware and software integrity. If these secrets are handled carelessly — stored in configuration files, checked into source code repositories, passed as environment variables in plaintext, or left unchanged indefinitely — they become attack vectors that circumvent all other security controls.

Secrets management in defense CI/CD pipelines is not primarily about preventing developer mistakes (though it does that too). It is about implementing a security architecture where secrets are never in plaintext outside of their authorized use context, where every secret use is audited, where secrets have defined lifetimes and are automatically rotated, and where the compromise of any individual secret has a limited blast radius because of short expiry and scoped access.

Secret Types in Defense Pipelines

Defense CI/CD pipelines encounter a wider variety of secret types than commercial equivalents, because the systems they deploy have stricter access control and authentication requirements:

TLS certificates authenticate service identities in mTLS deployments and protect network communications. In a defense Kubernetes cluster with a service mesh, each microservice has its own certificate, potentially thousands of certificates total, all requiring rotation before expiry. The certificates for external-facing services must chain to an authorized CA; internal service certificates may chain to an internal CA managed by Vault or the service mesh control plane.

API keys and access tokens authorize service-to-service calls, access to external threat intelligence feeds, SIEM API access, and integration with classified system backends. These are typically static secrets (not dynamically generated) and are the most likely to appear in source code repositories through developer error.

Database credentials protect access to operational and classified databases. Static database credentials — a username and password pair that never changes — are a significant security risk: if the credential is compromised, access persists until the credential is manually rotated, which may not happen for months or years. Dynamic credentials with short lifetimes are significantly more secure.

Code signing keys are used to sign software releases, container images, firmware packages, and configuration bundles. These keys are extremely sensitive — a compromised code signing key allows an attacker to sign malicious code that will be trusted by all systems that trust the key. Code signing keys should be protected by HSM hardware and should require multi-party authorization for use.

HashiCorp Vault: Dynamic Secrets and Audit Log

HashiCorp Vault is the standard secrets management platform for defense DevSecOps environments. Vault provides a centralized, audited store for secrets, a unified API for secret retrieval, and a dynamic secrets engine that generates time-limited, purpose-specific credentials rather than requiring applications to store long-lived static secrets.

Dynamic secrets are Vault's most powerful security feature. Instead of storing a static database password that applications retrieve, Vault dynamically generates a new database user with a time-limited credential each time an application requests access. The credential automatically expires and the database user is revoked after the lease period (typically 1–24 hours). An attacker who captures a dynamic database credential has a narrow window to exploit it before it expires; an attacker who captures a static credential has indefinite access until the credential is manually rotated.

Vault's dynamic secrets engines cover PostgreSQL, MySQL, MSSQL, MongoDB, Cassandra, Elasticsearch (for log management), PKI (certificate issuance), AWS IAM (cloud credentials), and more. For defense environments, the PKI secrets engine — which enables Vault to act as an intermediate CA issuing short-lived TLS certificates on demand — is particularly valuable: certificates issued with 24-hour TTLs eliminate the window for certificate abuse if a certificate is compromised.

Vault's audit log records every API call to Vault: every secret request, every authentication attempt, every policy lookup. The audit log is append-only and cannot be modified by Vault administrators. For defense environments, the audit log provides the evidence trail required by accreditation: who accessed what secret, when, and from what system. Vault audit logs should be forwarded to the SIEM for correlation with application and network events.

Hardware Security Modules: When Software Vault Is Insufficient

HashiCorp Vault secures its own encryption keys (the master key used to unseal Vault and encrypt its secret store) using a software-based key encryption approach. For most environments, this is adequate security. For defense environments with FIPS 140-2 Level 3 requirements — which apply to National Security Systems and systems handling classified cryptographic key material — the root keys must be protected by hardware, not software.

FIPS 140-2 Level 3 requires tamper-evident physical security, identity-based authentication for operators, and critical security parameters (private keys) that are not exported in plaintext at any time. Software-based key stores, however well-encrypted, do not satisfy this requirement because the encryption key itself exists in software memory where it is potentially accessible to a privileged software attacker.

HSM auto-unseal for Vault is the standard architecture: Vault's master key is wrapped by an HSM key, and Vault auto-unseals by sending its wrapped master key to the HSM for unwrapping on startup. The HSM performs the decryption within its tamper-resistant boundary — the master key never exists in plaintext outside the HSM. This architecture satisfies FIPS 140-2 Level 3 requirements for the root key protection layer.

Supported HSM integration for HashiCorp Vault Enterprise includes Thales (formerly SafeNet) Luna, nCipher nShield, AWS CloudHSM, and Azure Dedicated HSM. For air-gapped defense environments, the on-premises HSM options (Thales Luna, nCipher nShield) are required — cloud-based HSM services are not accessible from air-gapped networks.

Key Rotation Without Downtime

Key rotation — replacing an existing cryptographic key with a new one — is essential for security hygiene: it limits the exposure window if a key is compromised and satisfies regulatory requirements for key lifetime limits. But key rotation that requires application downtime or complex manual coordination is often deferred indefinitely, negating its security value.

Vault's key versioning enables zero-downtime rotation for secrets and encryption keys. When Vault's transit encryption engine (which provides encryption-as-a-service for applications) rotates a key, it creates a new key version while retaining older versions for decryption of data encrypted with previous versions. Applications encrypting new data use the current key version; existing ciphertext can still be decrypted using the old version until the application is updated to re-encrypt it. The rotation is transparent to the application — it does not need to be taken offline.

Grace periods for certificate rotation allow for the gradual rollout of new certificates: the new certificate is distributed and trusted before the old certificate is revoked, so that there is no window where some services have the new certificate and others have not yet received it. Vault's PKI secrets engine supports configuring certificate TTLs and renewal periods to automate this grace period management.

CI/CD Integration: Secret Injection Patterns

Secrets must reach the applications and infrastructure components that need them without being exposed in CI/CD pipeline logs, environment variable dumps, or container image layers. Three integration patterns dominate defense CI/CD environments:

Vault agent sidecar (in Kubernetes) deploys a Vault Agent container alongside the application container. The Vault Agent authenticates to Vault using the pod's Kubernetes service account identity, retrieves the configured secrets, and writes them to a shared in-memory volume or injects them into the application container's environment. The secrets never appear in the pod specification or container image — they are injected at runtime from Vault.

External Secrets Operator is a Kubernetes operator that synchronizes secrets from external secret stores (Vault, AWS Secrets Manager, Azure Key Vault) into Kubernetes Secrets. It provides a declarative, GitOps-compatible approach: the ExternalSecret custom resource in the GitLab/Kubernetes configuration declares what secrets are needed and where they come from; the operator handles retrieval and synchronization. The Kubernetes Secrets created by the operator can be mounted into pods normally.

For GitLab CI pipelines, the GitLab-Vault integration (GitLab CI/CD Vault integration, using JWT authentication) allows pipeline jobs to authenticate to Vault using their GitLab JWT identity and retrieve secrets for the duration of the pipeline job. Secrets are available as environment variables within the job and are never stored in the GitLab CI configuration or repository.