Software quality in defense programs is not just a development best practice — it is a mission readiness issue. Code deployed to command and control systems, logistics platforms, and intelligence tools where defects cause operational failures or security vulnerabilities has consequences that commercial software quality failures don't. Test automation in this context is the mechanism that makes high confidence in software quality achievable at delivery speed.
Why Manual Testing Is Not Sufficient for Modern Defense Software
Traditional defense software programs relied heavily on formal test events — independent test teams executing scripted test cases at program milestones. This model fails in three ways for modern DevSecOps programs:
Speed mismatch: Agile programs push sprint deliverables every 2 weeks. A manual test team executing a regression test suite takes weeks. Manual-only testing creates a testing bottleneck that defeats the continuous delivery objective.
Coverage limitations: Manual testers executing scripted cases cannot achieve meaningful coverage of complex system behaviors. Automated regression suites run thousands of test cases in hours that a manual team could not complete in days.
Repeatability and consistency: Human testers tire, miss steps, and apply different judgment across test executions. Automated tests execute identically on every run.
The solution is not to eliminate human testers — it is to shift human testing toward exploratory, scenario-based, and usability testing where human judgment adds value, while automating the regression, integration, and security validation layers.
DoD Software Testing Policy Context
The DoD Software Modernization Strategy and the DoD DevSecOps Reference Design both address testing automation as part of the Software Factory model:
Continuous testing in the CI/CD pipeline: The DoD DevSecOps model requires automated testing as a gate in every pipeline stage. Code that fails unit tests, static analysis, or integration tests does not advance to staging or production.
Independent Verification and Validation (IV&V): Many DoD programs require IV&V — testing by an organization independent from the development team. Modern IV&V is increasingly automation-augmented: the IV&V team reviews test automation architecture, code review, and coverage metrics in addition to executing independent test cases.
JITC testing: The Joint Interoperability Test Command (JITC) provides interoperability testing for DoD programs — ensuring new systems integrate correctly with existing DoD networks, protocols, and systems. JITC test plans include automated test tools for protocol compliance validation.
Building a Defense Software Test Automation Strategy
Test Pyramid for Defense Systems
A well-structured test suite follows the test pyramid principle:
Unit tests (broad base): Test individual functions, classes, and modules in isolation. Should cover the majority of the codebase. Fast to run (milliseconds per test), run on every commit. Target >80% code coverage for mission-critical components.
Integration tests (middle layer): Test interactions between components — API contracts, database interactions, message queue processing, external service integrations. Run in pre-commit or CI environment. May require test containers (database instances, message brokers) in CI.
End-to-end tests (narrow top): Test complete user workflows through the system. Expensive to maintain and slow to run — keep the count manageable. Focus on highest-value workflows.
Security tests (woven throughout): Static analysis (SAST) at unit level, dynamic analysis (DAST) at integration and E2E level, software composition analysis (SCA) at build time.
Security Test Automation
For defense programs specifically, security test automation is non-negotiable:
SAST (Static Application Security Testing): Tools like Fortify, Checkmarx, or SonarQube analyze source code for security vulnerabilities without executing the code. Run on every pull request, blocking merge on high and critical findings.
DAST (Dynamic Application Security Testing): Tools like OWASP ZAP or Burp Suite Enterprise scan the running application for exploitable vulnerabilities — injection attacks, authentication weaknesses, insecure configurations. Run against staging environments.
Software Composition Analysis (SCA): Defense programs are particularly scrutinized for supply chain risks. SCA tools (Black Duck, Snyk, Dependabot) inventory all open-source dependencies, match against known vulnerability databases (NVD, OSV), and flag components with license compliance issues. The Defense Supply Chain Software Procurement Act and related DoD guidance increasingly require SBOM (Software Bill of Materials) documentation.
Container image scanning: Programs using containerized deployments must scan container images for vulnerabilities. Trivy, Grype, or Prisma Cloud scan images in the CI pipeline, blocking deployment of images with unacceptable vulnerabilities.
Test Data Management in Classified Environments
One of the most challenging aspects of defense testing automation is test data. Production data from classified systems cannot be used in development and test environments. Solutions:
- Synthetic data generation: Generate realistic test data using data synthesis tools that produce statistically representative data without containing real individuals, operational details, or classified information
- Data masking: Transform production data by replacing sensitive values with masked equivalents before importing to test environments — preserving data relationships while removing sensitive content
- Reference data environments: For systems that integrate with authoritative DoD data sources, use authorized test instances of those source systems rather than copies of production data
Test Coverage Metrics for DoD Programs
Standard coverage metrics — line coverage, branch coverage — measure what code is executed during testing but don't assess test quality. For defense programs, also track:
- Mutation testing score: Automated mutation testing introduces bugs into the code and verifies that existing tests catch them. High mutation scores indicate tests are effectively finding real defects.
- SAST finding remediation rate: Percentage of SAST findings resolved per sprint, and average age of open findings by severity
- Regression defect rate: Defects found in testing that represent regressions from previously passing functionality — a high regression rate indicates insufficient automated regression coverage
Rutagon builds and operates test automation programs for defense and government software delivery organizations. Contact us to discuss test automation strategy for your program.
Frequently Asked Questions
What is the difference between IV&V and standard testing for DoD programs?
IV&V (Independent Verification and Validation) is testing performed by an organization independent from the development team — typically a separate contract or government function. IV&V verifies that the system is built correctly (verification) and validates that it meets mission requirements (validation). IV&V is not a replacement for the development team's own test automation; it provides an independent assessment layer. The DoD Software Acquisition Policy allows IV&V in both traditional waterfall and agile delivery contexts.
What SAST tools are approved for DoD programs?
Fortify (Micro Focus) is one of the most widely used SAST tools in DoD programs, with approved use on classified systems. Checkmarx is another widely deployed option. SonarQube is used in some DoD Software Factories. The specific tools allowed depend on the program's security authorization and the software factory's approved tool inventory. Check the applicable STIG and security guidance for your specific program environment.
Is open-source software allowed in defense programs?
Yes, with risk management. The DoD has issued explicit guidance supporting open-source software use in defense programs, with requirements for license compliance review, vulnerability management, and SBOM documentation. Open-source components with restrictive licenses (GPL copyleft) require legal review. Components with known exploitable vulnerabilities must be updated or replaced. SCA tooling handles the automated tracking and alerting for this requirement.
How does software testing integrate with the ATO process?
The System Security Plan (SSP) documents how SA&A (Security Assessment and Authorization) control requirements are met. Test automation addresses specific controls — particularly those in the SA (System and Services Acquisition) and SI (System and Information Integrity) control families. Evidence of automated security testing (SAST/DAST results, SCA scans, penetration test reports) is submitted to the authorizing official as part of the security assessment package. Continuous testing results feed ongoing Continuous Monitoring (ConMon) activities post-ATO.