Government IT programs historically define reliability through availability requirements in the Statement of Work: "the system shall maintain 99.9% availability" appears in countless government contracts. But stating an availability target in a contract document is not the same as engineering toward that target with a mechanism that governs delivery tradeoffs.
Site Reliability Engineering's error budget model provides a rigorous framework for quantifying reliability targets, measuring performance against them, and making operationally sound decisions when systems degrade. Applied to federal cloud systems, error budgets align engineering behavior with program reliability commitments.
What an Error Budget Is
An SLO (Service Level Objective) is a measurable reliability target. An error budget is the inverse: the maximum amount of unreliability your SLO permits.
If your SLO is 99.9% availability over a rolling 30-day window:
Total minutes in 30 days: 43,200
99.9% availability = 99.9% of 43,200 minutes must be "good"
Error budget = 0.1% of 43,200 = 43.2 minutes of allowed downtime per 30-day windowThe error budget is not a target — it is a constraint. Teams that spend their error budget on incidents have no remaining budget for planned maintenance or risky deployments. Teams with full error budgets can move faster. This creates a direct, measurable link between system reliability and deployment velocity.
Defining SLOs for Federal Programs
Government IT SLOs must be grounded in two inputs: user experience requirements (what does degraded service actually cost users?) and contractual commitments (what does the Statement of Work require?).
SLO Categories for Government Systems
Availability SLO: The percentage of time the system is serving valid requests without returning server errors.
SLO: 99.9% availability measured over a rolling 30-day window
SLI (measurement): (valid_requests - server_errors) / valid_requests
Error budget: 43.2 minutes per 30 daysLatency SLO: The percentage of requests served within a specified latency threshold.
SLO: 95% of requests served within 500ms, 99% within 2,000ms
SLI (measurement): histogram_quantile(0.95, http_request_duration_seconds)
Error budget: 5% of requests may exceed 500ms; 1% may exceed 2,000msData Processing Timeliness SLO: For batch or stream processing systems common in government programs.
SLO: 99% of data records processed within 15 minutes of receipt
SLI (measurement): (records_processed_within_15m) / total_records_received
Error budget: 1% of records may take longer than 15 minutesCompliance Overlay
Federal systems may have contractual availability requirements that set a floor on SLO targets. If the contract requires 99.5% availability, your SLO should be at or above 99.5% — but you can choose a more ambitious internal target (99.9%) and use the gap as your safety margin.
FISMA FIPS 199 impact levels also influence appropriate SLO targets:
- Low impact: 99.5% availability may be appropriate
- Moderate impact: 99.9% is a reasonable baseline
- High impact: 99.95%+ with disaster recovery objectives defined
Document SLO targets in the System Security Plan's availability commitment section (SC-6, System Use Notification is insufficient — the relevant control is CP-8 Telecommunications Services and CP-2 Contingency Plan).
Error Budget Policies
An error budget is only useful if it changes behavior. Define explicit policies for what happens when the error budget reaches specific thresholds:
# error-budget-policy.yaml
slo_target: 99.9%
measurement_window: 30d
policies:
budget_remaining_100_to_50_percent:
deployment_velocity: normal
risky_changes: allowed
maintenance_windows: as_scheduled
budget_remaining_50_to_10_percent:
deployment_velocity: normal
risky_changes: require_additional_approval
maintenance_windows: defer_non_critical
note: "Alert engineering leadership. Review recent incidents for systemic causes."
budget_remaining_under_10_percent:
deployment_velocity: freeze_non_critical_deployments
risky_changes: blocked
maintenance_windows: emergency_only
escalation: notify_program_manager_and_co
note: "SLO at risk. All engineering focus on reliability."
budget_exhausted:
deployment_velocity: freeze_all_feature_work
risky_changes: blocked
escalation: trigger_incident_review_board
reporting: include_in_monthly_status_report_to_governmentFor federal programs, the policy should align with reporting requirements. A system burning through its error budget is a reportable status item — the contracting officer and government program manager should know before the SLA violation triggers a formal finding.
Alert Architecture: SLO-Based Alerting
Traditional threshold-based alerts (alert when CPU > 80%) don't map directly to user experience or reliability targets. SLO-based alerting fires when the error budget is burning at an unsustainable rate.
Burn rate alerts detect when you're spending your error budget faster than it's being replenished:
# Burn rate alert logic (pseudo-code for monitoring configuration)
# Alert if the current burn rate would exhaust the budget in < 2 hours
ERROR_BUDGET_MINUTES = 43.2 # 99.9% SLO over 30 days
ALERT_THRESHOLD_HOURS = 2
def calculate_burn_rate(error_rate_5min: float, error_budget: float) -> float:
"""
Returns how many times faster than 'budget-neutral' we're burning.
Burn rate of 1.0 = exactly spending budget at the rate it replenishes.
Burn rate of 10.0 = spending 10x faster than replenishment.
"""
budget_per_minute = error_budget / (30 * 24 * 60)
return error_rate_5min / budget_per_minute
# Alert when burn rate > 14.4x (would exhaust 30-day budget in 2 hours)
FAST_BURN_ALERT_THRESHOLD = 14.4This approach from Google's SRE workbook (sre.google/workbook) creates alerts that are both sensitive to genuine reliability problems and resistant to false positives from brief transient spikes.
Implement the alerting in AWS CloudWatch Alarm expressions or in your SIEM, and route alerts to both engineering and (for federal programs) to a monitoring dashboard visible to the government COR or designated monitor.
Connecting to Continuous Monitoring Requirements
NIST 800-53's CA-7 (Continuous Monitoring) requires ongoing assessment of control effectiveness. SLO-based reliability monitoring satisfies the availability dimension of continuous monitoring for operations controls (CP controls):
- CP-10 (System Recovery and Reconstitution): SLO measurement data demonstrates actual recovery times versus the RTO targets in the contingency plan
- SI-2 (Flaw Remediation): Error budget policies that freeze deployments during budget exhaustion create a natural control gate that prevents deploying flawed updates when the system is already degraded
- IR-4 (Incident Handling): Incident response can be triggered by automated error budget alerts, creating a documented, auditable response initiation
For ATO evidence packages, SLO dashboards exported to a compliance evidence repository (as described in our continuous ATO automation approach) satisfy assessor requests for evidence of operational monitoring.
Production Patterns for GovCloud
In AWS GovCloud, implement SLO measurement and error budget tracking using:
- CloudWatch Metrics: Custom metrics namespace for SLI measurements. CloudWatch Math Expressions calculate error rates from raw request/error counts
- CloudWatch Dashboards: Per-service SLO dashboard with current error budget displayed as a percentage and absolute time remaining
- CloudWatch Alarms: Burn rate alarms using metric math expressions as described above, with SNS targets for PagerDuty/OpsGenie integration
- S3 + Athena: Long-term SLO history stored in S3 Parquet files. Athena queries generate monthly SLO reports suitable for inclusion in government program status reports
This observability architecture integrates with the OpenTelemetry distributed tracing patterns we use for deep system visibility.
Frequently Asked Questions
How does SRE error budget relate to the SLA penalties in a government contract?
An SLA (Service Level Agreement) is a contractual commitment with associated remedies — penalties, credits, or cure obligations if service falls below the specified level. An SLO is an internal engineering target that should be set above the SLA threshold, creating a buffer. If your SLA requires 99.5% availability and your internal SLO is 99.9%, burning through your error budget triggers an engineering response before you breach the SLA. SLAs govern the contract relationship; SLOs govern engineering behavior.
Can we apply SRE practices to batch processing systems that don't have HTTP request metrics?
Yes. SLOs for batch systems define reliability in terms of the system's outputs: percentage of jobs completing successfully, percentage of records processed within the time window, percentage of reports generated before their deadline. The SLI is measured against these output characteristics rather than HTTP request success rates. For government data processing pipelines, define SLOs around data processing timeliness and completeness rather than infrastructure availability.
How do we report SLO performance to government stakeholders?
Monthly operational status reports to government CORs and program managers should include: current SLO performance over the reporting period, cumulative error budget consumption, incidents that consumed budget with brief root cause summaries, and forward-looking reliability risk indicators. Most government program management frameworks already require monthly status reports — SLO data provides a quantitative reliability section that replaces vague narrative descriptions with measurable performance data.
What happens when the government's contractual requirements conflict with the error budget policy?
If the government's Statement of Work requires 99.9% availability and your system experiences an outage that threatens that target, contract obligations supersede the error budget policy. The error budget is an engineering management tool — contract obligations are legal requirements. The value of maintaining an error budget in federal programs is that it creates advance warning. A well-designed error budget policy triggers escalation to the program manager before you breach the contractual threshold, giving time for remediation before formal reporting is required.
How many SLOs should a federal system define?
Start with two or three SLOs that directly measure what government users experience: availability, latency, and data processing timeliness (if applicable). Avoid defining SLOs for every internal metric — more SLOs create noise and dilute engineering focus. As the system matures, add SLOs for specific high-priority user journeys or for components identified as reliability risks in past incidents. The goal is a small number of SLOs that, together, accurately represent whether the system is serving users well.
Rutagon engineers reliable, compliant cloud systems for federal programs →
Discuss your project with Rutagon
Contact Us →