Spot Instances offer discounts of 60-90% off On-Demand pricing — and most engineering teams avoid them entirely because of one bad experience with an interrupted workload taking down production. The discount is real. So is the interruption risk. The engineering work is in architecting around it, not avoiding Spot altogether.
Where We Recommend Spot (and Where We Don't)
Not every workload belongs on Spot. We evaluate three factors before recommending it to a client:
- Statelessness — can the workload be interrupted and resumed elsewhere without data loss?
- Time flexibility — does the job tolerate a delay of minutes to hours, or does it need guaranteed completion by a specific time?
- Horizontal scalability — can the workload run across a diversified pool of instance types and Availability Zones?
Good Spot candidates: batch data processing, CI/CD build runners, non-critical background jobs, stateless web tiers behind a load balancer, machine learning training jobs with checkpointing.
Poor Spot candidates: primary databases, stateful session servers without external state storage, anything requiring guaranteed sub-second availability with no fallback path.
The Architecture: Diversification + Graceful Interruption
The core mistake we see in failed Spot adoptions is running a single instance type in a single Availability Zone. AWS reclaims Spot capacity based on demand for that specific instance type in that specific AZ — concentrating on one combination maximizes interruption risk.
Our standard pattern:
resource "aws_autoscaling_group" "spot_workers" {
mixed_instances_policy {
instances_distribution {
on_demand_base_capacity = 2
on_demand_percentage_above_base_capacity = 0
spot_allocation_strategy = "capacity-optimized"
}
launch_template {
launch_template_specification {
launch_template_id = aws_launch_template.worker.id
}
override {
instance_type = "m6i.large"
}
override {
instance_type = "m6a.large"
}
override {
instance_type = "m5.large"
}
}
}
min_size = 2
max_size = 20
vpc_zone_identifier = var.private_subnet_ids
}
Setting on_demand_base_capacity ensures a floor of guaranteed capacity stays running even during a broad Spot reclamation event, while the bulk of scaling capacity runs on discounted Spot across multiple instance families and AZs.
Handling the 2-Minute Interruption Notice
AWS gives a 2-minute warning before reclaiming a Spot instance via the instance metadata service. Production workloads should poll this endpoint and drain gracefully:
import requests
def check_spot_interruption():
try:
response = requests.get(
"http://169.254.169.254/latest/meta-data/spot/instance-action",
timeout=1
)
if response.status_code == 200:
drain_connections_and_checkpoint()
deregister_from_load_balancer()
except requests.exceptions.RequestException:
pass # no interruption pending
For containerized workloads on EKS, we use the AWS Node Termination Handler, which automates this drain-and-cordon behavior at the Kubernetes level rather than requiring custom polling logic in every service.
Results From a Recent Engagement
For a client running batch ETL jobs and CI/CD build fleets, moving eligible workloads to a diversified Spot pool cut that portion of the compute bill by roughly 70%, while build failure rates attributable to interruptions stayed under 2% — well within the tolerance for jobs that automatically retry on a different instance.
FinOps Is a Process, Not a One-Time Migration
Spot pricing and capacity availability shift over time. We set up ongoing monitoring — CloudWatch metrics on interruption rate, Cost Explorer tracking Spot vs. On-Demand spend ratio — so the strategy adapts rather than degrading silently as workloads or AWS capacity patterns change.
Want a free AWS cost review to find your own Spot opportunities? Talk to us — 907-841-8407 or contact@rutagon.com.
Frequently Asked Questions
How much can Spot Instances actually save compared to On-Demand?
Discounts typically range from 60-90% depending on instance type, region, and demand at the time. Actual savings depend on how much of your workload can tolerate interruption and how well the architecture diversifies across instance types and zones.
What happens if all my Spot capacity gets reclaimed at once?
A well-architected pool spread across multiple instance types and Availability Zones significantly reduces the chance of simultaneous reclamation, since AWS reclaims capacity per instance-type-per-AZ combination, not globally. Maintaining a small On-Demand base capacity provides an additional floor.
Can I run a production database on Spot Instances?
Generally no — primary databases need guaranteed availability and aren't a good fit for interruption-tolerant Spot capacity. Stateless application tiers, batch processing, and CI/CD runners are much better candidates.
How does Kubernetes handle Spot Instance interruptions?
The AWS Node Termination Handler watches for the 2-minute interruption notice and automatically cordons and drains the node, rescheduling pods elsewhere in the cluster before the instance is reclaimed.
Is Spot allocation strategy something I set once, or does it need ongoing tuning?
It benefits from ongoing monitoring. Capacity availability and pricing shift over time, so tracking interruption rates and cost ratios lets you adjust instance type diversity and On-Demand base capacity as conditions change.