Kubernetes gives teams enormous flexibility in how they deploy and scale workloads — and that same flexibility makes it remarkably easy to over-provision resources without anyone noticing until the cloud bill arrives. Here's the systematic approach we use to find and fix Kubernetes cost waste.
The Most Common Waste Pattern: Resource Request Inflation
Kubernetes schedules pods based on their resource requests, not their actual usage. When developers set generous requests "to be safe" without measuring real usage, the cluster reserves far more CPU and memory than workloads actually consume — and the cluster autoscaler adds nodes to accommodate those inflated reservations even when actual utilization is low.
# Before: guessed, inflated requests
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "4"
memory: "8Gi"
# After: based on measured p95 usage with headroom
resources:
requests:
cpu: "500m"
memory: "768Mi"
limits:
cpu: "1"
memory: "1Gi"
We use tools like the Vertical Pod Autoscaler in recommendation mode (not auto-apply mode, which can cause unexpected restarts) to generate data-driven request recommendations based on actual observed usage over time, rather than developer guesswork.
Cluster Autoscaling Configuration
Poorly tuned cluster autoscaler settings leave nodes running well below capacity. We tune scale-down thresholds and utilization targets so the autoscaler consolidates workloads onto fewer nodes during low-traffic periods, rather than leaving nodes half-empty indefinitely:
# Cluster Autoscaler configuration
scaleDownUtilizationThreshold: 0.5
scaleDownUnneededTime: 10m
scaleDownDelayAfterAdd: 10m
Spot/Preemptible Node Pools for Fault-Tolerant Workloads
Stateless, fault-tolerant workloads (batch jobs, CI runners, horizontally-scaled stateless services with graceful pod eviction handling) are strong candidates for spot node pools, typically at a significant discount versus on-demand pricing. We segment clusters into on-demand node pools for stateful or latency-critical workloads and spot pools for everything that tolerates interruption:
apiVersion: v1
kind: Node
metadata:
labels:
node-lifecycle: spot
workload-type: batch
Pod disruption budgets and proper terminationGracePeriodSeconds configuration matter more on spot pools, since nodes can be reclaimed with short notice — we make sure workloads scheduled there handle this gracefully rather than dropping requests.
Namespace-Level Cost Visibility
Teams can't optimize costs they can't see. We implement cost allocation tooling (such as Kubecost or OpenCost) that attributes cluster spend down to namespace, deployment, and even label level, giving engineering teams visibility into which specific services are driving cost — turning cost optimization from an infrastructure team's abstract concern into something individual service owners can act on directly.
Idle Environment Cleanup
Development and staging clusters often run 24/7 at full scale despite being used only during business hours. We implement scheduled scale-down for non-production environments — scaling deployments to zero replicas outside working hours via a CronJob-triggered scaler — cutting non-production compute costs.
# CronJob to scale down dev environment nightly
schedule: "0 20 * * 1-5"
command: ["kubectl", "scale", "deployment", "--all", "--replicas=0", "-n", "dev"]
Results
For a client running a mid-size Kubernetes cluster with no prior cost optimization work, this combination of measured resource requests, autoscaler tuning, spot node pools for batch workloads, and non-production scale-down reduced monthly Kubernetes infrastructure spend by roughly 40%, with no measurable impact on production performance or reliability.
Running Kubernetes and suspect there's meaningful cost waste? Get a cluster cost review — 907-841-8407 or contact@rutagon.com.
Frequently Asked Questions
What's the biggest source of Kubernetes cost waste?
Inflated resource requests set without measuring actual usage are typically the largest contributor — the cluster reserves and pays for capacity workloads never actually use, and the autoscaler adds nodes to accommodate those inflated reservations.
Is it safe to use spot/preemptible nodes in production?
Yes, for fault-tolerant, stateless workloads with proper pod disruption budgets and graceful termination handling. Stateful workloads or latency-critical services generally belong on on-demand node pools instead.
How do you get cost visibility into a shared Kubernetes cluster?
Tools like Kubecost or OpenCost attribute cluster spend down to the namespace, deployment, or label level, giving individual teams visibility into what their specific workloads cost rather than treating cluster spend as one undifferentiated infrastructure line item.
Should the Vertical Pod Autoscaler be used in auto-apply mode?
Generally we recommend recommendation mode rather than auto-apply, since automatic resource changes can trigger unexpected pod restarts. Recommendation mode still gives data-driven sizing guidance without that operational risk.
How much can Kubernetes cost optimization typically save?
It varies significantly by starting state, but clusters that haven't had dedicated cost optimization work often see 30-45% reductions from a combination of resource rightsizing, autoscaler tuning, spot node adoption, and non-production environment scheduling.