The CFO sees CloudWatch as “monitoring.” The staff engineer owns what the application actually emits. CloudWatch custom metrics cost is not log retention. It is the monthly charge for each unique metric series you publish, and high-cardinality dimensions are how a mid-market SaaS mint tens of thousands of series from one PutMetricData helper. Rutagon builds the emit contract: which dimensions stay, which become logs or traces, and which dashboards are allowed to demand a new series.
This article stays on custom metrics. If your bill is StorageBytes on log groups, that is a different line and a different fix — we already covered CloudWatch log retention cost. Mixing the two is how custom metrics stay unowned while someone shortens retention and wonders why the Metrics usage type did not move.
CloudWatch Custom Metrics Cost and High-Cardinality Dimensions
CloudWatch bills custom metrics by unique series. A series is the tuple of namespace, metric name, and the full dimension set. Latency with {service=checkout} is one series. Latency with {service=checkout, customerId=…} is one series per customer. If you also add requestId, you have built a unique series factory. The application feels honest. The bill is a cardinality product.
AWS documents how metrics and dimensions work in the CloudWatch metrics guide. The public pricing page lists custom metrics as a per-metric monthly meter, with a separate meter for API requests and another for high-resolution metrics. We do not quote a dollar figure. Regional rates change, and a “per metric” sentence without your series count is marketing, not a forecast. Open Cost Explorer, filter CloudWatch, and group by usage type that mentions metrics or API requests.
Standard resolution is one data point per minute. High-resolution metrics accept finer periods and sit on a higher meter. Teams turn on high resolution because a latency graph looked blocky. That is a product decision. Most mid-market SaaS SLOs do not need sub-minute custom series on every tenant dimension. Traces already have the high-cardinality path.
Embedded Metric Format (EMF) does not make cardinality free. EMF is a convenient way to extract metrics from logs. Each unique dimension combination still becomes a custom metric series. EMF plus tenantId plus route plus status is how a “we only log JSON” team still pays a metrics bill. The log line and the series are related. They are not the same meter.
Dimensions That Belong on a Series
A dimension should be something you will alarm on or roll up on a dashboard for the life of the product. Service name, environment, and a coarse operation (checkout.submit, billing.invoice) usually qualify. Tenant id qualifies only if you have a small tenant count and a real per-tenant SLO. Request id, session id, user id, sha of a git commit, and pod name almost never qualify.
Pod name and task id are the classic container leak. Every deploy mints new series. Old series stay billable until they go unused long enough for CloudWatch to drop them. During a busy release week the unique series count steps up and does not step down on the same day. The CFO sees a CloudWatch increase that “looks like traffic.” It is identity churn.
We treat the dimension allowlist as code, not a wiki. The emit helper takes a closed struct. Adding a field requires a review that asks: will we alarm on this, and can we bound the distinct values? If the answer is “we might filter a dashboard someday,” it is a log field.
Metric Math and Contributor Insights exist because high cardinality is a known problem. Contributor Insights can show top-N without turning every contributor into a standing custom series. Use that when the question is “which tenant is hot right now,” not “store a gauge for every tenant forever.”
PutMetricData, EMF, and the API Meter
Custom metrics cost is not only the series month. PutMetricData is request-priced. A sidecar that flushes every dimension combination every ten seconds will show up on the API usage type even if the series count looks modest. Batch. Set a flush interval that matches the resolution you paid for. Do not PutMetricData from inside a tight request loop.
EMF written to CloudWatch Logs still incurs log ingest and storage. Then the extracted metrics incur the custom metric meter. That double path is fine when you wanted both a debug log and a rollup. It is waste when the log exists only to carry EMF and nobody reads the log group. If the log group is the expensive part, fix retention on that group. If the Metrics line is the expensive part, fix dimensions. Do not shorten retention to “fix custom metrics.”
Namespace sprawl is a people problem. Every squad invents MyApp/Prod and myapp.prod and Checkout. Cost Explorer then cannot tell a story. We pick one namespace per product and one environment dimension. The AWS cloud infrastructure work we do for landing zones includes that emit contract so a new service does not invent a second CloudWatch product.
What We Build Instead of “Add a Dimension”
When a product manager wants “latency for this customer on this route,” we do not add two unbounded dimensions. We keep service and operation on the metric. We put tenant and route on traces or on a sampled log. If a specific tenant is in incident, we enable a temporary high-cardinality emit behind a flag with an expiry. Standing per-tenant series for every customer is an observability tax, not an SLO.
Alarms should attach to low-cardinality series. An alarm on a metric that includes request id will never have a stable threshold. That is how teams create hundreds of alarms nobody acknowledges. The custom metrics bill and the on-call load have the same root cause.
We watch the series count the same way we watch AWS Cost Anomaly Detection setup. A deploy that adds a dimension should page like a cost anomaly, because it is one. Anomaly Detection on the CloudWatch service filter will not name the dimension. The staff engineer still needs the emit allowlist. The detector buys time.
CloudFront and edge metrics are not this article. If the bill you care about is request and transfer at the CDN, start with CloudFront cost optimization. Custom application metrics are the line behind the origin.
A Bounded Emit Helper
The shape we ship is small:
ALLOWED = {"service", "environment", "operation"}
def put_latency(name, value, dims):
extra = set(dims) - ALLOWED
if extra:
raise ValueError(f"blocked dimensions: {sorted(extra)}")
# PutMetricData or EMF emit with dims only
That raise in CI is cheaper than a month of unique series. Reviewers can still add operation. They cannot add customerId without changing the allowlist and answering the bound question.
Frequently Asked Questions
Does Embedded Metric Format avoid CloudWatch custom metrics cost?
No. EMF is a transport, and unique namespace, name, and dimension combinations still become custom metric series on the custom metrics meter. EMF can also add log ingest if you write the payload to CloudWatch Logs. Use EMF when you want log-shaped emission, not as a way to hide cardinality.
How long do unused custom metric series stay on the bill?
CloudWatch stops billing a custom metric after it receives no new data for a period defined in current service behavior, and that delay is long enough that a bad deploy still costs through the next planning cycle. Do not rely on aging out as the cleanup plan. Remove the dimension in code and wait, or delete the widgets and alarms if you need the series gone from dashboards now.
Are high-resolution custom metrics worth it for a SaaS API?
Sometimes, for a single SLO graph on a coarse operation, and rarely for every tenant and every route. High-resolution metrics use a higher meter than standard-resolution custom metrics. If you already have traces at the request grain, a one-minute custom metric on service and operation is usually enough to alarm. Buy resolution on purpose, not because the default SDK example used it.
Why did custom metrics rise when request volume was flat?
Cardinality and API chatter, not traffic. A new dimension with a high distinct count multiplies series without multiplying users, and a tighter PutMetricData loop multiplies API charges without multiplying customers. Compare series-creating code changes and flush intervals to the Cost Explorer usage types. If volume is flat and Metrics moved, look at identity fields on the emit path.
If the CloudWatch Metrics line is the one you cannot explain, send the usage types and the emit helper. We will trace the specific bill line and return a dimension allowlist you can enforce in CI. Call 907-841-8407, email contact@rutagon.com, or use rutagon.com/contact.