Amazon MSK (Managed Streaming for Apache Kafka) clusters running regulated workloads — audit event streams, transaction logs, anything with retention or replication requirements tied to a compliance framework — face a specific cost tension: the same retention and replication settings that satisfy compliance requirements also directly drive broker storage and instance costs. Optimizing MSK cost without understanding which settings are compliance-driven risks quietly weakening a control while chasing a smaller AWS bill.
Separating Compliance-Driven Settings From Discretionary Ones
The first step in any MSK cost review for a regulated workload is explicitly categorizing configuration decisions: which settings exist because a compliance framework or internal policy requires them (retention period for audit logs, replication factor for durability guarantees), and which were set as defaults or best-guess sizing with no specific requirement behind them. Cost optimization should only touch the second category without a documented compliance re-analysis.
# Auditing current topic-level retention and replication config
kafka-configs.sh --bootstrap-server $MSK_BROKERS \
--entity-type topics --entity-name audit-events \
--describe
# Common finding: retention.ms set far beyond the actual compliance requirement,
# multiplying storage cost for data that could move to cheaper cold storage
Where Discretionary Overspend Actually Lives
In practice, most MSK cost overruns we find aren't in the compliance-required settings themselves — they're in decisions made around those settings without corresponding cost analysis:
- Broker instance sizing based on peak throughput that occurs rarely, rather than sustained average throughput with burst headroom modeled explicitly.
- EBS storage over-provisioned relative to actual retained data volume, often because initial sizing assumed higher throughput than production actually sees.
- Replication factor set higher than the durability requirement actually calls for — a replication factor of 3 is standard for genuine durability needs, but we occasionally find configurations replicating beyond what any documented requirement justifies.
- Topics retaining data in hot Kafka storage well beyond the compliance-required window, when a tiered approach (shorter Kafka retention, archived to S3 for the remainder of the compliance retention period) would satisfy the same requirement at lower cost.
Tiered Storage as the Compliance-Compatible Cost Lever
MSK's tiered storage feature (where supported by your cluster version) or a custom Kafka-to-S3 archival pipeline lets you keep only recent, actively-consumed data in expensive broker storage while moving older data — still within the compliance retention window — to S3 at a fraction of the cost. This is the single highest-leverage optimization for regulated workloads specifically because it reduces cost without reducing the actual retention period the compliance requirement demands.
# Kafka-to-S3 archival consumer preserving compliance retention window,
# reducing hot-broker storage to only recently-produced data
def archive_expired_hot_segments(topic, hot_retention_days, compliance_retention_days):
for partition in get_partitions(topic):
segments = get_segments_older_than(partition, hot_retention_days)
for segment in segments:
s3_client.upload(segment, archive_bucket, retention_tag=compliance_retention_days)
# Segment remains queryable via S3 for audit purposes,
# removed from hot broker storage once archived
Right-Sizing Brokers Without Touching Replication
Broker instance type and count can often be optimized independently of replication factor — moving to Graviton-based broker instances, or right-sizing from an oversized legacy instance type to one matched to actual sustained throughput, reduces compute cost while leaving the durability-critical replication configuration untouched.
What to Document Before Making Any Change
For a regulated workload, every cost optimization change should be paired with a brief written justification tying the change to actual usage data and confirming it doesn't reduce compliance-required retention or durability — this documentation itself becomes useful evidence if an auditor or internal risk review later asks why a configuration changed.
This tiered-storage approach is part of our AWS cloud infrastructure capability, alongside the broader data architecture work in data lake architecture on AWS GovCloud.
Talk to us about your AWS cost audit: 907-841-8407 or contact@rutagon.com.
Talk to us about your AWS cost audit →
Frequently Asked Questions
Does reducing MSK broker storage risk losing compliance-required data?
Not if implemented correctly — tiered storage or archival to S3 preserves the data for the full compliance retention window, it just moves older data out of expensive hot broker storage rather than deleting it early.
What replication factor do most compliance frameworks actually require?
Most frameworks don't specify an exact replication factor — they require a documented durability/availability standard, which a replication factor of 3 commonly satisfies for Kafka; the specific number should trace back to your own documented requirement, not just a default.
How much can tiered storage actually reduce MSK costs?
It varies by workload, but for topics with long compliance-driven retention windows and most consumption happening on recent data, moving older segments to S3-tier storage typically produces meaningful broker storage savings since S3 storage costs a fraction of EBS-backed broker storage.
Is MSK Serverless a better cost option for regulated workloads?
It can be, for variable-throughput workloads where provisioned broker capacity would otherwise sit underutilized — but serverless pricing models and compliance-relevant configuration options (like tiered storage support) should be compared directly against your specific requirements before switching.
Should compliance-driven retention settings ever be part of a cost optimization review?
Only with a documented re-analysis confirming the original requirement — if the retention period itself was over-specified relative to the actual compliance requirement, that's worth revisiting, but it needs sign-off from whoever owns the compliance requirement, not a unilateral engineering change.