A custom AWS webhook delivery system is what we build when a SaaS product must push signed events to customer HTTPS endpoints and a third-party webhook SaaS is the wrong control boundary. The product already lives on AWS. The events already exist on an EventBridge bus. The customer’s security questionnaire asks for HMAC signatures, per-tenant retry, a dead-letter trail, and a guarantee that tenant A’s burst cannot stall tenant B. That is a delivery architecture, not a for url in tenants: requests.post loop.
We ship it as EventBridge API Destinations for the HTTP hop, plus SQS standard queues with MessageGroupId so SQS fair queues keep dwell time sane under a noisy tenant. Signing happens in a Lambda before the destination, because API Destinations will send whatever body you hand them — they will not invent your product’s signature scheme.
Why Not One Lambda Per Event
The naive path is a rule target Lambda that POSTs to the customer URL. It fails in production the same four ways:
- One slow customer endpoint occupies the Lambda concurrency of the whole bus.
- Retries are whatever the function author remembered to code.
- Signing secrets sit in the function environment instead of per-tenant Secrets Manager.
- There is no fair scheduling when one tenant emits 50× the volume.
API Destinations exist to be the HTTP client: connection (auth to the destination, if any), invocation rate (1–300 per second per destination), 5-second timeout, Retry-After honored, then the rule’s retry policy, then an SQS DLQ. Official behavior is in API destinations as targets.
We still keep a Lambda — but upstream, as a signer and router, not as the socket that waits on a customer.
Custom AWS Webhook Delivery System Architecture
domain events
→ custom event bus
→ rule (detail-type = webhookable)
→ SQS standard (MessageGroupId = tenant_id) # fair queues
→ signer Lambda (HMAC, tenant secret, idempotency key)
→ EventBridge API Destination (per-tenant or per-tier)
→ customer HTTPS
↘ rule DLQ (standard SQS) after retries exhausted
Fair queues apply automatically on standard SQS when MessageGroupId is set. No consumer code change. SQS detects a tenant that dominates in-flight share or processing-time share and prioritizes quiet tenants. The noisy tenant still drains; everyone else does not wait behind their backlog. That is the multi-tenant lever. FIFO queues also use MessageGroupId but enforce per-group ordering and throughput limits — we use FIFO only when the product contract requires in-order delivery per tenant. Fair queues on standard is the default we ship for “at-least-once, fair dwell time.”
# Signer: product HMAC, then put to the tenant destination via EventBridge
import hashlib, hmac, json, os, time
import boto3
events = boto3.client("events")
sm = boto3.client("secretsmanager")
def handler(event, _context):
for rec in event["Records"]:
body = json.loads(rec["body"])
tenant = body["tenant_id"]
secret = sm.get_secret_value(SecretId=f"webhook/{tenant}/hmac")["SecretString"]
ts = str(int(time.time()))
payload = json.dumps(body["payload"], separators=(",", ":"))
sig = hmac.new(secret.encode(), f"{ts}.{payload}".encode(), hashlib.sha256).hexdigest()
events.put_events(Entries=[{
"EventBusName": os.environ["DELIVERY_BUS"],
"Source": "webhooks.delivery",
"DetailType": "SignedWebhook",
"Detail": json.dumps({
"tenant_id": tenant,
"destination_arn": body["destination_arn"],
"body": payload,
"headers": {
"X-Webhook-Timestamp": ts,
"X-Webhook-Signature": f"v1={sig}",
"Idempotency-Key": body["event_id"],
},
}),
}])
The delivery bus rule matches SignedWebhook and targets the API Destination. Input transformers map headers and body onto the HTTP request. Connection objects hold customer-facing auth only when the customer issued us a bearer token; our HMAC is a header we compute, not an EventBridge connection password.
Per-tenant destinations get their own invocation rate. A customer who documented 10 req/s gets 10. A shared destination for a cheap tier gets a pooled rate and still has fair queues in front so one noisy tenant cannot fill the 24-hour EventBridge retry window for the others.
resource "aws_cloudwatch_event_api_destination" "tenant" {
name = "wh-${var.tenant_id}"
invocation_endpoint = var.https_url
http_method = "POST"
invocation_rate_limit_per_second = var.rate_limit
connection_arn = aws_cloudwatch_event_connection.none.arn
}
resource "aws_cloudwatch_event_target" "dest" {
rule = aws_cloudwatch_event_rule.signed.name
event_bus_name = aws_cloudwatch_event_bus.delivery.name
arn = aws_cloudwatch_event_api_destination.tenant.arn
role_arn = aws_iam_role.events_invoke.arn
dead_letter_config { arn = aws_sqs_queue.dest_dlq.arn }
retry_policy {
maximum_event_age_in_seconds = 86400
maximum_retry_attempts = 185
}
}
DLQs are standard SQS (EventBridge will not use FIFO as a DLQ). We attach a queue policy that allows events.amazonaws.com SendMessage with aws:SourceArn locked to the rule. Operators replay from the DLQ through the same signer so signatures stay fresh; we do not re-POST a six-hour-old signed body.
Product Rules We Do Not Leave Implied
- HTTPS only. Destination URLs are validated on save; no redirects to a different host.
- Idempotency-Key is the domain event ID. Customers can safely retry-detect. We persist delivery attempts (status, latency, destination) for the same reason the usage metering build persists raw events — disputes.
- Secret rotation is a new HMAC version header (
v1,v2) overlapping for a window. Standing webhook secrets in a shared.envare out of scope for this design. - Pipes vs rules. For a single stream-to-HTTP we will use EventBridge Pipes. For fan-out from a product bus, rules stay cheaper and clearer — see EventBridge cost at scale.
- API Destinations are not GraphQL. Customer-facing query APIs stay on the AppSync build. Webhooks are outbound.
This is full-stack development on AWS: the product feature, the tenant isolation, and the operations story. We have shipped the same event-driven shape on production SaaS platforms — never as a generic “we posted to Slack.”
Need this delivery path on your bus? 907-841-8407 or contact@rutagon.com.
Talk to us about a custom AWS webhook delivery system →
Frequently Asked Questions
Why not let EventBridge API Destinations call the customer URL directly from the domain bus?
You can, if every event is already signed and every tenant shares one rate limit. The moment tenants have different URLs, secrets, or rates, a signer/router in front is the control point. Direct-from-bus also makes one customer’s 5-second timeouts occupy the destination’s retry budget for everyone on that destination.
Do SQS fair queues require a new queue type?
No. Fair queues activate on standard queues when you set MessageGroupId (we use tenant_id). Consumers do not change. Use FIFO only if the product requires per-tenant ordering; FIFO is a different throughput and ordering contract.
What happens when a customer endpoint is down for a day?
EventBridge retries within the retry policy (up to the 24-hour event age default). After that the event lands on the rule DLQ. We page on DLQ depth, not on the first 5xx. Replay is through the signer so HMAC timestamps are current. A destination that is down for days is a customer ticket, not an infinite retry loop.
How do you stop tenant A from delaying tenant B’s webhooks?
MessageGroupId on the ingress SQS queue so fair queues deprioritize the noisy group, plus per-tenant (or per-tier) API Destination rate limits. Lambda reserved concurrency on the signer is a third brake if a tenant’s secret-fetch pattern is the bottleneck.
Can API Destinations add our HMAC for us?
No. Connections store destination credentials (API key, OAuth, basic). Product HMAC over the body is application logic. We compute it in the signer Lambda and pass the headers through the input transformer. Treating EventBridge connection secrets as the product signature is a mismatch we do not ship.