A serverless data pipeline built for a production SaaS platform needed to ingest data from multiple upstream sources with different reliability characteristics, validate and enrich it, and deliver clean output downstream without a dedicated ops team babysitting the pipeline. Here's the architecture that held up.
The Core Design Constraint: No Standing Compute for Bursty, Uneven Load
The upstream data sources this pipeline consumed produced highly uneven volume — quiet for hours, then a burst of thousands of records in minutes. A traditional always-on ingestion service would need to be sized for the burst, sitting mostly idle otherwise. The serverless architecture — Lambda functions triggered by events, scaling automatically with load — matched this pattern far better than provisioned compute would have.
Stage 1: Ingestion With Idempotency Built In
Each upstream source's data landed in an S3 bucket (either via direct API push or an intermediary polling Lambda), triggering an ingestion Lambda via S3 event notification:
def handler(event, context):
for record in event["Records"]:
bucket = record["s3"]["bucket"]["name"]
key = record["s3"]["object"]["key"]
idempotency_key = compute_idempotency_key(bucket, key)
if already_processed(idempotency_key):
logger.info(f"Skipping duplicate: {idempotency_key}")
continue
raw_data = fetch_s3_object(bucket, key)
validated = validate_schema(raw_data)
queue_for_enrichment(validated, idempotency_key)
mark_processed(idempotency_key)
Idempotency mattered enormously here because S3 event notifications and any retry logic in the upstream sources could occasionally deliver the same object more than once — without an explicit dedup check, this would have produced duplicate downstream records, a subtle bug that's much harder to catch after the fact than to prevent at ingestion.
Stage 2: Schema Validation as a Hard Gate
Malformed or unexpected-shape data from an upstream source needed to be caught before it propagated further into the pipeline, not discovered downstream where the failure mode is harder to trace back to its source. A validation stage checked each record against a defined schema, routing failures to a dead-letter queue with the specific validation error attached rather than silently dropping or, worse, attempting to process malformed data:
def validate_schema(record: dict) -> ValidatedRecord:
errors = schema_validator.validate(record, schema=INGEST_SCHEMA)
if errors:
send_to_dlq(record, errors)
raise SchemaValidationError(errors)
return ValidatedRecord(**record)
Stage 3: Enrichment via SQS-Buffered Fan-Out
Validated records queued into SQS rather than invoking the enrichment Lambda directly from the validation stage — this decoupling gave the pipeline natural backpressure handling. If the enrichment stage (which called out to external APIs for additional data) hit rate limits or slowed down, records simply queued rather than the ingestion stage failing or the system falling over under burst load.
def enrich_handler(event, context):
for message in event["Records"]:
record = json.loads(message["body"])
try:
enriched = call_enrichment_api(record, timeout=5)
deliver_downstream(enriched)
except EnrichmentAPIError as e:
if message["attributes"]["ApproximateReceiveCount"] < MAX_RETRIES:
raise # SQS will retry with backoff
send_to_dlq(record, str(e))
Stage 4: Delivery With Dead-Letter Visibility
Records that failed at any stage — validation or enrichment — landed in a dead-letter queue with structured error context, feeding a lightweight monitoring dashboard rather than disappearing silently. This mattered as much operationally as architecturally: a pipeline that fails loudly and traceably is dramatically easier to operate without dedicated headcount than one that fails silently and requires log archaeology to diagnose.
Production Lessons
Lambda cold starts mattered more at the enrichment stage than ingestion — the enrichment Lambda's dependencies (an HTTP client with connection pooling, a schema library) added enough cold-start latency under bursty invocation patterns that provisioned concurrency for that specific function, rather than the whole pipeline, was the targeted fix.
SQS visibility timeout tuning was more impactful than expected — setting the visibility timeout too short relative to actual enrichment processing time caused duplicate delivery from SQS re-queuing messages the Lambda was still legitimately processing, which idempotency at the enrichment stage (not just ingestion) ultimately had to handle.
Dead-letter queue depth as a leading indicator — monitoring DLQ depth (not just pipeline throughput) surfaced upstream data quality regressions before they became a customer-facing issue, since a spike in validation failures usually meant an upstream source had changed its data format.
Frequently Asked Questions
Why choose Lambda over a managed streaming service like Kinesis for this pipeline?
The bursty, non-continuous nature of the traffic pattern fit Lambda's pay-per-invocation model better than a continuously-provisioned streaming service — Kinesis makes more sense for genuinely high-throughput, continuous streaming workloads, which this wasn't.
How does this architecture handle a downstream outage?
The SQS buffering between stages naturally absorbs downstream slowdowns or brief outages — messages queue rather than failing outright, and standard SQS retry/backoff handles transient errors before escalating to the dead-letter queue.
Is AWS Glue used anywhere in this architecture, or purely Lambda?
For this specific pipeline, Lambda handled the full flow given the record-level processing pattern; Glue becomes more relevant for batch-oriented ETL over large datasets where Spark-based distributed processing outperforms record-by-record Lambda invocation.
How do you prevent runaway cost from a misbehaving upstream source flooding the pipeline?
Rate limiting at the ingestion stage combined with SQS's natural buffering prevents a traffic spike from directly translating into uncontrolled downstream cost — enrichment API calls, often the most expensive stage, are throttled independently of raw ingestion volume.
What monitoring did you build around this pipeline?
CloudWatch alarms on DLQ depth, enrichment Lambda error rate, and end-to-end processing latency (measured from ingestion timestamp to delivery timestamp) gave visibility into both throughput health and the specific stage responsible when something degraded.
Discuss your project → rutagon.com/contact or call 907-841-8407.