Secrets Management Patterns in AWS Production

Secrets management in AWS production systems is the difference between a secure architecture and a breach waiting to happen. Hardcoded credentials in environment variables, static API keys committed to repositories, and long-lived access tokens passed between services — these patterns persist across government and commercial systems despite years of documented exploits. At Rutagon, we treat secrets management as a core architectural concern, not an afterthought bolted on during compliance reviews.

This article walks through the patterns we deploy across production workloads: AWS Secrets Manager rotation, Parameter Store hierarchies, cross-account access, and the zero-credential architectures that eliminate static secrets entirely.

Why Static Secrets Fail in Production

Static secrets — credentials that don't change unless someone manually rotates them — create compounding risk over time. The longer a secret lives, the more likely it has been logged, cached, or exposed through a configuration drift. In regulated environments governed by NIST 800-53 controls, static secrets violate the principle of least privilege and make audit trails unreliable.

The attack surface grows with every service that shares a credential. A database password used by five Lambda functions, two ECS services, and a CI/CD pipeline has five-plus potential exposure points. Rotation isn't optional — it's the minimum viable security posture.

We've written extensively about eliminating credentials entirely with OIDC and zero-trust architectures. Secrets management patterns complement those approaches for the credentials that genuinely must exist.

AWS Secrets Manager: Rotation-First Architecture

AWS Secrets Manager is purpose-built for secrets that require rotation. Unlike Parameter Store, it includes native rotation Lambda functions for RDS, Redshift, DocumentDB, and custom secret types.

Rotation Configuration

For an RDS PostgreSQL credential with 30-day automatic rotation:

# CloudFormation — RDS secret with rotation
DatabaseSecret:
  Type: AWS::SecretsManager::Secret
  Properties:
    Name: !Sub "${Environment}/database/primary"
    Description: Primary RDS PostgreSQL credentials
    GenerateSecretString:
      SecretStringTemplate: '{"username": "app_service"}'
      GenerateStringKey: password
      PasswordLength: 32
      ExcludeCharacters: '"@/\'

DatabaseSecretRotation:
  Type: AWS::SecretsManager::RotationSchedule
  Properties:
    SecretId: !Ref DatabaseSecret
    RotationLambdaARN: !GetAtt RotationFunction.Arn
    RotationRules:
      AutomaticallyAfterDays: 30

The rotation Lambda follows a four-step protocol: createSecret, setSecret, testSecret, finishSecret. Each step is idempotent, so failed rotations can be safely retried without leaving the database in an inconsistent state.

Alternating Users Strategy

For zero-downtime rotation, we deploy the alternating users pattern. Two database users — app_service_a and app_service_b — alternate as the active credential. While one is in use, the other is rotated. The secret's AWSCURRENT staging label always points to the active credential:

def set_secret(service_client, arn, token):
    pending = service_client.get_secret_value(
        SecretId=arn, VersionId=token, VersionStage="AWSPENDING"
    )
    secret = json.loads(pending["SecretString"])

    current = service_client.get_secret_value(
        SecretId=arn, VersionStage="AWSCURRENT"
    )
    current_user = json.loads(current["SecretString"])["username"]

    new_user = "app_service_b" if current_user == "app_service_a" else "app_service_a"
    secret["username"] = new_user

    conn = get_connection(current["SecretString"])
    conn.execute(f"ALTER USER {new_user} WITH PASSWORD %s", [secret["password"]])

This pattern ensures applications reading AWSCURRENT always get a valid credential, even mid-rotation.

Parameter Store: Hierarchical Configuration

Not every secret needs rotation machinery. Feature flags, configuration values, and non-rotating references fit better in AWS Systems Manager Parameter Store. The hierarchical namespace maps cleanly to environment and service boundaries:

/production/api/database-url          (SecureString)
/production/api/redis-endpoint        (String)
/production/api/feature-flags/v2-ui   (String)
/staging/api/database-url             (SecureString)

Retrieving Parameters in Lambda

import boto3
from functools import lru_cache

ssm = boto3.client("ssm")

@lru_cache(maxsize=32)
def get_parameter(name: str, decrypt: bool = True) -> str:
    response = ssm.get_parameter(Name=name, WithDecryption=decrypt)
    return response["Parameter"]["Value"]

def get_parameters_by_path(path: str) -> dict:
    paginator = ssm.get_paginator("get_parameters_by_path")
    params = {}
    for page in paginator.paginate(Path=path, WithDecryption=True, Recursive=True):
        for param in page["Parameters"]:
            key = param["Name"].split("/")[-1]
            params[key] = param["Value"]
    return params

The get_parameters_by_path call retrieves all parameters under a prefix in a single API call, reducing Lambda cold start latency — a pattern we combine with the cold start optimizations we deploy across government workloads.

Cross-Account Secrets Access

Multi-account AWS architectures — standard for government systems following the AWS multi-account strategy — require secrets to flow across account boundaries. A shared services account might host database credentials consumed by workload accounts.

Resource Policy for Cross-Account Access

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowWorkloadAccountAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111222333444:role/WorkloadServiceRole"
      },
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalTag/Environment": "production"
        }
      }
    }
  ]
}

The condition block restricts access to roles tagged with the correct environment, adding an attribute-based access control (ABAC) layer beyond simple account-level trust. We use this pattern extensively in the multi-account Terraform architectures we deploy for regulated clients.

KMS Key Policy

The secret's encryption key must also grant cross-account kms:Decrypt:

{
  "Sid": "AllowCrossAccountDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111222333444:role/WorkloadServiceRole"
  },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

Missing KMS permissions are the most common failure mode in cross-account secret access. The GetSecretValue call succeeds at the API level but returns an encrypted blob the caller cannot decrypt.

Secrets Management in CI/CD Pipelines

CI/CD pipelines are high-value targets because they typically need broad access to deploy across environments. We eliminate static credentials in pipelines entirely using OIDC federation, but build-time secrets like Docker registry tokens and signing keys still require management.

GitHub Actions with OIDC + Secrets Manager

jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::999888777666:role/GitHubActionsDeployRole
          aws-region: us-west-2

      - name: Retrieve deployment secrets
        run: |
          DB_URL=$(aws secretsmanager get-secret-value \
            --secret-id production/api/database-url \
            --query SecretString --output text)
          echo "::add-mask::$DB_URL"
          echo "DB_URL=$DB_URL" >> $GITHUB_ENV

The ::add-mask:: directive prevents the secret from appearing in GitHub Actions logs. Combined with OIDC, the pipeline never stores static AWS credentials — the role assumption produces short-lived tokens scoped to the deployment.

Monitoring and Alerting on Secret Access

CloudTrail logs every GetSecretValue and PutSecretValue call. We build CloudWatch metric filters to detect anomalous access patterns:

{
  "filterPattern": "{ ($.eventName = GetSecretValue) && ($.errorCode = AccessDenied) }",
  "metricTransformations": [{
    "metricName": "SecretAccessDenied",
    "metricNamespace": "Security/SecretsManager",
    "metricValue": "1"
  }]
}

Repeated AccessDenied events on secret retrieval indicate either misconfigured permissions or credential probing. We alert on these in real time and correlate with our observability stack to trace the calling identity.

Secrets Management Patterns for Government Compliance

For systems pursuing FedRAMP authorization or operating under NIST 800-53, secrets management maps directly to several control families:

• IA-5 (Authenticator Management): Automated rotation satisfies password lifecycle requirements without manual intervention.
• SC-12 (Cryptographic Key Establishment): KMS-encrypted secrets with key rotation meet cryptographic management controls.
• AU-2 (Audit Events): CloudTrail integration provides the audit trail required for secret access events.
• AC-6 (Least Privilege): Resource policies and ABAC conditions restrict secret access to the minimum required scope.

These aren't theoretical mappings — they're the evidence artifacts we generate as part of our cybersecurity delivery practice.

Frequently Asked Questions

When should I use Secrets Manager vs. Parameter Store?

Use Secrets Manager for credentials that require automatic rotation — database passwords, API keys with expiration, and OAuth client secrets. Use Parameter Store for configuration values, feature flags, and references that change infrequently. Parameter Store's free tier handles most configuration needs, while Secrets Manager charges per secret per month but includes rotation infrastructure.

How do you handle secret rotation without application downtime?

The alternating users pattern maintains two valid credentials at all times. Applications cache the current secret and refresh on authentication failures, picking up the newly rotated credential automatically. Combined with connection pooling that validates connections before use, rotations are invisible to end users.

What happens if a secret rotation fails mid-process?

AWS Secrets Manager's four-step rotation protocol is idempotent. If rotation fails at the setSecret or testSecret stage, the AWSCURRENT label remains on the previous version. The rotation can be manually retried or will automatically retry on the next scheduled rotation. CloudWatch alarms on the RotationFailed metric ensure failures are caught immediately.

How do cross-account secrets work with AWS Organizations?

You can use Organizations SCPs to restrict which accounts can access secrets in a shared services account, and combine resource policies with ABAC conditions on principal tags. This layered approach ensures only authorized workload accounts in the correct environment can retrieve specific secrets.

Is AWS Secrets Manager sufficient for CMMC Level 2 compliance?

Secrets Manager with automatic rotation, KMS encryption, and CloudTrail logging addresses several CMMC Level 2 practices around authenticator management and access control. However, CMMC compliance requires a holistic approach — secrets management is one component alongside network segmentation, endpoint protection, and incident response capabilities.