Standing keys in a container image or a task definition are a credential that outlives every deploy. Mid-market SaaS teams still find AWS_ACCESS_KEY_ID in ECS task defs, in Secrets Manager payloads that are just keys by another name, and in baked images. Rutagon builds the ECS task role vs access keys cutover: inventory the keys, attach a task role with the same calls, then delete the IAM user key so the task cannot fall back.
This is not a lecture on IAM. It is a delivered migration for services you already run on ECS or Fargate. We treat the task role as the only AWS identity the container is allowed to use. Environment variables that look like “config” are in scope if they are keys.
The same idea shows up in CI as OIDC instead of access keys. ECS is the runtime twin: the role is assumed for you, the keys are not in the def.
ECS Task Role vs Access Keys in Task Definitions
ECS task role vs access keys starts with a search you can repeat. We read task definition revisions, Parameter Store and secret ARNs referenced by the def, and common env names (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, vendor-specific aliases). A secret that holds a key is still a key. Rotating that secret is a bridge, not the finish.
Each hit becomes a row: service, cluster, task family, revision, where the key lives, which IAM user it belongs to, and which AWS calls the task actually makes. We learn the call set from code, from traces, and from CloudTrail around that user. Guessing AdministratorAccess is not a cutover.
Then we mint or reuse a task IAM role with that call set, set taskRoleArn on a new revision, drop the key env and secret refs, deploy, and watch. Only after the task is healthy and CloudTrail shows the role—not the user—do we deactivate the access key. The user delete waits until no other workload shares it. Shared users are a finding of their own.
How We Find AWS_ACCESS_KEY_ID Without Trusting One Grep
Grep of the live def is necessary and incomplete. Keys hide in JSON secrets, in older revisions still referenced by a stuck service, and in EC2-hosted agents that are not ECS. We also scan CI vars that inject keys at deploy time, because a clean def can still receive keys from the pipeline.
We compare task role presence to key presence. A task that has both is not “half migrated.” It is a fallback path. We fail the row until the key is gone. A task with neither, talking to AWS, is using the instance role or a default chain we did not intend. That is a different cutover.
GitLab OIDC into AWS covers the pipeline that builds the image. This article stays on the running task. Mixing them in one change set is how rollbacks reintroduce keys from the pipeline while the role looks fine.
The Cutover We Run
The deploy is a new task definition revision. We do not mutate the live revision in place. Rollback is “redeploy the previous revision,” and that previous revision must not still contain the key if you have already deleted it. Sequence matters: role first, traffic on the role, key deactivate, then def cleanup so rollback cannot resurrect the key from git history without a new secret.
family: payments-worker
was:
env: AWS_ACCESS_KEY_ID
secret: /ecs/payments/aws-user
now:
taskRoleArn: arn:aws:iam::ACCOUNT:role/ecs-payments-worker
env: []
prove:
cloudtrail_principal: ecs-payments-worker
key_status: inactive
IAM for ECS tasks is documented as the task IAM role. We follow that model. We do not invent a sidecar that injects long-lived keys “for compatibility.”
AWS cloud infrastructure is the delivery surface when the cluster, the role, and the deploy pipeline all have to move together.
Evidence That the Key Is Gone
A screenshot of the IAM console is not the sample. We keep the old and new task definition JSON (redacted), the service event that rolled the revision, CloudTrail excerpts for the role, and the access-key last-used and inactive timestamps. If a second key exists on the same user, it is on the list. Partial disable is a failed close.
We also keep the search query and the empty result after cutover. “We think we got them” is not a close. The same search runs on a schedule so a new def cannot reintroduce the env names without a ticket.
If the application still reads AWS_ACCESS_KEY_ID from a library default, we change the app to the default credential chain and the task role. Leaving the env names in code, empty, is how someone fills them later.
Images, sidecars, and the key that never sat in the task def
A clean task definition can still ship a key. Application images sometimes bake ~/.aws/credentials. Sidecars sometimes read a file from a volume that a bootstrap container wrote. Init containers sometimes call a secrets API and export env into the main process. We inspect the image history and the entrypoint chain for those paths. Finding AWS_ACCESS_KEY_ID only in the def is the easy win. Finding it in a layer is the one that survives a “we removed the env.”
We also inventory scheduled tasks and one-off run-task executions. A service can be clean while a cron task family still has the key. Those families do not show up on the service list. They show up in the account’s task definition list and in EventBridge or scheduler targets. The cutover includes every family that still runs.
Windows and Linux containers differ in where the credential file lives. We do not assume a Linux path. We treat the default credential chain as the target: the task role, then nothing. If a library requires static keys, that library is a code change in the same engagement. Leaving a “temporary” file for that library is how the key returns on the next incident.
After cutover we watch two signals. CloudTrail should show the task role for the APIs the app needs. The IAM user key should show no new last-used. If last-used moves after deactivate, something else still holds a copy—another cluster, a laptop, a forgotten CodeBuild project. That copy is a new row. The ECS job does not close while the user key still lives.
Local developers who exported the same user key are out of band for the task def and in band for the user retirement. We rotate or delete the user only when the inventory says the key is unused everywhere. That is why the first step is the user-to-workload map, not a hopeful deactivate.
The task IAM role stays the application identity. We keep the execution role on image pull and secret fetch. Mixing those ARNs in the def is a review comment, not a migration strategy.
Frequently Asked Questions
Can we keep the access key as a break-glass for the same task?
No. A key in the def or the secret store will be used under pressure. Break-glass belongs in a separate, monitored role assumption path with a ticket, not in the task family. If you need a human in production, that is a different identity.
Does a task execution role replace the task role?
No. The execution role pulls images and secrets for the agent, while the task role is what the application code uses. Keys in the application still need the task role. Confusing the two is a common incomplete migration.
What if two services share one IAM user?
Then the cutover is two roles and one user retirement at the end. We do not point both tasks at the same role unless the call sets truly match. Shared users are why key deletion keeps getting delayed. Split first.
Will Fargate and EC2 launch types differ?
The task role mechanism is the same. The extra risk on EC2 is instance-role bleed if the task can reach the instance metadata in a way you did not intend. We check that path as part of the cutover. Fargate still needs the key removed from the def.
Talk to us about your credential elimination pilot. Call 907-841-8407, email contact@rutagon.com, or use rutagon.com/contact.