AWS Glue bills primarily on DPU-hours (data processing unit hours), and teams that inherited a data pipeline built by someone who's since left the company often have no idea which jobs are actually expensive versus which just run frequently. The fix usually isn't rewriting ETL logic — it's finding the specific configuration mistakes that inflate DPU-hours without adding value.
Where the Waste Actually Hides
Idle crawlers scheduled too frequently. Glue Crawlers scan data sources to update the Data Catalog, and they bill for the compute time they run, regardless of whether the underlying data actually changed. A crawler scheduled hourly against a source that updates daily burns DPU-hours discovering nothing new 23 times out of 24.
Job bookmarks disabled or misconfigured. Bookmarks let a Glue job process only new data since its last successful run. Without bookmarks enabled correctly, a job can reprocess the entire dataset on every run — turning what should be an incremental job into a full reprocessing job, with the DPU-hour cost to match.
import boto3
glue = boto3.client("glue")
def audit_glue_job_bookmarks(job_names):
"""Flags jobs where bookmarks are disabled or the job doesn't
reference a bookmark-aware read pattern."""
findings = []
for job_name in job_names:
job = glue.get_job(JobName=job_name)["Job"]
bookmark_setting = job.get("DefaultArguments", {}).get("--job-bookmark-option", "job-bookmark-disable")
if bookmark_setting == "job-bookmark-disable":
findings.append({
"job_name": job_name,
"issue": "Bookmarks disabled — job likely reprocesses full dataset each run",
"worker_type": job.get("WorkerType"),
"number_of_workers": job.get("NumberOfWorkers"),
})
return findings
Over-provisioned worker counts. Glue jobs specify a number of workers (DPUs) and a worker type (Standard, G.1X, G.2X). Jobs configured for peak-load worker counts that run at that scale for every invocation — including small incremental runs — pay for capacity they don't use most of the time.
Flex execution left off for non-time-sensitive jobs. Glue's Flex execution class runs jobs on spare capacity at a meaningfully lower price for workloads that can tolerate variable start times — batch jobs that run overnight are usually good Flex candidates, but teams often leave every job on standard execution by default.
A Practical Audit Sequence
- Pull Cost Explorer data filtered to the Glue service, grouped by job name via cost allocation tags, to identify the top DPU-hour consumers
- For each top consumer, check bookmark configuration, worker count relative to actual data volume, and whether Flex execution is a fit
- Check crawler schedules against actual source data change frequency — most crawlers can move from hourly to daily or even weekly without any practical impact on catalog freshness
- Re-baseline and monitor for 2-3 billing cycles to confirm savings materialize as projected, since Glue cost attribution can lag actual job runs slightly
What Typically Comes Out of This
The most common finding in a first-pass Glue cost review is a handful of crawlers scheduled far more frequently than the source data changes, plus one or two ETL jobs running with disabled bookmarks that have been silently reprocessing full historical datasets for months. Neither requires rewriting the underlying transformation logic — both are configuration fixes.
Frequently Asked Questions
How much can bookmark fixes typically save on a Glue bill?
It depends entirely on how large the historical dataset is relative to the incremental daily volume, but for jobs that have been unnecessarily reprocessing full history, the reduction in DPU-hours per run can be substantial — often the majority of that job's cost.
Does Flex execution affect job reliability?
Flex jobs may experience variable start times since they run on spare capacity, so they're best suited for workloads without a strict completion deadline. Time-sensitive jobs should stay on standard execution.
How do we attribute Glue costs to specific jobs in Cost Explorer?
Tag your Glue jobs with cost allocation tags (team, project, or pipeline name) and activate those tags in the Billing console, then group Cost Explorer results by tag to see per-job or per-team DPU-hour spend.
Is reducing worker count risky for job performance?
It can extend job runtime if reduced too aggressively, so changes should be tested against realistic data volumes before deploying to a production schedule — the goal is right-sizing, not minimizing at the expense of reliability.
See what an AWS cost audit finds in your data pipeline spend → rutagon.com/contact or call 907-841-8407.