AWS Cost Review & Optimization Automation — From 2 Days to 30 Minutes
Summary: A manual AWS cost review used to take 1–2 days and depended on the reviewer's experience. I built an automated cost review system in Python — it collects billing data, inventories resources, detects waste against a 7-pillar framework, cross-checks evidence, and produces a report. Results on a real engagement: 90% less time, $248–360/month in potential savings identified, and every recommendation backed by concrete cost evidence.
(~18–24h → ~30 min)
per month
internal estimate
standardized
Technology
My Contribution
Designed and built the entire automated AWS cost review system myself: cost data collection, resource inventory, waste detection, evidence cross-checking, savings estimation, and client report generation.
Step 1 — The problem: manual review doesn't scale
Before building this system, each AWS cost review for a client took 1–2 days of manual work. The process usually went:
- Open AWS's cost analysis tool, filter by each service, take screenshots
- Log into the console and manually inspect every resource type (volumes, snapshots, network endpoints...)
- Compile everything into a spreadsheet and write the report
The issues: reliance on the reviewer's experience, no consistent standard, and hard to scale across clients — every review started from scratch, with no guarantee of full, even coverage.
Case anonymized — all absolute figures and client resource names are hidden.
Step 2 — Building the automated system (Python + AI)
I designed the workflow around 3 principles:
- Fixed framework, changing data — always review against the same 7 pillars and a consistent way of classifying waste
- Machines do the repetitive work, humans do the judgment — scripts collect & compute automatically; humans verify and make the call
- Evidence, not guesswork — every conclusion traces back to a specific data source
Input
Provided by the clientAutomated processing
Python — runs directly, no infrastructure to set upAI interpretation & drafting
Speeds up the most time-consuming part — but doesn't decideHumans verify & decide
What AI and the machine are not allowed to do aloneResult
Delivered to the clientThe full workflow — Python handles data, AI handles interpretation & drafting, humans keep the decisions.
Technical detail: the pipeline modules For engineers
The pipeline is written in Python 3.10+ (standard library + boto3 only), running directly through AWS CLI v2 under the client's profile (auth via SSO / saml2aws) — no dedicated infrastructure needed.
| Module | Responsibility |
|---|---|
fetch_costs.py | Pull Cost Explorer bills by month (Unblended Cost), grouped by SERVICE / USAGE_TYPE |
analyze_costs.py | Pareto of top services, architecture grouping, Estimate vs Actual, month-over-month anomalies (rule-based) |
investigate_findings.py | Scan EBS / snapshot / VPC Endpoint / EIP / ALB for orphaned or idle resources (evidence-based, with traffic checks) |
investigate_provisioning.py | Right-sizing via Compute Optimizer |
investigate_commitments.py | Savings Plans / Reserved Instances utilization |
investigate_finops_tools.py | Check Budgets, Cost Anomaly Detection, Trusted Advisor, Cost Allocation Tags |
drill_costs.py | Deep-drill by USAGE_TYPE, compute run-rate around an event/cutover |
Configured via engagement.json (client, analysis period, region scope, auth profile). Outputs: run-summary.md + investigation-summary.md (raw data, no narrative yet) for the reviewer to read before writing findings.
Step 3 — The 7-pillar review framework
Every engagement is reviewed against the same 7 pillars, so nothing gets missed:
| Pillar | Description | Example findings |
|---|---|---|
| 1. Unused resources | Resources still existing but serving nothing | Orphaned volumes, extra snapshots, unused network endpoints |
| 2. Right-sizing | Resources provisioned larger than actual need | Over-provisioned servers and databases |
| 3. Prepaid commitments | Making use of long-term committed pricing | Commitment plans under-utilized or missing |
| 4. Storage lifecycle | Managing old data to cut storage cost | Storage with no policy to clean up old data |
| 5. Data transfer | Cost of moving data within/out of AWS | NAT gateways, cross-region transfer |
| 6. Operational scheduling | Turning resources off/on by working hours | Test environments running 24/7 unnecessarily |
| 7. Cost governance | Controls & alerting for cost | Budget alerts, tagging, an owner reviewing bills |
Step 4 — Pilot on a real client
Applied to the first engagement — a production AWS environment in a US region (3 months of billing analyzed):
| Metric | Value |
|---|---|
| Average spend / month | Hidden for confidentiality |
| Actual spend vs client's estimate | 33.8% higher |
| Savings opportunities identified | $248–360/month |
| Cost governance maturity | 1.5/5 (internal scale) |
Cost breakdown by architecture group (3 months combined):
| Group | % of total bill |
|---|---|
| Compute | 83.5% |
| Security & logging | 9.6% |
| Network & CDN | 3.4% |
| Data & storage | 2.9% |
| Other | 0.6% |
Figures anonymized — only percentages shown, no absolute amounts or client resource names.
Step 5 — Key findings
EBS volumes — $218–300/month in savings
- Finding: 27 problematic volumes — orphaned ones (attached to nothing), and production databases still on the older, pricier volume type (gp2) instead of the newer gp3 at the same performance.
- Recommendation: Delete orphaned volumes, migrate databases to the cheaper newer volume type.
Technical detail: detection & gp2 → gp3 For engineers
How wasteful volumes are detected (evidence-based):
- Orphan: volume in
availablestate (noattachment) — not attached to any instance. - Attached-to-stopped: volume
in-usebut the target instance isstopped— still billing for storage even though nothing runs. - Cross-checked against Compute Optimizer and Trusted Advisor (idle volumes) to raise confidence before recommending.
gp2 → gp3: gp3 is ~20% cheaper at the same baseline (3,000 IOPS / 125 MB/s free), decoupling capacity from performance. Migrated via modify-volume — no detach, no downtime, but best done in a maintenance window to monitor.
aws ec2 modify-volume --volume-id vol-xxxx \ --volume-type gp3 --iops 3000 --throughput 125
Savings estimated conservatively — only the certain parts (orphan + gp2/gp3 delta), no optimistic stacking of assumptions.
Long-stopped servers still billing for storage
- Finding: A common cost trap — a stopped server (EC2) no longer bills for compute, but its attached volumes (and static IP, if any) keep billing continuously. Many machines had been off for months with nobody noticing, because the bill never "complains".
- Recommendation: For machines stopped long enough → consider deleting the machine and its volumes. For recently stopped ones → confirm with the owner first, to avoid deleting something intentionally paused.
Technical detail: classifying stopped machines for deletion For engineers
The system reviews from an instance-level view (not just per isolated volume): it rolls up total attached EBS + number of static IPs + days stopped into one row, to directly answer "which machine should be removed".
Days stopped is derived from the instance's StateTransitionReason (format User initiated (YYYY-MM-DD ...)) — no extra API call. Three states:
| State | Condition | Action |
|---|---|---|
| Terminate candidate | Stopped ≥ 30 days (threshold adjustable) | Strong deletion candidate — off for long, unlikely still needed |
| Watch | Recently stopped, or stop time unknown | Confirm with the owner before acting |
| Running / other | — | Skipped, not included |
Boundary note: StateTransitionReason only keeps the most recent state change, and doesn't cover things like Capacity Reservations / licenses billed separately while stopped — so still cross-check with the client's input when in doubt.
Network endpoints (VPC Endpoints) — $30–60/month in savings
- Finding: 10 private connections to AWS services costing ~$128/month, some of them barely used.
- Recommendation: Keep the essential endpoints, review and remove the rarely used ones.
Cost governance — indirect impact
- Finding: No budget alerts, no mandatory resource tagging, no one reviewing bills regularly.
- Recommendation: Enable budget alerts and build a tagging policy to catch abnormal cost increases early.
Step 6 — Every recommendation is backed by evidence
The guiding principle: no cost-cutting proposals based on guesswork. Every recommendation is cross-checked across multiple data sources (billing, AWS advisory tools, actual resource configuration) and its real impact verified before it enters the report.
- Check the relationships between resources before recommending deletion — e.g. a snapshot may still be backing a machine image — to avoid breaking a running system.
- Estimate savings conservatively, with specific resource IDs so the engineering team can act immediately.
- Classify clearly: optimizable / accepted (due to business requirements) / needs further monitoring.
Technical detail: the cross-check logic before recommending deletion For engineers
A classic example: some snapshots initially look "orphaned" because their source volumes were deleted. But if a snapshot is registered by an AMI as a block device mapping, deleting it breaks the AMI — you can no longer launch from it.
Decision rule before flagging a snapshot as waste:
snapshot has no source volume?
│
▼
registered by any AMI? ── YES ─▶ KEEP (not waste)
│
NO
│
▼
has a backup marker (AWS Backup / DLM)? ── YES ─▶ KEEP
│
NO
│
▼
deletion candidate (awaits human confirmation)
The same principle applies to other resources: EIPs check association, ALBs check target-group traffic, VPC Endpoints check actual consumers. Nothing is deleted based on surface state alone.
Results
| Metric | Before | After | Change |
|---|---|---|---|
| Time per review | 1–2 days | ~30 min | 90% less |
| Potential savings | — | $248–360/month | New |
| Recommendation confidence | Depends on experience | Cost evidence + resource IDs | Standardized |
| Review coverage | Easy to miss things | All 7 pillars | Consistent |
Lessons learned
- Automate for speed, keep humans for correctness — the machine handles collection & computation, the expert handles judgment & verification.
- A fixed framework prevents blind spots — 7 pillars ensure full, consistent coverage across every engagement.
- Every recommendation is backed by evidence — cross-check multiple data sources before proposing.
- Governance matters as much as optimization — budget alerts & tagging catch cost spikes early and prevent recurrence.
- Measurable and actionable results — conservative savings estimates with concrete actions for the engineering team.
Need an AWS cost review & optimization?
I take on AWS cost reviews — from building an automated process to detailed manual audits, with a concrete savings roadmap.
Get in touch