Nearly $1M a Year in Savings Without a Single Cost-Related Incident 

"We knew the cloud bill was too high, and we had the reports to prove it. What we didn't have was anyone actually turning those reports into a smaller invoice." - Client representative

This is a wallet holding users' funds, so availability and transaction correctness are the core of the product. If a transaction fails during a sharp market move, a person can't move their own money exactly when it matters most.

Load is uneven. A chain event, a product launch, or a partner integration can increase traffic by an order of magnitude within minutes. The infrastructure is correspondingly large:

• EKS across multiple regions, including Frankfurt and Singapore;
• a 13-node CockroachDB cluster replicated across regions;
• data processing on Fargate;
• terabytes of data in DynamoDB and S3;
• Datadog for hosts, containers, traces, and logs.

Removing a replica or lowering sampling is technically easy. But a change like that can leave the on-call engineer without the signal they need during an incident, or remove capacity that will be needed at the next peak. So we worked by strict rules:

• no maintenance windows and no development freeze;
• every change has a rollback plan and a ticket;
• we cut only telemetry that duplicates other signals;
• savings count only once they show up on the bill, not based on projections.

We ran cloud cost optimization as ongoing delivery work, not as an audit. Every saving got its own ticket in a dedicated project and was counted only after the reduction showed up on the bill. We measured results as annualized run-rate savings: how much costs would drop over a year if the current level holds. This made progress toward the target visible week by week and made it impossible to inflate the number or count the same saving twice.

First, we made costs visible. We extended tagging, including to Karpenter nodes, so teams could see most of the spend on their own resources. When a team has its own number, it also has accountability for it.

Then we tackled the two largest cost pools in parallel: unused AWS resources and Datadog bills. We treated Datadog as an engineering problem, not a procurement one: we looked for telemetry that gave teams no useful signal rather than for a vendor discount. We also reviewed storage separately. S3 data was moved into the right storage classes and cleaned up. For DynamoDB, it turned out that the access pattern for the terabytes of data stored there made cheaper storage worth considering.

In August, these workstreams delivered $456K/year:

• $186K on AWS: as part of our AWS consulting work, we removed unneeded database replicas in two regions and optimized Fargate and S3 costs. In parallel, we extended tagging so costs could be tracked by team;
• $270K on Datadog: reduced spend on unnecessary infrastructure hosts, log volume, and instrumentation.

Next, we addressed the uncapped bills. We either capped them with tiered alerting or replaced them with our own services.

Why Rightsizing Didn't Deliver the Bulk of the Savings

The first step was rightsizing: smaller instances, tighter pod bin-packing, more aggressive autoscaling. It delivered results, but the main savings potential lay elsewhere: in observability bills and in stateful resources that no longer served their purpose. Rightsizing doesn't surface costs like these.

What we ruled out: reducing the fidelity of all telemetry at once. That approach would have produced bigger savings on paper but left on-call engineers without the signals they need. So we reviewed each type of telemetry separately, the same way we approach any log management and monitoring setup: 

• container metrics that duplicated host-level signals were removed;
• info-level logs were excluded from indexing and kept in the archive so they could still be searched when needed;
• production trace sampling was lowered because far more traces were being collected than anyone actually looked at.

Lessons from the client's earlier attempt. The client had previously rolled out centralized database performance alerting across the company but abandoned it because there were too many alerts to act on. So we first tested slow-query monitoring on a single service using a monitor-plus-runbook pattern, then handed it to teams as a template each one configures with its own thresholds.

~68 Tickets in August and Zero Incidents Caused by Cost Optimization

Each individual change is simple. The risk shows up later, during the next incident, when it turns out the signal you removed was exactly the one the on-call engineer needed. In August, the team closed around 68 tickets across several workstreams: cost, developer experience, access, database reliability, and ongoing platform support. That's why every cost-related change shipped separately, with its own justification and rollback path, rather than as one big release.

The second challenge was reporting credibility. Projected savings are easy to claim and hard to verify. Because we reconciled everything against the bills, the program reported a smaller number than it could have, but the finance team could verify every figure.

Custom Solutions

Banshee. An in-house notification service for PRs and deployments, built to replace n8n. We built it from scratch in under a week: repository, migration of the logic and data schema, Helm chart, CI, image registry with image-writer, GitOps, and ingress. Per-execution payments to n8n stopped. We deliberately left this saving out of the total because there was no confirmed dollar figure, even though the risk of an uncapped bill was gone.

GitHub webhook router refactor. While Banshee was in development, we moved the highest-volume workflows to a dedicated worker. n8n invocations dropped by about 85%, and the bill started going down before the switch to the new service.

Log volume notification service. It shows each team how many logs it generates and records whether the team acted on the notification. This turned tags into a feedback mechanism rather than just data for a report.

Spend limit alerts for the gas provider. A warning at 50% of the limit, a page to on-call at 90%, and repeat alerts while the condition stays active. Previously, the team learned the limit had been hit only after the service stopped. Now it gets warned before the limit runs out and has time to respond.

Reliability and Security Alongside Cost Work

Access. During a security review of the mirrord rollout, we found a dangerous combination of permissions: cluster-admin plus cross-cluster access. It could have allowed unnoticed access to production data. The excess permissions were audited and removed.

Database. A brief cross-region loss of database connectivity was spotted right away by the on-call team. In under two minutes, they confirmed the connection had recovered on its own and all nodes were healthy. No incident was declared, and users noticed nothing.

CI. A CI outage blocked the shared build queue ahead of a cross-team branch cut. The team cleared over 100 queued jobs the same day, and the release that depended on them shipped on schedule.

Stack

Cloud & platform: AWS (EKS, RDS, Fargate, S3, ECR, ElastiCache, DynamoDB, Inspector, IAM, VPC), Kubernetes, Karpenter, Helm, ArgoCD / GitOps, Docker

Databases: CockroachDB, DynamoDB

Observability: Datadog (infrastructure, APM, logs)

IaC & CI/CD: Terraform, GitHub Actions

Other: mirrord, Linear

Specialized Tools

• Karpenter cost-allocation tags for short-lived autoscaler nodes;
• Datadog log index exclusion filters and log archiving;
• ECR lifecycle policies;
• mirrord for local development connected to the cluster. Its rollout surfaced an access issue.