How BMW Group detects cost anomalies across 14, 000 cloud accounts
Example alert (redacted): yesterday’s EC2 spend was expected to be $809.30 but jumped to $2, 584.43, an overshoot of $1, 775.13 (219.34%). That single row (expected vs actual vs impact) is the unit of work for CLEA, BMW Group’s in-house FinOps engine that finds cost surprises and puts them in front of account owners daily.
The loop: baseline, detect, filter, notify
CLEA (Cloud Efficiency Analytics), built by BMW Group with Reply on AWS, follows a tight loop: ingest billing exports, normalize to a single daily grain, produce per-account-service baselines, compare yesterday’s spend to expectation, and send an alert only when the signal is truly actionable. The trick isn’t building forecasts. It’s deciding which deviations deserve human attention.
- Raw input: primary source is the AWS Cost and Usage Report (CUR) plus equivalent provider exports. CLEA ingests about 3 billion raw billing rows per month; the CUR plus enrichments expand to roughly 500 fields before aggregation.
- Normalized grain: CLEA aggregates to daily cost per account × service (T-1 lag, yesterday’s spend is analyzed today).
- Baseline: a Prophet model (Meta’s open-source library) is trained on 365 days of daily history per account-service pair, producing per-day predicted values, confidence intervals, and a 12-month rolling forecast. The forecasting layer is intentionally pluggable.
- Detection: the system computes impact = actual spend − expected spend and measures percentage deviation from the baseline.
- Noise reduction: multiple pre-filters and thresholds remove low-value and noisy signals before any owner is notified.
Practical detection rules that stop inboxes from flooding
Raw forecasts generate many deviations. CLEA’s value comes from pragmatic rules that turn deviations into credible alerts:
- Pre-exclusions: services averaging below $0.10 over the prior three days, series with fewer than 10 days of history, and known irrelevant charge types are dropped early.
- Deviation thresholds: a default of 40% (actual vs expected); a higher 60% threshold for volatile services such as AWS Glue, Amazon Athena, and Amazon EC2.
- Account-level sensitivity: accounts on a reduced-sensitivity list must exceed three times the standard thresholds to trigger an alert.
- Clustered minimum-dollar-impact: accounts are grouped by their trailing 3-month average spend and must clear a minimum absolute impact to alert:
- Cluster 1 (trailing 3-month avg < $100k) → alert if impact > $300
- Cluster 2 ($100k-$250k) → alert if impact > $500
- Cluster 3 ($250k-$500k) → alert if impact > $750
- Cluster 4 (> $500k) → alert if impact > $1, 000
- Range grouping: consecutive flagged days are merged into coherent date ranges, and historical model snapshots are referenced to avoid fragmented alerts.
“The part that took the most iteration was not the forecast. It was deciding which deviations deserve an email, and account owner feedback still drives how we tune those thresholds.”, BMW Group CLEA team
Architecture & daily run: serverless, fan-out, and speed
CLEA is built as a serverless pipeline so it can run frequently without idle infrastructure costs. The daily run processes roughly 14, 000 accounts end-to-end in about 20 minutes by using AWS Step Functions in Distributed Map mode to fan out up to 500 concurrent AWS Lambda workers.
The team reports core serverless compute costs of about $50 per month. That figure covers the Step Functions plus Lambda compute budget and is core compute only (it excludes storage, data catalog jobs, QuickSight SPICE, and any broader tooling or networking costs).
The system’s run strategy assumes a small, defined tolerance for per-account failures. CLEA accepts up to five account-level failures in a single run (details about retries and backoff policies are not published). That tolerance sets the required success rate for the daily job and shaped engineering choices around idempotency and monitoring.
Pipeline, in practical steps
The daily flow that converts 3 billion raw rows into owner emails follows these high-level stages:
- dbt (data build tool) re-partitions monthly cost data into per-account Parquet files aggregated by service and day.
- Time-based event starts the Step Functions workflow and a preparation Lambda writes the account list to Amazon S3.
- Distributed Map workers process each account’s time series and write one JSON anomaly file per account into raw S3.
- An AWS Glue job consolidates account JSON files into a single daily Parquet in the source layer. Amazon Athena exposes views. dbt models apply threshold logic, range grouping, and alert labeling.
- The alert engine reads labeled output and sends email notifications to account owners with contextual data and an attached Excel export for one-click investigation.
What an alert contains and how owners investigate
Alerts include account ID and name, account owners and department hierarchy, the affected service, anomaly date range and duration, expected vs actual spend, absolute impact and percentage deviation, accumulated impact across concurrent anomalies, and an Excel attachment with the full table. Alerts that began within the last four days trigger immediate emails. Older anomalies only re-alert if they remain ongoing.
Owners can open an Amazon QuickSight drilldown (backed by SPICE for fast analytics) to investigate dimensions like operation and usage type, for example, distinguishing EUC1-InstanceUsage:db.r6g.large from EUC1-BoxUsage:g6.48xlarge. The goal is self-service root-cause work so platform teams aren’t interrupted for routine investigations.
Roadmap, attribution, ITSM, and agentic assistance
- ITSM integration: create tickets automatically in incident-management workflows so finance and platform teams can track investigations.
- Recommendation management portal: give owners control over sensitivity and let them accept or decline suggested threshold changes.
- CloudTrail integration: surface the user, role, or configuration change behind a cost increase to make attribution immediately actionable.
- An LLM-assisted investigator endpoint: a planned agentic endpoint to synthesize anomalies, telemetry, and CloudTrail into candidate root causes and remediation suggestions (human-in-the-loop).
“An anomaly has to clear both the deviation threshold and its cluster’s minimum dollar impact before it earns an alert.”, CLEA detection policy
Known limits and transparency
CLEA is explicit about what it does not yet provide. Quantitative detection performance metrics (false positive/negative rates, precision/recall, mean time to detect) are not published. The team also hasn’t released the precise multi-cloud normalization rules, details on handling credits/refunds or currency conversions, or the specific retry/reconciliation policies for failed per-account workers. These are areas the team plans to address as they integrate telemetry and owner feedback loops.
Why this matters to business leaders
Three practical reasons a FinOps leader or C-suite should care:
- Daily, account-level baselines scale accountability. Aggregating to daily cost per account-service lets teams catch overshoot quickly and assign responsibility without chasing contextual data.
- Forecasting is necessary but not sufficient. Baselines create detections, and pragmatic filters plus owner-driven sensitivity determine signal quality and the real workload reduction.
- Serverless orchestration makes scale affordable. A fan-out model with Lambda and Step Functions processed about 14, 000 accounts in roughly 20 minutes and kept core compute costs low, freeing engineering time to improve filters and the owner experience.
Practical 30/60/90 steps to adopt this pattern
- 30 days: Aggregate billing to daily account × service and identify account owners. Start with a small pilot (50-100 accounts).
- 60 days: Build one-year baselines (or pluggable forecasts), implement a 40% deviation + minimum-dollar-impact rule, and expose a simple drilldown dashboard for owners.
- 90 days: Scale the fan-out pipeline, introduce account clusters based on trailing 3-month averages, and iterate thresholds using owner feedback. Plan integrations for ITSM and telemetry (CloudTrail) to improve attribution.
Curious readers, concise answers
- How many accounts does CLEA monitor?
CLEA monitors more than 14, 000 cloud accounts across BMW Group’s estate.
- How much raw billing data is processed monthly?
Approximately 3 billion raw billing rows per month are ingested; after enrichment the data surface expands to roughly 500 fields before aggregation to daily account-service series.
- What forecasting method is used?
CLEA uses Prophet trained on 365 days of daily history per account-service to produce per-day expected values and a 12-month rolling forecast; the forecasting layer is pluggable, and series with fewer than 10 days of history are excluded.
- How fast and how cheap is the daily run?
The full pipeline for ~14, 000 accounts completes in about 20 minutes using AWS Step Functions Distributed Map with up to 500 concurrent Lambda workers. The team reports core serverless compute costs of roughly $50/month (compute-only; excludes storage, data-catalog jobs, QuickSight SPICE, and other tooling).
- What rule decides whether an anomaly generates an email?
An anomaly must pass a percentage deviation threshold (40% default; 60% for noisy services) and exceed its account cluster’s minimum-dollar-impact before it triggers an alert; account-specific overrides can require higher thresholds.
Final thought
Large-scale cloud cost governance is less about perfect forecasts and more about shaping human attention. CLEA shows that with daily baselines, sensible filters, and owner-first tooling you can move from reactive dashboards to proactive cost control. And you can do it without drowning teams in noise.