Skip to content
All work

Forecasting data center power: $400K in avoided rack spend

Glue and dbt over 500M+ daily telemetry events, forecast to 1.9% MAPE

Planners for data center sites serving the government were building against power values padded by buffers stacked on buffers. I built the pipeline that made the telemetry plannable and the forecast that cut error to 1.9% MAPE, which released 21% more usable capacity inside the same power envelope and avoided roughly $400K of rack spend across 33 AZs.

1.9%
Forecast MAPE, against a 6.3% baseline
21%
Usable capacity released, same power envelope
$400K
Rack spend avoided across 33 AZs
6h → 30min
Weekly capacity reporting cycle
AWS Glue loads hardware telemetry into Redshift, dbt models it, and the forecast runs on top. The baseline's mean absolute percentage error band of plus or minus 6.3 percent is roughly three times the width of the ensemble model's 1.9 percent band around the same actual utilization curve. 500M+ / day hardware telemetry glue → redshift dbt marts power utilization baseline ±6.3% MAPE ensemble ±1.9% MAPE → $400K across 33 AZs
Fig. 1Glue loads the telemetry, dbt models it in Redshift, and the forecast runs on top. Cutting the error band to under a third of baseline exposed headroom the buffers had been hiding. Before, it sat inside the noise.

Context

Data center power is a forecasting problem with money attached in both directions. Over-provision and you strand capital. Under-provision and you constrain the business, risk blacking out customers, and push thermal limits on the hardware.

The existing forecast carried a 6.3% mean absolute percentage error. Because no team wanted to be the one that ran out of power, each layer added its own margin on top of the last, and the number planners actually built against was padded several times over. A site could look full while real draw sat well below the plan.

The data underneath was heavy: 500M+ hardware telemetry events a day, and getting it into a shape anyone could plan against took six hours of manual assembly every week.

The problem

Three gaps, and the modeling one wasn’t the most valuable to close first.

  • The planned values were too conservative, so real capacity went unused. The forecast was also too coarse to plan a single AZ against.
  • The data path was manual and inconsistent in granularity across databases, so the forecast was neither fresh nor standardized.
  • Demand was tracked by hand. Products like S3 Glacier had demand watched in a spreadsheet, which means a change was noticed whenever someone next happened to look.

What I built

An ELT path with the transform in the warehouse. AWS Glue jobs extract the telemetry and load it into Redshift; dbt models it there into BI-ready marts. At 500M+ events a day those models have to be incremental, so a run processes the new partition instead of rebuilding history, and the marts hold rolled-up grain rather than raw events. dbt mattered less for the transformations themselves than for making them reviewable: the definitions stopped living in one analyst’s saved queries. The marts feed five dashboards, and the weekly capacity reporting cycle went from six hours to thirty minutes for a team of eight planners.

An XGBoost forecast, blended with a Random Forest. Utilization runs several regimes at once: steady baseline draw, seasonal shape, and step changes from deployments. A single model compromised across all three, so blending let each component carry what it was good at, with peak events fed in as their own signal rather than smoothed away. That is where most of the 6.3% → 1.9% MAPE improvement came from.

Applied per site, the tighter band exposed headroom the padding had been hiding. Planned power values were recalculated against the forecast rather than against the buffer, which released 21% more usable capacity inside the same power envelope. Demand that would otherwise have been met by buying more racks got absorbed by capacity the sites already had, roughly $400K of avoided spend across 33 AZs in the government region. None of that would have shipped on the error number alone: the recalculated values had to clear the availability target the sites are designed to before anyone would build against them.

One capacity-planning layer in place of seven sources. My planning team was assembling the same picture out of seven or eight separate systems. Consolidating them into a single modeled layer meant the numbers quoted in a planning review came from one place, and the team could see related KPIs against each other instead of one tab at a time.

A natural-language query interface on Bedrock, over that layer. A planner asks in plain English; the interface translates the question into SQL against the Redshift capacity marts and returns the answer, with the visual side surfaced in Tableau. It cut data-gathering by four hours per planning project, roughly 200 hours a year across the team. The mart layer underneath is the reason it worked at all: text-to-SQL over seven inconsistent sources would have produced confident, wrong answers, which is worse than no interface.

Automated anomaly detection for S3 Glacier demand. Statistical process control limits on the global demand series, run on an Airflow schedule, replacing a spreadsheet somebody updated by hand. It surfaces about fourteen anomalies a month, early enough to act on before they become capacity misses. Control limits rather than a learned model on purpose: the series is a stable process with occasional real shifts, and a planner will act on a threshold they can explain.

The hard part

Beating the baseline honestly. It’s easy to post a better error number by evaluating on a window that flatters the model. A defensible 1.9% meant being strict about holdout periods that included the step changes, the regimes where 6.3% was actually costing decisions.

A better average is not the argument. Removing a buffer changes what a miss costs: low by a little stops being a rounding error and starts being an outage. So the average error was never the case for the change on its own. What mattered was behavior at the peaks, which is where the old forecast was worst and where the padding had quietly been doing the work.

What I’d do differently

Ship the pipeline before the model. The forecast got the attention because it was the interesting problem, but six-hours-to-thirty-minutes is what planners felt week to week, and it was the prerequisite for the model being usable at all.