Skip to content
All work

Forecasting data center power: $400K in savings

Pipeline & ensemble model on 500M+ daily telemetry events

Planners at government data center sites were coordinating builds around power values made overly conservative by buffers stacked on buffers. I built the ensemble that forecast power from trend and peak events, cutting error to 1.9% and letting planned values be recalculated so more racks could be delivered, with 99.9999% uptime held.

1.9%
Forecast MAE, against a 6.3% baseline
$400K
Cost-saving initiative informed
99.9999%
Uptime held as buffers came down
6h → 30min
Weekly reporting time
Hardware telemetry flows through dbt into Redshift and feeds a forecast. The baseline's error band of plus or minus 6.3 percent is roughly three times the width of the ensemble model's 1.9 percent band around the same actual utilization curve. 500M+ / day hardware telemetry dbt → Redshift power utilization baseline ±6.3% MAE ensemble ±1.9% MAE → $400K across 12 AZs
Fig. 1Cutting the error band to under a third of baseline exposed headroom the buffers had been hiding. It sat inside the noise before.

Context

Data center power is a forecasting problem with money attached in both directions. Over-provision and you strand capital. Under-provision and you constrain the business, risk blacking out customers, and push thermal limits on the hardware (fires!).

The existing forecast carried a 6.3% mean absolute error. Because no team wanted to be the one that ran out of power, each layer added its own margin on top of the last, and the number planners actually built against was padded several times over. A site could look full while real draw sat well below the plan.

The data underneath was heavy: 500M+ hardware telemetry events a day. Getting it into a shape anyone could plan against took six hours of manual assembly every week.

The problem

Three gaps, and the modeling one wasn’t the most valuable to close first.

  • The planned values were too conservative, so real capacity went unused. The forecast was also too coarse to plan a single AZ against.
  • The data path was manual and inconsistent in granularity across databases, so the forecast was neither fresh nor standardized.
  • Demand was tracked by hand. Products like S3 Glacier had demand watched in a spreadsheet, which means a change was noticed whenever someone next happened to look.

What I built

ELT pipelines in dbt and Redshift. 500M+ daily events modeled into BI-ready structures. dbt mattered less for the transformations than for making them reviewable, so the definitions stopped living in one analyst’s saved queries. Weekly reporting went from six hours to thirty minutes.

An XGBoost and Random Forest ensemble. Utilization has several regimes at once: steady baseline draw, seasonal shape, and step changes from deployments. A single model compromised across all three, so the ensemble let each component carry what it was good at, with peak events fed in as their own signal rather than smoothed away. That is where most of the 6.3% → 1.9% improvement came from.

Applied per site, the tighter band exposed headroom the padding had been hiding. Planned power values were recalculated against the forecast instead of against the buffer, which freed capacity for more racks in the same footprint and informed a $400K initiative across 12 AZs. Uptime stayed at 99.9999%.

One dashboard in place of seven. My planning team was assembling the same picture out of seven or eight separate sources. Creating data models and pipelines to consolidate them meant the numbers quoted in a planning review came from one place, and the team could see related KPIs against each other instead of one tab at a time.

Automated S3 Glacier demand tracking, with anomaly detection that surfaced about fourteen a month, replacing an Excel spreadsheet somebody updated by hand.

The hard part

Beating the baseline honestly. It’s easy to post a better error number by evaluating on a window that flatters the model. A defensible 1.9% meant being strict about holdout periods that included the step changes, the regimes where 6.3% was actually costing decisions.

A better average is not the argument. Removing a buffer changes what a miss costs: low by a little stops being a rounding error and starts being an outage. So the average error was never the case for the change on its own. What mattered was behavior at the peaks, which is where the old forecast was worst and where the padding had quietly been doing the work.

What I’d do differently

Ship the pipeline before the model. The forecast got the attention because it was the interesting problem, but six-hours-to-thirty-minutes is what planners felt week to week, and it was the prerequisite for the model being usable at all.