Making 3,000 dashboards answerable by an AI agent

Context

AWS’s infrastructure org runs on Tableau. Thousands of dashboards encode how capacity gets measured: the definitions, the thresholds, the calculations people argue about in planning reviews.

None of it was legible to software. A dashboard is a binary archive: a zip wrapping XML, with layout, calculated fields, data-source bindings and parameters tangled together inside. A person can open one. A retrieval system can’t.

Why an org pays for this

The infrastructure supply chain buys things with long lead times. A rack arrives months after it is ordered. A data center shell takes years. Every one of those commitments is placed against a forecast, and being wrong costs money in both directions: over-build and capital sits in a building nobody is using, under-build and there is no capacity to sell, which is the more expensive mistake because the customer goes elsewhere and does not come back.

So capacity planning is a bet on the quality of its own information. And that information lived in three thousand Tableau dashboards.

That is not a filing problem. Three specific things follow from it.

The org’s operating definitions were human-readable only

Every planning review quotes a dashboard, and each dashboard encodes its own definition of what it measures: what counts as available capacity, which buffer is already applied, whether a figure is a point-in-time snapshot or a cumulative position. Those definitions are the content. Locked inside a binary workbook, the only way to reconcile two teams quoting different numbers is to find the person who built each view and ask them.

Answering a question cost hours, so fewer questions got asked

Working out which of three thousand dashboards is authoritative, opening it, applying the right filters and reading the right tab is most of a day for anyone who does not already know. The real cost is not those hours. It is the decisions that got made without the check, because the check was too expensive to run.

Institutional knowledge sat in individuals

Knowing that of two identically named dashboards one is live and the other is a frozen copy is written down nowhere. It leaves when the person leaves.

Structuring the estate attacks all three at once. Definitions become comparable instead of argued about. A question takes a sentence instead of an afternoon. And which dashboard is authoritative becomes a scored, published artifact rather than tribal memory.

The scale of what this protects is the reason to do it properly. A forecast I built the previous summer got 4.4 points more accurate and released 21% more usable capacity across 33 availability zones, worth roughly $400K in racks nobody had to buy. That was one analysis, by hand, over one internship. This is the infrastructure for making that class of decision continuously and fleet-wide instead of one heroic analysis at a time.

The problem

• Fidelity beats coverage. A summary that drops a calculated-field formula is worse than no document. It reads authoritative while being unable to answer the actual question.
• The estate moves weekly. A one-shot dump is stale before it finishes.
• Scale turns soft failures hard. At three thousand items, a 1% failure rate fails thirty times a run, and each one has to be survivable unattended.

What I built

A Step Functions state machine fanning out one branch per workbook, in parallel, nothing kept on disk:

1. Extract. Unwrap the archive, parse the XML, project it into an explicit typed contract: worksheets, calculated fields, parameters, bindings, layout. That contract is the ceiling on what can ever reach the knowledge base, so it lives in one reviewed file instead of as assumptions scattered across the codebase.
2. Transform. Render the contract into small, single-concept Markdown with machine-readable frontmatter. Chunking is a retrieval decision, so it belongs here.
3. Publish. Mirror each workbook’s folder to S3, prune keys with no source, hand off to curation.

Extract and transform are strictly pure, enforced by a check rather than a convention. Everything touching the world lives elsewhere.

I also rebuilt the platform’s data model and wrote its retrieval-evaluation framework, which is what attributed the 13% accuracy gain to the data rather than the prompt.

The hard part

Deciding what “done” means for one dashboard. One document per workbook retrieves badly: you get an enormous chunk that matches everything weakly. Splitting by concept, so a question about a calculation retrieves the calculation, moved answer quality more than any model or prompt change.

A silent failure taught me how to design failure. A formatting detail in the document metadata made the downstream ingestion layer reject 13,055 documents while our pipeline reported a clean run. Every layer behaved reasonably alone; the failure lived in the seam, and the seam had no test because the two systems were coupled only through S3.

The rule I took from it: the choice isn’t loud versus quiet, it’s fatal versus non-fatal crossed with recorded versus unrecorded. Three of those are legitimate. Non-fatal and unrecorded is where the expensive incidents hide.

Outcome

The agent answers questions about any dashboard, with a citation: what it measures, how a field is calculated, which tables feed it. Queryable data up 85%, accuracy up 13%, self-serve analytics reaching 5,000+ people. Refreshes are incremental and unattended.

What I’d do differently

Write the contract first, and a contract test across the seam on day one. The version I shipped is the third shape, and two systems exchanging documents through S3 with no shared schema will drift. The only question is which side notices, and how long after.