A dashboard you cannot diff

I put Metabase over two million Nashville 311 service requests to see whether it could carry self-serve analytics on genuinely messy public data. On that question it did fine — the community DuckDB driver held, the star schema cost nothing, cold queries over R2 ran ~3s and warm ones under a second.

None of that is what I want to write about. The interesting failure was narrower and much more annoying: I had a Bayesian changepoint analysis of that same data, and I wanted to put it on a dashboard. Getting one chart onto a page took 659 lines of Python, and when I was done there was nothing I could review, diff, or hand to anyone.

The chart I needed

The analysis produces, per series, three things: a fitted rate that is flat inside a regime and steps between them, a 90% credible interval around it, and a separate posterior probability that a regime boundary falls in each month. Reading it requires both together — the top says how much changed, the bottom says how sure the model is about when. They have to share an x-axis, or the pair is meaningless.

This is not an exotic chart. It’s a line with a band and a bar panel under it.

What it cost in Metabase

There is no band. Metabase’s line chart has no ribbon or interval mark, so the interval has to arrive as data. The query UNION ALLs the series into one long table — observed, fitted, lower bound, upper bound — as four named series, with the two bounds given the same colour and label so they read as a band without being one:

SELECT month AS "Month", '1. Observed' AS "Series", ...
UNION ALL SELECT month, '2. Model estimate',    round(lambda_fitted, 3) ...
UNION ALL SELECT month, '3. Credible interval', round(lambda_lo, 3) ...
UNION ALL SELECT month, '4. Credible interval', round(lambda_hi, 3) ...

The numeric prefixes are load-bearing. Left alone, Metabase ordered the legend with the model estimate after both interval bounds, so the series order had to be declared explicitly, and each series styled by string key:

"series_settings": {
    "2. Model estimate":     {"color": "#2a78d6", "line.size": "L"},
    "3. Credible interval":  {"color": "#9ec5f4", "line.size": "S",
                              "line.marker_enabled": False},
    ...
}

There is no faceting, so the two panels are two separate cards stacked on the dashboard grid. Which introduces a problem I did not anticipate: they wouldn’t line up. Metabase sizes each card’s plot area independently, and an axis title on one and not the other is enough to shift the x-positions apart. Two stacked panels that don’t share an x-position cannot be read together — which is the entire point of the pair. The fix in my source is a comment and a deletion:

# No axis titles. They were the only thing making this panel's plot area a
# different width from the one below, and two stacked panels that do not
# share an x-position cannot be read together -- which is the entire point.
"graph.x_axis.title_text": "",
"graph.y_axis.title_text": "",

I removed the axis labels from a statistical chart to trick two cards into alignment. Then the goal-line label had to go too, because it printed across the data at the right-hand edge, which is where the interesting part is. And the probability bars sit one month late by construction — the model places a boundary after a month, so the new regime starts the following one — which is fixed with month + INTERVAL 1 MONTH in SQL, because there is nowhere else to put it.

Every one of these lives in a visualization_settings dict as an untyped magic string: graph.y_axis.auto_range, stackable.stack_type, graph.series_order_dimension. There is no schema for that object. I found the keys by building a chart in the UI and reading back what the API returned.

And then you cannot find it

The second problem is one I’d have called a non-problem before doing this. From a comment in my own script:

Everything changepoint-related lives in one collection. Left in the root it sits among the project’s own models and metrics — 25 questions in one undifferentiated list is most of why this is hard to navigate.

Metabase’s unit of storage is the saved question, and they accumulate in a flat namespace. Fifteen cards for this dashboard, plus the two models and six metrics from the semantic layer, plus everything exploratory, and the list stops being browsable well before it stops growing. Collections help, but they’re a folder you have to remember to file into — and filing has to be done by the script, because I certainly wasn’t going to do it by hand every rebuild.

Which exposes the real issue: there is no declarative state, so the script has to do its own garbage collection.

for name, card in existing.items():
    if name.startswith(("Local changes detected", "Places that changed",
                        "Systemic months", "Biggest local changes",
                        "When Nashville changed", "Units with the most",
                        ...)) and name not in keep:
        mb.call("PUT", f"/api/card/{card['id']}", {"archived": True})

That is a hand-maintained list of names from previous versions of my own dashboard, matched by prefix, archived so the question list stays honest. Cards are identified by name, because name is the only stable handle I have. Rename a card and you orphan it; the script grew a rename-adoption path for exactly that reason.

And the filters are wired to Metabase’s internal field IDs:

FIELD_IDS = {"geo_type": 240, "unit": 241, "metric": 242}

Autoincrement integers from the application database. They are correct on my instance and meaningless on yours. Rebuild the app DB and the dashboard silently points at the wrong fields — the same fragility as the star schema’s foreign keys, which the DuckDB driver also doesn’t sync and which also have to be declared through the API. The relationships, the display names, the descriptions, the dashboard itself: all of it is configuration living in a mutable server, not schema living in the repo.

So I have “dashboards as code” in the least useful sense. There is a file, and it is 659 lines, and it is an imperative program that mutates a remote server. I cannot diff two versions of the dashboard. I cannot review a change to it in a pull request in any meaningful way — the diff shows Python, not what the chart became. I cannot check out a branch and see the old one.

What Lightdash does instead

I’ve run the other pattern at work, on an insurance warehouse, and the difference is not a feature — it’s which artifact is authoritative.

The semantic layer is dbt YAML. Dimensions and metrics are declared in schema.yml next to the model they describe, so the metric definition and the SQL it depends on move in the same commit and get reviewed by the same person. There’s no separate place to go declare that a column is a currency, and no way for the two to drift apart.

Charts and dashboards are also files. lightdash download writes them into lightdash/charts/ and lightdash/dashboards/ as YAML; lightdash upload pushes them back. A chart is its query, its filters, and its chartConfig — a thing you can read:

chartConfig:
  type: big_number
  config:
    selectedField: fct_billing_price_elasticity_avg_bill_increase_pct
    showBigNumberLabel: true
    comparisonFormat: raw

That is roughly the same information Metabase holds, with two differences that matter. It sits in the repo, so a change to a chart is a diff a reviewer can read. And the field references are names derived from the dbt model ({table}_{column}), not integers from a server’s database — so they mean the same thing on every machine.

The iteration loop is the part I actually miss. lightdash preview spins up an ephemeral project against your dbt branch, with every dependent chart pointed at it. Change a model, preview it, see what breaks downstream, before anything is deployed. Then lightdash deploy runs from CI on merge to main. Editing a metric definition and seeing every chart that consumes it re-render is a different activity from clicking through fifteen saved questions hoping you remembered them all.

There’s also more room at the bottom. Lightdash’s cartesian charts expose the underlying eCharts series config, so you can reach past the UI’s vocabulary into the charting library’s:

eChartsConfig:
  series:
    - type: bar
      encode:
        xRef: { field: bill_change_bucket }
        yRef: { field: total_bills }

Metabase’s visualization_settings is a fixed dictionary of options someone chose to expose. Lightdash’s is an escape hatch to eCharts. For a chart that isn’t a bar, a line, or a number, that distinction decides whether the thing is possible at all.

Where I should be honest

Lightdash’s YAML is not pleasant. My notes from that project are mostly a list of ways it bit me: a filter-format change that returned bare 422s until filters were split into separate metrics / dimensions / tableCalculations sections each with an id and an and array; a Cannot read properties of undefined (reading 'pivotValues') error that turned out to mean encode needed xRef/yRef with nested field keys rather than plain x/y; and a pile of boilerplate — metricOverrides: {}, additionalMetrics: [], customDimensions: [] — that must be present and empty. The format drifted between versions and I ended up writing a conversion checklist.

But look at what that debugging was. Two files, side by side, old format and new, and a diff between them. When Metabase’s equivalent knowledge changes, it changes inside a Python script that reverse-engineers an HTTP API, and you find out by reading a 500 response.

The other honest caveat: Lightdash’s files are not the source of truth by default. People edit in the UI, so the documented workflow starts with lightdash download before you begin and again before you merge, to catch manual changes and resolve them as git conflicts. It’s a two-way sync, not a one-way build. That’s a real weakness, and it’s still enormously better than having no file at all.

And neither tool would have drawn my chart properly. The version on this site is a hand-written Astro component emitting SVG, about a hundred lines, and it does the band as a band and the two panels on one shared axis because I told it to. That’s what it takes. A BI tool is for the charts a BI tool makes, and a posterior over segmentations is not one of them.

What I’d actually say

Metabase is a good product for the thing it is: a question-and-answer tool for people who want to ask a question and get a number. Handed a clean star schema with its foreign keys declared and its traps encoded as metrics, it is genuinely pleasant, and a non-SQL analyst can get real work done in it.

It is a bad fit for analysis that is authored — where the output is an argument with a specific shape, built over weeks, revised, and reviewed. Not because the charts are bad, but because the dashboard is rows in an application database and the only way to version it is to write a program that recreates it. Everything I built is reproducible and none of it is reviewable.

The code-first model wins on iteration speed, and it’s worth being precise about why. It isn’t that YAML is faster to write than clicking; it usually isn’t. It’s that a file gets you the whole apparatus for free — branches, diffs, review, CI, and the ability to preview a change against real data before it lands. Once the dashboard is a file, changing it is just changing code, and we already know how to do that safely.

---

The Metabase dashboard is 659 lines of Python building 15 cards across 4 tabs, idempotent by name-matching, with a hand-maintained archive list to garbage- collect cards from previous versions of itself. It works, it rebuilds cleanly, and I would not want to hand it to anyone.