ADR 0005. Gold layer of aggregate tables

Status: accepted. Implemented 2026-08-24 in kevinvu184/up. Builds on ADR 0003 and 0004; uses the naming scheme of ADR 0006. Marts re-grained to daily by ADR 0009. The schedule details in Consequences are historical: silver and gold now run quarter-hourly (silver :03/:18/:33/:48, gold :08/:23/:38/:53), so a transaction reaches gold within about 20 minutes of landing in bronze. Amended 2026-08-24 (second): the category-spend transfer exclusion gained a transaction_type filter (Transfer, Scheduled Transfer, Round Up) — the transferAccount relationship alone let a large volume of relationship-untagged internal transfers count as spend; see ADR 0010's context and models/goldModels.ts.

Context

Silver answers "what happened", one typed row per transaction. The questions worth asking daily are aggregates: spend by category per month, cashflow per account per month. This change adds a gold layer for them, rebuilt from silver by a third cron, gold.cron.ts, hourly at :45. dbt was explored as the tool for this layer and the exploration needed an honest verdict.

Decisions

dbt as a discipline, not a dependency

dbt-core does not fit this runtime: it is a Python CLI that needs an adapter with direct database access, vals run Deno, and the val-scoped SQLite is reachable only through the std/sqlite HTTP module, which no dbt adapter targets. Val Town also does not expose the val database to external tools, so running dbt outside the platform is not an option either. What survives the exploration is dbt's ideas: SQL-first models, one uniform materialisation wrapper the models never see, idempotent rebuilds. If the warehouse ever moves to a user-owned Turso or Postgres, the model SELECTs port to real dbt nearly unchanged, and that portability costs nothing to keep.

Every metric lives in one SQL file

goldModels.ts exports a list of { name, table, select }, pure data. Each gold table is one declarative SELECT over silver; the store wraps every model in the same DELETE plus INSERT INTO SELECT. A change to what a metric means edits one SELECT; a new gold table is one list entry plus one migration. The single coupling, stated at the top of the file: each SELECT's column order matches its table's column order, because the runner inserts positionally.

Two starter models, opinionated

Monthly category spend counts settled, negative-amount, non-transfer rows, stored as positive cents. Monthly per-account cashflow counts settled rows with transfers included, because per-account flow is the point and transfers net out across accounts. Months bucket on created_at_utc as UTC YYYY-MM: exact Melbourne months would need daylight-saving rules in SQL, and a fixed +10 offset misclassifies hours around month boundaries. Accepted limit, fixable as a model edit.

Full rebuild, atomically per model

Aggregates over a personal account are small, so each run rebuilds whole tables: DELETE plus INSERT in one transaction per model, meaning readers see the old table or the new one, never a half-built one. This removes all incremental bookkeeping from the layer, and makes gold disposable the way silver is: nothing exists only in gold.

Skip when silver has not moved

The gold run records the silver_watermark it built from inside gold_last_run. The next trigger compares and exits before touching gold when the watermark is unchanged, so the hourly schedule is free on quiet hours. Deleting gold_last_run forces a rebuild. A run that exhausts its budget mid-list fails loudly rather than recording a success that covers models it never rebuilt; each table already rebuilt is individually consistent and the next hour retries the whole list.

Migration 6 through the existing runner

Both tables and the gold_lock row arrive as the sixth numbered migration. The runner did not change.

Consequences

A transaction reaches gold within about an hour of landing in bronze (sync daily 18:00 UTC, silver :30, gold :45). Interpretation now has exactly two homes: row meaning in silverTransform.ts, metric meaning in goldModels.ts; the crons and store stay rule-free. The gold cron follows the established pattern (own lock, own state keys, no cross-job coordination), so the layer added one composition root and one pure module and touched nothing else.