ADR 0003. Silver layer of typed transactions

Status: accepted. Implemented 2026-08-24 in kevinvu184/up. Builds on ADR 0002; amended by ADR 0004, which moves promotion to its own cron. Amended 2026-08-24: the same-timestamp stall guard described under "Coarse watermark" now fails the run loudly instead of exiting cleanly — a cluster at or above the batch size would repeat the identical read forever and the pre-check would then hide it, so it throws; see jobs/silver.cron.ts.

Context

Bronze holds raw JSON, which answers "what did Up say" but not "what can I query". This change derives a silver_transactions table of cleaned, typed records. The constraints are unchanged: one production database, no staging, the 45 second budget, and the project's rule that quiet corruption is the failure class to design against.

Decisions

Silver is derived and disposable

Bronze is the source of record; silver holds nothing that exists only in silver. A single meta key, silver_watermark, tracks the highest bronze last_seen_at already transformed, and deleting that key rebuilds silver from all of bronze on the next run. This is the escape hatch for every future interpretation change: no data migration, just reset and rerun. It is safe precisely because the transform is pure and the upserts are idempotent.

One pure transform file owns all interpretation

transform.ts exports toSilver(id, raw), a pure function whose only inputs are its arguments and whose only output is its return value: no sqlite, no fetch, no clock. It returns the typed record or a rejection naming the field that failed and why. The cron holds no transform logic, the store holds no interpretation, so a change to what silver means edits this file and the schema migration and nothing else. Purity also made it testable before any database existed.

Typed means typed, with two cleaning rules baked in

Money comes from valueInBaseUnits only, stored as signed integers; the decimal value string is never parsed, and floats never touch money. Timestamps are normalised to UTC ISO strings at transform time, because Up sends mixed +10:00 and +11:00 offsets across daylight saving and mixed offsets break lexicographic ordering in SQLite. Required fields (account, status, description, amount, currency, createdAt) reject when missing or wrongly typed; optional fields tolerate absence but reject malformation. A rejection is never a default.

Fail loud, lose nothing

A row the transform rejects fails the run through the existing error path rather than landing in a quarantine table. Bronze keeps the raw, so the cost of a rejection is a rerun after a code fix, never data. A quarantine table was considered and dropped: it is more machinery for a failure mode a single-developer pipeline handles by reading last_error, which carries the exact id and field.

Coarse watermark with atomic batches

Each pass selects bronze rows with last_seen_at >= watermark, transforms them, and lands each batch as one transaction: the upserts plus the watermark advance together, the same rule as bronze pages. The >= comparison re-transforms rows at the boundary timestamp on the next pass; a bronze page shares one timestamp and holds at most 100 rows, upserts are idempotent, so the rework is bounded and free, and keyset pagination on a compound cursor would buy nothing but complexity. The batch size (500) sits above any same-timestamp cluster so the watermark can always advance, and a stall guard exits cleanly if it ever cannot.

Migration 3 through the existing runner

The table arrives as the third numbered migration, applied atomically with its version write. The runner did not change.

Consequences

Silver is queryable with correct ordering and integer money, and every interpretation question has one home. A transform rejection halts promotion until the code is fixed, which is deliberate: the alternative is a silver table that silently disagrees with bronze. As first built, promotion ran as a fourth step inside the poller; ADR 0004 records why that placement was wrong and where it moved.