ADR 0002. Bronze table for raw transactions

Status: accepted. Implemented 2026-08-24 in kevinvu184/up. Builds on ADR 0001.

Context

The poller from ADR 0001 walked the account and discarded the records: backfill advanced a cursor, the incremental window counted. This change keeps the records in a bronze_transactions table so later steps can read them. Both walks revisit records by design (backfill refetches at most one page after a kill, and the 30 day window re-reads the same records every day), and Up mutates records in place when a HELD transaction settles. The run budget, lock, cursor, and single production database from ADR 0001 all still apply.

Decisions

Bronze means raw, with minimal promoted columns

Each row stores the transaction resource as JSON in a raw column, plus only what the write path needs: the Up transaction id as primary key, created_at for ordering, and first_seen_at / last_seen_at. Nothing else is promoted. Status, amounts, and categories stay inside raw for a later silver step to read, because promoting them now bakes interpretation into bronze and forces a migration every time the interpretation changes. raw is the record re-serialised from the parsed page, not the wire bytes; the API is JSON and key order carries no meaning.

Upsert on id rather than append-only

A revisited record updates its existing row: raw, created_at and last_seen_at replace, first_seen_at keeps its original value. Append-only bronze would grow a copy of every record on every daily window pass and would need dedup logic downstream. Upsert with latest raw winning fits how Up behaves, since a HELD transaction settles by mutation of the same id, and a history of HELD states is out of scope. If that history ever matters it is a different schema, not a change to this one.

Atomic page writes

The page is the unit of durability. Each consumed page lands as one batch transaction: the upserts plus, during backfill, the cursor advance; the final page of history lands together with the done sentinel the same way. The previous design persisted the cursor as a separate write after counting, which allowed a kill between the two writes to leave a cursor pointing past rows that were never stored. That failure mode is now impossible by construction rather than merely unlikely.

A malformed resource fails the page

A resource missing its id or attributes.createdAt turns the whole page into an error result instead of being skipped. A silent skip would corrupt bronze quietly, and quiet corruption is the exact failure class this project is designed against. The error path from ADR 0001 (write last_error, release the lock, TTL as backstop) handles it unchanged.

Migration 2 through the existing runner

The table arrives as the second numbered migration, applied atomically with its version write. This is the case the runner was built for; the runner itself did not change.

Consequences

Writes spend the same 45 second budget as fetches and backoff, bounded at one storage round trip per page, so a large account's backfill spans several runs with the cursor carrying progress between them. The last_run_rows key now counts rows upserted per run and stays the liveness signal. One accepted limit: a mutation on a record older than the 30 day window is never picked up, so bronze keeps the last raw seen inside the window; HELD states resolve within days, so widening the window is deferred until it matters. Silver reads raw and owns all interpretation; nothing in the write path changes when the interpretation does.