Roverwatch Vision Lab is a standalone, deterministic visual test harness for developing and evaluating Roverwatch's screenshot-understanding pipeline without loading Overwatch or depending on live gameplay.

• Control Mode: https://roverwatch-vision-lab.val.run/
• Test Screen Mode: https://roverwatch-vision-lab.val.run/test
• Current machine-readable state: https://roverwatch-vision-lab.val.run/current.json
• Val source: https://www.val.town/x/pchinjr/roverwatch-vision-lab/code/

Vision Lab is intentionally separate from the main Roverwatch repository. It produces controlled visual stimuli and their expected answers. Roverwatch captures those visuals from Twitch, performs inference, and reports what it detected.

Testing computer vision against a live game makes failures difficult to reproduce:

• the game state changes constantly;
• screenshots do not have explicit ground truth;
• a missed detection may be caused by capture, compression, timing, rendering, or inference;
• reproducing the exact same visual state is difficult;
• testing requires a running game and a known game situation.

Vision Lab replaces those uncontrolled inputs with stable scenario IDs, known visuals, and explicit expected results. It lets the Roverwatch development process answer smaller questions in order:

1. Can Roverwatch capture the stream?
2. Can it retrieve and decode the screenshot?
3. Can it identify a simple shape and color?
4. Can it associate the inference with the correct active scenario?
5. Can it write the expected message to Twitch chat?
6. Can it handle text, numbers, HUD regions, motion, occlusion, noise, and temporal changes?
7. Can repeated runs be compared objectively?

The initial end-to-end proof is deliberately simple:

Vision Lab displays a green circle
        ↓
OBS sends the Test Screen to Twitch
        ↓
Roverwatch captures a frame
        ↓
The vision model reports "green circle"
        ↓
Roverwatch reads Vision Lab ground truth
        ↓
The result is marked correct or incorrect
        ↓
Roverwatch writes the result to Twitch chat

• deterministic scenario definitions;
• rendering visual test stimuli;
• a clean OBS-compatible Test Screen;
• the currently active scenario state;
• structured expected results;
• stable scenario and sequence-step IDs;
• difficulty transformations such as blur, scale, noise, and occlusion;
• local operator history;
• authoritative server-side ground-truth history and JSON export;
• capture-time ground-truth resolution;
• non-authoritative Test Screen observation records;
• explicit scheduled, published, observed, and received timestamps.

• Twitch stream acquisition;
• frame capture and S3 screenshot storage;
• screenshot selection and timestamps;
• model invocation;
• inference parsing and normalization;
• Twitch chat output;
• comparison of actual inference with expected Vision Lab results;
• evaluation metrics, latency tracking, and operational logs.

• direct calls into Roverwatch;
• Twitch credentials or chat authentication;
• reading screenshots from S3;
• model inference;
• automatic scoring;
• confusion matrices;
• labeled-image capture;
• production Overwatch UI artwork.

Keeping this boundary explicit prevents the test harness from becoming coupled to one model, one capture implementation, or one Roverwatch deployment.

The current application supports:

• separate Control and Test Screen modes;
• cross-browser synchronization between Chrome and OBS;
• shape and color scenarios;
• large and small OCR text;
• numeric percentages;
• generic game HUD states;
• multiple simultaneous HUD values;
• scale, position, contrast, brightness, blur, opacity, noise, background, and occlusion controls;
• basic motion modes;
• a timed blank → green circle → blank → red square → blank → blue triangle sequence;
• seeded random scenario selection;
• structured ground-truth display and copying;
• scenario JSON export;
• local test-session recording and export;
• keyboard shortcuts;
• an HTTP endpoint for the currently active state;
• append-only server-side transition history;
• capture-time state lookup through /api/state-at;
• separate Test Screen observation history;
• ground-truth-only server exports.

1. Open Control Mode.
2. Click Open Test Screen.
3. Place the two windows side by side.
4. Select Green Circle in Control Mode.
5. Confirm that the Test Screen shows only a green circle.
6. Confirm that the Expected Result panel reports shape circle and color green.
7. Select Red Square and Blue Triangle and confirm that the previous scenario's difficulty settings do not leak into the new one.
8. Run the baseline timeline and confirm that blank states appear between shapes.

1. Add a Browser Source in OBS.
2. Use https://roverwatch-vision-lab.val.run/test as its URL.
3. Set width to 1920 and height to 1080.
4. Leave custom CSS empty.
5. Disable Shutdown source when not visible if sequences must keep running while the source is hidden.
6. Keep Refresh browser when scene becomes active disabled unless a fresh load is desired.
7. Change scenarios from Control Mode.
8. Confirm that the OBS source updates within approximately 500 ms.

The Test Screen has no controls, requires no mouse input, hides the cursor, fills the viewport, and polls shared state while it is open.

frontend/scenarios.ts
  Scenario types, defaults, presets, baseline sequence
                    ↓
frontend/components/App.tsx
  Control Mode, Test Screen, renderer, timeline, session recorder
                    ↓
POST /api/state
  Atomically appends a ground-truth transition and updates current state
                    ↓
Val-scoped SQLite
  Current state + immutable transitions + separate observations
                    ↓
GET /api/state, /current.json, /api/history, /api/state-at
                    ↓
Test Screen polls every 500 ms, renders the revision, then records observedAt

File	Responsibility
index.ts	Hono HTTP entrypoint, routes, SQLite initialization, shared-state reads and writes
frontend/root.tsx	HTML shell, metadata, Twind loader, immutable frontend entry URL
frontend/index.tsx	Mounts the React application
frontend/components/App.tsx	Control Mode, Test Screen, renderers, timeline, random mode, exports
frontend/scenarios.ts	Scenario contracts, presets, default difficulty, baseline sequence
frontend/favicon.svg	Application icon
README.md	Architecture, operations, extension, and Roverwatch integration contract

• Val Town hosts the application and HTTP routes.
• Hono handles routing.
• React 18.2 renders both modes.
• Twind provides utility styling without a build process.
• Val-scoped SQLite stores shared display state.
• Browser localStorage stores control-only history and seed data.
• Immutable versioned frontend URLs prevent stale JavaScript after a deployment.
• Polling is used because Val Town does not accept incoming WebSocket connections.

Understanding this distinction is essential for maintenance.

The current screen state is stored in the val-scoped SQLite table vision_lab_state. There is exactly one row, with ID 1.

Column	Meaning
id	Singleton row ID
payload	JSON-encoded active LiveState
revision	Monotonically increasing state revision
updated_at	Server timestamp of the latest update

Shared state exists because Chrome and OBS use separate browser profiles. They cannot reliably share localStorage or BroadcastChannel state. Control Mode therefore publishes to the server, and Test Screen Mode polls the server.

Every accepted state update is also appended to vision_lab_transitions in the same SQLite batch as the current-state update. A transition can never be edited through the application API. It contains revision, scenario ID, step, scheduled time, server publication time, expected JSON, and the resolved state JSON.

Test Screen observations are written to vision_lab_observations. They are explicitly marked non-ground-truth and never change a transition. Each observation records revision, observer ID, client observation time, and server receipt time.

The current shared state is intentionally last-write-wins. Opening multiple Control Mode windows means whichever window publishes last controls every connected Test Screen. Historical transitions remain append-only.

The key roverwatch-vision-lab.v1 stores:

• the current random-test seed;
• recent session history.

This data is local to the browser profile. It is not visible to OBS and is not intended as authoritative evaluation data until exported.

A scenario separates what is rendered from what Roverwatch is expected to detect.

interface Scenario {
  id: string;
  name: string;
  description: string;
  category: "shape" | "ocr" | "hud" | "sequence";
  visualState: {
    kind: "blank" | "shape" | "text" | "number" | "hud";
    // Renderer-specific fields
  };
  expectedResult: Record<string, unknown>;
  difficulty?: Partial<Difficulty>;
}

Example:

{
  "id": "shape.green-circle",
  "name": "Green Circle",
  "description": "Green Circle recognition baseline",
  "category": "shape",
  "visualState": {
    "kind": "shape",
    "shape": "circle",
    "color": "green"
  },
  "expectedResult": {
    "scenarioType": "shape",
    "visible": true,
    "shape": "circle",
    "color": "green"
  }
}

A LiveState is the resolved scenario being shown now.

interface LiveState {
  scenario: Scenario;
  difficulty: Difficulty;
  step: number;
  scheduledAt: string;
  publishedAt?: string;
  observedAt?: string;
  revision?: number;
}

The server assigns publishedAt and the authoritative revision. A Test Screen adds its own observedAt only after it receives that revision. The current-state response also includes updatedAt for compatibility.

Timestamp	Clock owner	Meaning	Authoritative use
scheduledAt	Control browser	When the controller intended the state to begin	Sequence intent and controller delay
publishedAt	Vision Lab server	When the ground-truth transition and current state committed atomically	Authoritative state timeline
observedAt	Individual Test Screen	When that client first received the revision	Display-observation latency only
receivedAt	Vision Lab server	When an observation record reached the server	Observation transport diagnostics
Roverwatch capturedAt	Roverwatch capture process	When the evaluated frame was acquired	Lookup key for /api/state-at
Roverwatch inference timestamps	Roverwatch	When inference started and completed	Model and pipeline responsiveness

These clocks may be skewed. Use publishedAt to define the authoritative ground-truth timeline. Use capturedAt to resolve a frame against that timeline. Treat observedAt as evidence that a particular Test Screen received a revision, not proof that Twitch had already encoded and delivered it.

The transitions table and ground-truth exports contain only scenario intent, resolved visual state, expected result, revision, and authoritative timing. They never contain model output, Twitch chat output, confidence, match status, or pass/fail decisions.

Observations are stored in a separate table and are labeled non-ground-truth. Roverwatch results must remain in Roverwatch storage. Evaluation is a later join:

Vision Lab immutable transition
        +
Roverwatch frame and inference record
        ↓
Comparison record and responsiveness metrics

This separation makes it possible to change scoring logic without rewriting the historical expected record.

interface Difficulty {
  scale: number;
  position: "center" | "top-left" | "top-right" |
            "bottom-left" | "bottom-right" | "random";
  contrast: "high" | "medium" | "low";
  brightness: number;
  blur: number;
  opacity: number;
  noise: number;
  background: "solid" | "gradient" | "simple" | "busy";
  occlusion: number;
  motion: "stationary" | "horizontal" | "vertical" |
          "diagonal" | "bounce" | "random-walk";
  speed: number;
}

The Control Mode ground-truth panel wraps the scenario's expected result with execution context:

{
  "scenarioId": "shape.green-circle",
  "scenarioName": "Green Circle",
  "step": 0,
  "revision": 217,
  "scheduledAt": "2026-08-28T20:33:01.614Z",
  "publishedAt": "2026-08-28T20:33:07.480Z",
  "expected": {
    "scenarioType": "shape",
    "visible": true,
    "shape": "circle",
    "color": "green"
  }
}

Ground truth comes from scenario.expectedResult. It is not inferred from the rendered pixels. That separation is what makes the harness useful for evaluation.

The application is public and currently uses these routes:

Method	Route	Purpose
GET	/	Control Mode
GET	/test	Clean Test Screen
GET	/api/state	Current shared LiveState
POST	/api/state	Replace current shared LiveState
GET	/current.json	Public machine-readable alias for current state
GET	/api/history	Append-only ground-truth transitions after an optional revision
GET	/api/state-at?at=...	Ground truth that had been published at a capture timestamp
GET	/api/observations	Non-ground-truth Test Screen observations
POST	/api/observations	Append a Test Screen observation without changing ground truth
GET	/source	Redirect to editable Val source
GET	/__immutable/*	Versioned frontend modules and assets

curl https://roverwatch-vision-lab.val.run/current.json

The main Roverwatch process should initially treat this endpoint as a debugging and correlation source, not as a production dependency.

POST requests must contain:

• scenario.id;
• scenario.visualState;
• scenario.expectedResult.

Payloads above 100,000 characters are rejected. The server increments revision, stores the complete payload, and returns the saved state.

Publishing a state and appending its transition occur atomically in one SQLite batch. The transition history endpoint is marked groundTruthOnly: true; the observations endpoint is marked groundTruth: false.

For capture-time correlation, Roverwatch should request:

GET /api/state-at?at=<URL-encoded capture timestamp>

The endpoint returns the most recent transition whose server-issued publishedAt is less than or equal to the capture time.

All endpoints are currently public. Do not place secrets, Twitch credentials, private screenshots, user data, or model credentials in scenario state.

ScenarioStage reads two separate inputs:

1. scenario.visualState selects the content renderer.
2. difficulty transforms the rendered target and background.

Current renderer branches are:

Kind	Renderer behavior
blank	Renders no target
shape	Renders circle, square, or CSS triangle
text	Renders exact text with size, alignment, and colors
number	Uses the text renderer for numeric recognition
hud	Renders health, ultimate, life state, team count, and enemy count

Do not derive expected results inside render components. Renderer code controls appearance; scenario definitions control expected meaning.

Stable IDs are required because Roverwatch logs and evaluation results will eventually join against them.

Recommended format:

<category>.<specific-state>

Examples:

shape.green-circle
shape.red-square
ocr.large-text
number.ult-99
hud.player-dead
hud.ult-ready
difficulty.occluded-target
motion.moving-target
sequence.baseline.02-green

Rules:

• IDs should be lowercase and machine-readable.
• Do not reuse an ID for a semantically different test.
• Visual refinements that preserve the same intended detection may keep the ID.
• Changed expected meaning should receive a new ID.
• Sequence steps need stable IDs, not only numeric indexes.
• For versioned evaluation sets, add an explicit version namespace later rather than silently changing the old set.

Preset difficulty overrides are scenario-local. Selecting a new preset or moving to a new sequence step must:

1. begin with defaultDifficulty;
2. merge only the new scenario's difficulty overrides;
3. publish the resolved result.

Never merge a newly selected preset onto the previous scenario's resolved difficulty. Doing that causes state leakage: a moving or occluded preset can make every later scenario move or remain occluded.

Manual difficulty controls intentionally modify the active scenario after selection. Selecting another preset resets those controls to defaults plus the new preset overrides.

This rule is enforced through:

merged(nextScenario, defaultDifficulty)

Treat it as an invariant when changing scenario-selection, sequence, or random-mode code.

The initial sequence proves temporal capture and classification:

Step ID	Duration	Expected visual
sequence.baseline.01-blank	8 seconds	Blank
sequence.baseline.02-green	15 seconds	Green circle
sequence.baseline.03-blank	8 seconds	Blank
sequence.baseline.04-red	15 seconds	Red square
sequence.baseline.05-blank	8 seconds	Blank
sequence.baseline.06-blue	15 seconds	Blue triangle

Blank intervals matter. They let Roverwatch demonstrate that it detects absence, does not repeat a stale inference, and can associate frames with state transitions.

Random mode uses a deterministic seeded pseudo-random generator. Given:

• the same seed;
• the same preset ordering;
• the same interval;
• the same starting point;

it should produce the same selection order.

Changing the preset array order changes the generated scenario sequence even when the seed remains the same. Treat preset ordering as part of the reproducibility contract until random mode records the generated schedule explicitly.

Starting a session records the current revision. End + export then downloads authoritative transitions after that revision. Export ground truth downloads the available history independently of a session.

The server export has this shape:

{
  "groundTruthOnly": true,
  "sinceRevision": 216,
  "count": 1,
  "transitions": [
    {
      "revision": 217,
      "scenarioId": "shape.green-circle",
      "step": 0,
      "scheduledAt": "2026-08-28T20:33:01.614Z",
      "publishedAt": "2026-08-28T20:33:07.480Z",
      "expected": {
        "scenarioType": "shape",
        "visible": true,
        "shape": "circle",
        "color": "green"
      },
      "state": {
        "scenario": {},
        "difficulty": {},
        "step": 0,
        "scheduledAt": "2026-08-28T20:33:01.614Z",
        "publishedAt": "2026-08-28T20:33:07.480Z",
        "revision": 217
      }
    }
  ]
}

Exports contain expected state only. Roverwatch inference, chat output, match status, and mismatch decisions must be stored in Roverwatch and joined afterward. This prevents model output from contaminating the source used to judge that output.

Browser-local operator history remains useful for immediate UI feedback, but responsiveness analysis should use server transitions, Test Screen observations, and Roverwatch capture/inference timestamps.

Use this first.

1. Open Control Mode and Test Screen.
2. Stream Test Screen through OBS to Twitch.
3. Select one scenario at a time.
4. Let Roverwatch capture a frame and run inference.
5. Inspect Roverwatch logs and Vision Lab's Expected Result panel.
6. Confirm the model output and Twitch chat message manually.

Success criterion: Roverwatch consistently distinguishes blank, green circle, red square, and blue triangle.

When a screenshot is selected for inference:

1. Roverwatch records its frame capture timestamp as close to acquisition as possible.
2. Roverwatch requests /api/state-at?at=<capturedAt>.
3. Roverwatch records scenarioId, revision, scheduledAt, and publishedAt from the returned transition.
4. Roverwatch records the frame capture timestamp and S3 object key.
5. Roverwatch records normalized inference output and model latency.
6. The combined record is written to Roverwatch logs.

Suggested evaluation record:

interface RoverwatchVisionEvaluation {
  frameId: string;
  s3Key: string;
  capturedAt: string;
  inferenceStartedAt: string;
  inferenceCompletedAt: string;
  model: string;
  visionLab: {
    scenarioId: string;
    revision: number;
    startedAt: string;
    updatedAt: string;
    expected: Record<string, unknown>;
  };
  actual: Record<string, unknown>;
  matched?: boolean;
  mismatchReasons?: string[];
}

A frame captured near a scenario change may show the previous state while /current.json already reports the next state.

Do not compare a frame blindly against whatever state is current after inference finishes.

Recommended approach:

1. Read Vision Lab state close to frame capture, not after a slow model call.
2. Resolve the frame through /api/state-at using the capture timestamp.
3. Record revision, scheduledAt, and publishedAt.
4. Use Test Screen observations to estimate controller-to-display delay without treating observations as ground truth.
5. Add a transition-settling window before scoring a new state.
6. Mark ambiguous transition-window frames as excluded rather than incorrect.

The baseline sequence now uses 8-second blank states and 15-second target states. Continue excluding frames near transitions until measured Twitch and capture latency supports a narrower settling window.

After correlation is reliable, normalize expected and actual outputs into a shared result schema and calculate:

• exact-match accuracy;
• per-scenario accuracy;
• false positives during blank states;
• false negatives;
• confusion pairs;
• model latency;
• end-to-end capture-to-chat latency.

Keep raw model output alongside normalized output. Normalization bugs must be distinguishable from model errors.

1. Open frontend/scenarios.ts.
2. Add an object to presets.
3. Assign a stable, unique ID.
4. Define only visual fields used by the selected renderer kind.
5. Write an explicit, machine-readable expected result.
6. Add only scenario-specific difficulty overrides.
7. Select the new preset in Control Mode.
8. Verify the Test Screen.
9. Verify /current.json.
10. Select a normal preset afterward and confirm no difficulty state leaked.

Example:

{
  id: "ocr.player-dead",
  name: "Player Dead Text",
  description: "Large death-state OCR target",
  category: "ocr",
  visualState: {
    kind: "text",
    text: "PLAYER DEAD",
    fontSize: 128,
    align: "center",
    foreground: "#ffffff",
    background: "#070b12"
  },
  expectedResult: {
    scenarioType: "text",
    text: "PLAYER DEAD"
  }
}

A new scenario type changes both the contract and renderer.

1. Add the new kind to ScenarioKind.
2. Add its visual fields to the Scenario type, or refactor visualState into a discriminated union.
3. Add a renderer component in App.tsx.
4. Add one branch in ScenarioStage.
5. Create at least one baseline preset.
6. Define the expected-result schema.
7. Confirm difficulty transformations still work.
8. Confirm blank and later presets reset the new type correctly.
9. Document the expected schema here.
10. Coordinate with Roverwatch before automated comparison depends on the new schema.

If scenario types grow significantly, the next maintainability refactor should make visualState a discriminated union and move renderers into separate component files.

Sequence steps live in milestoneSequence in frontend/scenarios.ts.

Each step needs:

• a stable step ID;
• a duration in milliseconds;
• a scenario definition;
• an explicit expected result through that scenario.

When changing durations, remember that Twitch latency, frame capture cadence, S3 upload delay, polling, and model latency all consume part of the visible window. A three-second state may yield very few usable frames in a delayed pipeline.

For automated evaluation, prefer longer states until measured end-to-end latency is known.

Run this checklist after changing state, scenarios, rendering, or sequences.

• Control Mode returns HTTP 200.
• Test Screen returns HTTP 200.
• Frontend modules return JavaScript, not HTML.
• /current.json returns valid JSON.
• /api/history returns append-only ground-truth transitions.
• /api/state-at resolves the expected revision for a known timestamp.
• /api/observations remains separate and marked non-ground-truth.
• The Test Screen has no visible controls.
• The OBS source updates after Control Mode changes.

• Green Circle renders green and circular.
• Red Square renders red and square.
• Blue Triangle renders blue and triangular.
• Expected Result matches the rendered scenario.
• Stable IDs appear in /current.json.
• Blank renders no target.
• Timeline steps occur in the documented order.

• Select Moving Target.
• Select Green Circle.
• Confirm motion is stationary.
• Select Partially Occluded Target.
• Select Red Square.
• Confirm occlusion is 0%.
• Select Busy HUD.
• Select Blue Triangle.
• Confirm background is solid and noise is 0.
• Refresh both Control and Test Screen.
• Confirm the current state remains internally consistent.

• OBS uses 1920×1080.
• The source continues while unfocused.
• Twitch receives the screen.
• Roverwatch captures the intended frame.
• Roverwatch logs S3 key, capture time, scenario ID, and revision.
• Twitch chat output matches the expected result.

Cause: the new scenario was merged onto the previous scenario's resolved difficulty.

Fix: make preset, sequence, and random selections resolve from defaultDifficulty, then merge only the new scenario override.

Refresh reads the shared SQLite state. This is intentional so OBS and Control Mode stay synchronized. Select a clean preset or Reset difficulty. If the stored payload is internally inconsistent, inspect the singleton vision_lab_state row.

Check:

• GET /api/state;
• Val Town logs;
• SQLite availability;
• browser network errors;
• whether the val's HTTP access is still public.

• Confirm Control Mode says synced.
• Confirm /current.json changes.
• Wait at least 500 ms.
• Refresh the OBS Browser Source.
• Confirm OBS points to /test, not the control URL.
• Confirm the source was not shut down while hidden.

The wrong URL is loaded. Use /test.

The HTML shell stamps versioned immutable module URLs. Reload the page so it receives the new asset version. Avoid hardcoding an old /__immutable/<version>/... URL.

This is probably a correlation race rather than a model error. Increase state duration, exclude transition windows, and log revision plus timestamps at capture time.

Check whether preset ordering, interval, starting state, or deployment version changed.

• The val and /current.json are public.
• The state update endpoint is also currently public.
• Do not store credentials or sensitive data in scenario payloads.
• A third party with the URL could change the active state.
• Before using this beyond a development spike, add update authorization or restrict HTTP access while preserving a safe read path for Roverwatch.
• The SQLite database is val-scoped. Do not switch the import to organization-global SQLite.
• The app stores only one current shared state; it is not an audit log.
• Multiple simultaneous control sessions are last-write-wins.
• Polling every 500 ms creates ongoing reads from every open Test Screen. Revisit this if usage grows.

• Motion labels are broader than the current visual implementations; several modes currently share simple CSS animation behavior.
• The speed control is represented in state but is not fully applied to every motion mode.
• Random position is currently deterministic and fixed rather than seed-derived per event.
• Sequence editing is not yet exposed in the UI.
• Session history is local to one browser.
• Transition history begins at the state present when the history schema was introduced; earlier revisions were not reconstructable.
• Ground-truth export is capped at 1,000 transitions per request and does not yet paginate automatically.
• There is no screenshot capture inside Vision Lab.
• There is no automatic inference comparison.
• There is no authentication on state updates.
• Ground-truth timestamps are not a distributed clock guarantee.
• Current state polling can observe a transition after the captured Twitch frame was produced.

These limitations should be treated as explicit backlog items, not assumed capabilities.

Priority order for Roverwatch development:

1. Complete the simple-shape Twitch capture and chat spike.
2. Resolve and log Vision Lab scenario ID and revision for each frame through /api/state-at.
3. Measure scheduled-to-published, published-to-observed, capture, upload, inference, and chat latency separately.
4. Add pagination and retention policy for transition and observation history.
5. Define shared normalized expected/actual result schemas.
6. Add automated comparison and mismatch reasons in Roverwatch, not in ground-truth history.
7. Measure capture, upload, inference, and chat latency separately.
8. Add versioned scenario sets.
9. Add screenshot labeling and evaluation exports.
10. Add update authentication before broader use.
11. Refactor renderers and visualState types as scenario complexity grows.
12. Add model accuracy, confusion, and latency reporting.

• Preserve the separation between visual state and expected result.
• Treat scenario IDs as durable API identifiers.
• Reset scenario-local difficulty when changing scenarios.
• Keep Test Screen free of controls and metadata.
• Prefer deterministic behavior over visual novelty.
• Record raw outputs before normalization.
• Do not classify transition-race frames as model failures.
• Keep the main Roverwatch integration read-only until the contract is stable.
• Document changes to schemas, IDs, timing, and synchronization behavior.
• Add regression scenarios for every production vision failure worth reproducing.

The first Roverwatch vision spike is complete when:

• Vision Lab is streamed through OBS to Twitch;
• Roverwatch captures screenshots from that Twitch stream;
• Roverwatch recognizes blank, green circle, red square, and blue triangle states;
• each capture is logged with S3 key, capture timestamp, Vision Lab scenario ID, and revision;
• Twitch chat receives the normalized detection;
• repeated runs can reproduce and explain failures;
• transition-window frames are excluded or labeled ambiguous.

At that point, the project has proven the full transport and inference loop. More realistic HUD tests should be added only after the simple pipeline is reliable.

---

Latency evaluation is an additive subsystem for measuring how quickly a published Vision Lab revision reaches the browser, is committed by React, appears in a rendered frame, and is later captured by Roverwatch. It does not replace Control Mode, Test Screen, the immutable transition history, or the normal scenario catalog.

Use these URLs:

URL	Purpose	Visible controls
/	Control Mode, including latency-session controls and diagnostics	Yes
/test	Normal scenario output for OBS and recognition tests	No
/latency?poll=100	High-contrast latency target with revision beacon	No
/api/time	Server clock sample for offset estimation	N/A
/api/beacon/1842	Encoder/decoder diagnostic for one revision	N/A

Latency Mode defaults to a 100 ms poll interval. The only supported intervals are 100, 250, and 500 ms. Invalid query values fall back to 100 ms.

The normal Test Screen intentionally renders arbitrary shapes, text, HUDs, motion, noise, and occlusion. Those features are useful for recognition testing but make transport-latency measurement harder to decode. Latency Mode instead renders a deterministic full-screen signal whose identity is recoverable from a screenshot.

Ground truth and observations remain separate:

• vision_lab_transitions is the authoritative, append-only record of what the server published.
• vision_lab_observations is diagnostic evidence reported by a browser. It can be late, duplicated, absent, or affected by an incorrect client clock.
• Roverwatch frame, inference, and chat timestamps belong in Roverwatch. They are joined to Vision Lab data after capture.

Never promote an observation to ground truth, and never write Roverwatch's live decision into the transition history.

The timestamps have deliberately narrow meanings:

Timestamp	Assigned by	Meaning	Authoritative
scheduledAt	Controller	When the controller intended the transition to occur	No
publishedAt	Vision Lab server	When the new revision was atomically persisted and published	Yes, for server publication
clientReceivedAt	Latency browser	When the state response containing a new revision was handled	Diagnostic
renderedAt	Latency browser	After React committed that revision and two animation frames completed	Diagnostic
receivedAt	Vision Lab server	When the browser observation POST reached the server	Diagnostic
capturedAt	Roverwatch	When Roverwatch acquired the frame	Authoritative within Roverwatch

The browser records renderedAt with a double requestAnimationFrame: the first callback follows the React commit; the second gives the browser an opportunity to paint that committed DOM. This is a practical paint proxy, not proof that OBS or Twitch has already captured the pixels.

Useful derived intervals are:

• controller scheduling delay: publishedAt - scheduledAt
• poll delivery delay: clientReceivedAt - publishedAt
• browser commit/paint delay: renderedAt - clientReceivedAt
• observation return delay: receivedAt - renderedAt
• display-to-capture delay: capturedAt - renderedAt
• publication-to-capture responsiveness: capturedAt - publishedAt

Cross-machine subtraction is only meaningful after clock-offset handling. /api/time returns serverTime and unixMs. A client can estimate offset by sampling its local clock immediately before and after the request, using the midpoint as the approximate local time corresponding to the server response. Record round-trip time and prefer the lowest-RTT sample. This estimates offset; it is not clock synchronization.

Every Latency Mode frame contains two representations of the same revision:

1. a large OCR label, REVISION <unsigned integer>;
2. a 6-row by 8-column binary beacon.

The binary layout is stable and versioned by the latency export schema:

Row	Bits	Purpose
0	11111111	solid orientation/header row
1	10101010	alternating orientation/header row
2	bits 31..24	revision, most-significant byte
3	bits 23..16	revision
4	bits 15..8	revision
5	bits 7..0	revision, least-significant byte

Black cells represent 1 and white cells represent 0. The value is an unsigned 32-bit integer in big-endian bit order. The page background alternates black and white on every revision, so even a downscaled or blurred video stream exposes a strong whole-frame transition.

The shared module frontend/beacon.ts contains encodeRevision, decodeRevision, and verifyBeaconRoundTrip. The diagnostic route /api/beacon/:revision runs the same codec and reports decodedRevision plus selfTestPassed.

A screenshot decoder should:

1. crop or rectify the beacon rectangle;
2. locate the two header rows and correct orientation;
3. sample the center of each data cell, avoiding grid boundaries;
4. classify black as 1 and white as 0;
5. concatenate rows 2 through 5;
6. decode the 32-bit unsigned big-endian value;
7. compare it with OCR when OCR is available;
8. mark the frame ambiguous if the two methods disagree.

Do not infer revision from background parity alone; parity only reveals whether a revision is odd or even.

Latency Mode uses recursive setTimeout, scheduled only after the preceding fetch finishes. Requests therefore never overlap. This matters at a 100 ms interval: setInterval would otherwise accumulate concurrent requests when a response takes longer than the interval.

On a poll failure, the page:

• keeps the last successfully rendered revision;
• logs the failure to the browser console;
• schedules the next attempt after the configured interval;
• does not fabricate a new observation.

A new observation is posted only when the revision changes. Reloading the page creates or reuses a session-scoped observer ID and may report the current revision once.

Control Mode has a Latency evaluation panel.

1. Select a browser poll interval: 100, 250, or 500 ms.
2. Select a transition interval: 250, 500, 1000, or 2000 ms.
3. Select a transition count from 1 through 100.
4. Open the supplied Latency Screen URL in a normal browser or OBS Browser Source.
5. Start the sequence.
6. Keep the latency target visible for the duration of the run.
7. Stop early if needed, or allow the configured count to finish.
8. Export the session JSON.

The controller publishes one immutable revision per step. Each latency scenario includes latencySessionId, sequenceId, and sequenceIndex in its expected result so joins remain explicit. The panel also displays the current revision and the latest clientReceivedAt, renderedAt, and server receivedAt.

A 250 ms transition interval with 500 ms polling will necessarily skip some intermediate revisions. That is an expected experimental outcome, not a history failure: the server transition export still contains every published revision, while the browser observation set contains only revisions it actually received and rendered.

Start a session:

POST /api/latency-sessions/start
Content-Type: application/json

{
  "pollIntervalMs": 100,
  "transitionIntervalMs": 500,
  "count": 20
}

Stop a session:

POST /api/latency-sessions/<sessionId>/stop

Export a session:

GET /api/latency-sessions/<sessionId>/export

The export keeps authoritative transitions and non-authoritative observations in separate objects:

{
  "schemaVersion": "vision-lab-latency-v1",
  "sessionId": "uuid",
  "visionLabVersion": "vision-lab-latency-v1",
  "startedAt": "ISO-8601",
  "endedAt": "ISO-8601 or null",
  "startingRevision": 100,
  "endingRevision": 120,
  "configuration": {
    "pollIntervalMs": 100,
    "transitionIntervalMs": 500,
    "count": 20
  },
  "groundTruth": {
    "authoritative": true,
    "transitions": []
  },
  "observations": {
    "authoritative": false,
    "records": []
  }
}

An export made before Stop uses the current revision as its provisional ending revision and leaves endedAt null. For a finalized evaluation artifact, stop the session first.

vision_lab_observations retains its original columns for backward compatibility and adds:

Column	Type	Notes
client_received_at	TEXT	ISO-8601 browser response-handling time
rendered_at	TEXT	ISO-8601 double-rAF paint proxy
poll_duration_ms	REAL	Fetch duration measured by the browser
poll_interval_ms	INTEGER	100, 250, or 500
route	TEXT	Normally /latency; helps separate Test Screen observations
session_id	TEXT	Optional latency-session join key

vision_lab_latency_sessions stores session identity, start/end timestamps, revision bounds, configuration JSON, and the Vision Lab schema/version identifier. Schema changes are additive. Startup migrations tolerate columns that already exist, so redeployment does not destroy history.

Legacy Test Screen posts containing only observedAt remain accepted. For those posts, the server uses observedAt as the compatibility value for the newer browser timestamps. New latency clients should always send the explicit fields.

For each captured latency frame, Roverwatch should store:

interface RoverwatchLatencyFrame {
  sessionId: string;
  frameId: string;
  capturedAt: string;
  s3Key?: string;
  beaconRevision?: number;
  ocrRevision?: number;
  decodeStatus: "matched" | "beacon-only" | "ocr-only" | "ambiguous" | "unreadable";
  inferenceStartedAt?: string;
  inferenceCompletedAt?: string;
  chatPublishedAt?: string;
}

After the run:

1. decode every captured frame;
2. load the finalized Vision Lab latency export;
3. join frames to groundTruth.transitions by revision;
4. join browser diagnostics from observations.records by revision and session ID;
5. preserve frames with unreadable or conflicting codes as explicit decode failures;
6. calculate separate distributions for publication-to-browser, browser-to-capture, inference, and chat;
7. report skipped revisions separately from incorrectly decoded revisions.

A revision present in ground truth but absent from observations means the Latency Screen did not report rendering it. A revision present in observations but absent from the session's revision range indicates a session-tagging or export-boundary problem. Neither condition should be silently discarded.

Use a dedicated OBS Browser Source:

• URL: https://roverwatch-vision-lab.val.run/latency?poll=100
• width: 1920
• height: 1080
• no custom CSS that changes contrast, scale, or visibility
• keep the source active during the entire session
• disable source shutdown while hidden if switching scenes

Do not crop the beacon out of the frame. If the Twitch or capture pipeline rescales video, verify the beacon remains large enough for center-cell sampling before collecting a full dataset.

Check the configured polling interval, request RTT, background-tab throttling, OBS Browser Source activity, server logs, and whether the previous request was slow. Expected polling wait alone ranges from nearly zero to roughly one poll interval.

Check main-thread contention, browser/OBS CPU load, React errors, GPU pressure, and frame throttling. The double-rAF measurement depends on the browser continuing to produce animation frames.

The delay is outside Vision Lab's browser commit path. Inspect OBS capture, encoding, Twitch ingest/playback buffering, Roverwatch sampling cadence, and clock offset.

Compare transition interval with poll interval first. A polling browser cannot guarantee observation of every state when transitions are faster than polling. Use the immutable transition list as the complete publication record.

Treat the frame as ambiguous. Check crop alignment, orientation, compression, thresholding, partial-frame capture during a transition, and whether OCR read a stale/blurred label.

Browser timestamps and server timestamps may come from different clocks. Estimate clock offset with repeated /api/time samples and retain RTT. Do not rewrite the raw timestamps in the export.

• /latency?poll=100 returns HTTP 200 and contains no controls.
• The whole-screen background alternates on each revision.
• The OCR label and binary beacon encode the same revision.
• /api/beacon/0, /api/beacon/1, /api/beacon/1842, and /api/beacon/4294967295 round-trip.
• Browser polling never has more than one state request in flight.
• A changed revision produces exactly one observation per open latency client.
• clientReceivedAt <= renderedAt for a client observation.
• receivedAt is assigned by the server and is not accepted from the client.
• Session export contains all authoritative transitions in the revision range.
• Session export keeps observations in a separately labeled non-authoritative collection.
• Existing /, /test, /current.json, /api/history, /api/state-at, and legacy observation posts still work.
• Moving Target followed by a normal preset resets motion to stationary.
• Partially Occluded Target followed by a normal preset resets occlusion to zero.

• Keep the beacon layout stable within a schema version. Introduce a new version before changing rows, bit order, colors, or headers.
• Keep publishedAt server-assigned inside the atomic transition write.
• Keep renderedAt tied to a revision-specific React commit and double-rAF.
• Keep polling single-flight; do not replace recursive scheduling with an unguarded interval.
• Preserve raw timestamps and raw decoded values.
• Keep latency exports self-describing with configuration, revision bounds, and version.
• Keep experimental diagnostics out of authoritative ground truth.
• Add codec round-trip cases whenever the beacon implementation changes.
• Verify old routes and legacy observations after every schema migration.

---

This section is the definitive, immutable pixel protocol for vision-lab-latency-v1. It supersedes any earlier geometry implied by responsive CSS. A decoder can be implemented from this section and GET /api/beacon-contract without reading renderer code.

The canonical latency frame is exactly 1920×1080 pixels. Coordinates use a top-left origin, with x increasing rightward and y increasing downward.

Property	1920×1080 source pixels	Normalized	Uniform 1280×720
x	240	0.125	160
y	300	0.2777777777777778	200
width	1440	0.75	960
height	720	0.6666666666666666	480

Normalization is against the complete frame: x/1920, y/1080, width/1920, and height/1080. The primary contract is the canonical/normalized geometry. The 1280×720 values are the exact result of uniform 2/3 scaling.

The latency renderer is a full-frame SVG with viewBox="0 0 1920 1080". A complete 16:9 capture therefore preserves this coordinate system under uniform scaling.

The beacon contains 6 rows and 8 columns. Rows are indexed top-to-bottom as 0–5. Columns are indexed left-to-right as 0–7.

At 1920×1080:

Property	Pixels
border width	16
outer padding after border	12
first cell x	268
first cell y	328
cell width	166
cell height	104
horizontal gap	8
vertical gap	8
column stride	174
row stride	112

The canonical rectangle for cell (row, column) is:

cellX = 268 + column * 174
cellY = 328 + row * 112
cellWidth = 166
cellHeight = 104

The canonical center sample is:

sampleX = 268 + column * 174 + 83
sampleY = 328 + row * 112 + 52

For any supported complete 16:9 frame of width W and height H:

scaledSampleX = sampleX * W / 1920
scaledSampleY = sampleY * H / 1080
pixelX = floor(scaledSampleX)
pixelY = floor(scaledSampleY)

At 1280×720, border width is 10.6667 px, padding is 8 px, horizontal gap is 5.3333 px, vertical gap is 5.3333 px, cell size is 110.6667×69.3333 px, and the first cell begins at (178.6667, 218.6667).

Concrete acceptance example: row 3, column 5 has canonical rectangle x=1138, y=664, width=166, height=104 and center (1221,716). At 1280×720 its rectangle is approximately x=758.6667, y=442.6667, width=110.6667, height=69.3333; its floating center is (814,477.3333), so the sampled raster pixel is (814,477).

Only center-point sampling is normative. Decoders must not sample borders, padding, gaps, or cell edges.

Canonical sRGB colors are:

Element	CSS color	RGB
bit 1 cell	#000000	(0,0,0)
bit 0 cell	#ffffff	(255,255,255)
beacon border, padding, and gaps	#808080	(128,128,128)
even-revision page background	#000000	(0,0,0)
odd-revision page background	#ffffff	(255,255,255)

The fixed gray beacon surround keeps the grid boundary distinguishable from either alternating page background. Page-background parity is diagnostic only and must not be used to recover the full revision.

Classify each center sample with sRGB luminance:

luminance = 0.2126 * R + 0.7152 * G + 0.0722 * B

The thresholds are normative:

luminance <= 96  => black => bit 1
luminance >= 160 => white => bit 0
96 < luminance < 160 => ambiguous cell

If any sampled cell is ambiguous, return ambiguous-cell and no revision. Do not round an ambiguous sample toward either bit.

Rows and columns are read in row-major order: top-to-bottom, then left-to-right.

row 0: 11111111       orientation/header
row 1: 10101010       orientation/header
row 2: bits 31..24    left-to-right
row 3: bits 23..16    left-to-right
row 4: bits 15..8     left-to-right
row 5: bits 7..0      left-to-right

The payload is an unsigned 32-bit integer. Bit 31 is the most-significant bit and bit 0 is the least-significant bit. Byte and bit order are big-endian/MSB-first.

Validation fields are intentionally absent:

checksum: none
parity: none
complement bits: none
error correction: none

Header mismatch returns invalid-header; it must never return a revision. Future validation or layout changes require a new schema version.

Revision 1842 is:

decimal: 1842
hexadecimal: 0x00000732
binary: 00000000 00000000 00000111 00110010

row 2: 00000000
row 3: 00000000
row 4: 00000111
row 5: 00110010

The following are validated by committed fixtures:

• complete 1920×1080 lossless PNG;
• complete 1280×720 LANCZOS-resized PNG;
• complete 1280×720 JPEG at quality 75 with 4:2:0 chroma subsampling.

Uniform resizing of a complete 16:9 frame is supported when coordinates are scaled from the canonical frame. JPEG compression comparable to the quality-75 fixtures is supported.

The following are unsupported and must not be silently decoded:

• a cropped frame or cropped beacon;
• non-uniform aspect-ratio distortion;
• letterboxing unless the bars are removed and the active 16:9 image bounds are known;
• rotation or perspective distortion;
• a relocated beacon;
• minor blur beyond what happens incidentally in the validated resize/JPEG fixtures;
• layouts with a frame aspect ratio differing from 16:9 by more than 0.001.

The logical bit codec remains in frontend/beacon.ts.

The independent pixel contract and reference pixel decoder are in:

frontend/beaconContract.ts

The decoder returns decoded, invalid-header, ambiguous-cell, out-of-bounds, or unsupported-layout, and never returns a revision for malformed input.

Round-trip and corruption tests are in:

tests/beaconPixelDecoder.test.ts

Byte-exact committed fixtures are in:

test-fixtures/beacon-v1/

Val Town project files are text-backed, so PNG/JPEG bytes are committed as .base64 files. Decode them with base64 --decode <fixture>.base64 > <fixture>. The fixture README lists SHA-256 hashes for every reconstructed image.

The fixture matrix contains revisions 0, 1, 1842, and 4294967295 in all three validated forms: canonical source PNG, resized 1280×720 PNG, and compressed 1280×720 quality-75 JPEG.

GET /api/beacon-contract returns the exact canonical frame, beacon rectangle, normalized rectangle, 1280×720 rectangle, grid geometry, colors, payload ordering, validation policy, center-sampling formula, luminance thresholds, and transformation support.

Latency-session exports contain both:

{
  "schemaVersion": "vision-lab-latency-v1",
  "beaconSchemaVersion": "vision-lab-latency-v1"
}

vision-lab-latency-v1 is now immutable. Any future change to dimensions, placement, cell geometry, headers, payload meaning, bit order, colors, thresholds, validation rules, or supported geometry must introduce a new beacon schema such as vision-lab-latency-v2. Do not modify v1 in place.