Gateway provider benchmark for deepseek/deepseek-v4.1-flash through Vercel AI Gateway. Remixed from valdottown/glm53-flash-benchmark with the model id and UI copy switched to DeepSeek. Diverges from upstream by also benchmarking gateway routing strategies (benchmark/strategies.ts) and showing ZDR / no-train flags.

Scope (v5+): only the US-based ZDR + no-train providers in US_PROVIDERS (benchmark/strategies.ts) are benchmarked, both solo and via three gateway:us-* strategies that all pass only: US_PROVIDERS. Steve's app config is the gateway:us-ordered options object. Cron: every 10 min → 144 runs/day × 48 attempts; measured ~$0.017/run ≈ $70/month at current success rates (failures cost $0, so it tracks below that).

Gotchas:

• Reasoning is ON and forced via providerOptions.deepseek.thinking = enabled (v4+): modal/wafer default thinking OFF at the gateway while particle/novita default it ON, so without the flag the comparison mixes different tasks. Keep the final prompt question pure recall so reasoning stays ~80–160 tokens under the 300 cap; in v2 an open-ended question produced no_text errors (all 300 tokens spent thinking). Any prompt/metric change → bump BENCHMARK_VERSION.
• reasoningEffort: "low" is set explicitly (v5+). The gateway default "medium" is rejected by togetherai's V4.1 chat template (reasoning_effort must be low, high, xhigh, max), which made togetherai 0 % in v4.
• tok/s (v5+) = all output tokens (reasoning included) ÷ (stream end − first reasoning/text delta); null when < 3 streamed chunks or < 100 ms window. v4's post-TTFT formula produced 80k tok/s because many providers (alibaba 93 %, wafer 33 %) deliver all text in a single chunk ~0.4 ms after the "first" text delta.
• inferenceRegion is NOT used: the gateway catalogue only declares inference_regions for baseten, so pinning geoRegion: "us" would route exclusively to baseten (16 % success). "US-based" here means company HQ, not gateway-enforced residency.
• Routing metadata lives in provider_metadata_json → $.gateway.routing.{finalProvider,totalProviderAttemptCount,planningReasoning,sort}. Only present on successful attempts.
• Request-level zeroDataRetention requires a Vercel Pro/Enterprise plan — confirmed working on this key.

Path	Type	Purpose
index.ts	http	Hono read-only app (/, /measurements, /api/results, /api/measurements)
benchmark/config.ts	script	Model + run config — change modelId here
benchmark/prompt.ts	script	Fixed multi-turn messages, BENCHMARK_VERSION
benchmark/attempt.ts	script	One streamed AI SDK request
benchmark/measurement.ts	script	Pure observation-to-row conversion
benchmark/endpoints.ts	script	Live Gateway provider discovery
benchmark/strategies.ts	script	US_PROVIDERS list + three gateway:us-* strategies (only + ZDR + no-train)
benchmark/lease.ts	script	Renewable execution lease
benchmark/execute.ts	script	Benchmark orchestration
benchmark/scheduled.ts	interval	Cron entrypoint — runs every 10 min
db/schema.ts	script	SQLite schema + sqlite export
db/queries.ts	script	Run/measurement writes and lookups
db/results.ts	script	Aggregate provider comparison query
db/measurements.ts	script	Paginated measurements listing
frontend/root.tsx	script	HTML shell + title/meta
frontend/index.tsx	script	Client entrypoint, page switch
frontend/components/App.tsx	script	Model header + provider table view
frontend/components/ProviderTable.tsx	script	Sortable provider table (used twice: strategies + providers)
frontend/components/StrategyCells.tsx	script	No store / No train ✓ cells, routing summary under strategy names
frontend/components/MeasurementsPage.tsx	script	Measurements page
frontend/components/Sparkline.tsx	script	History sparklines
frontend/components/InfoPopover.tsx	script	Column explainers
frontend/{providers,history,format}.ts	script	Provider names, hourly series, formatters
frontend/favicon.svg	file	Favicon (still the GLM-era glyph)
shared/types.ts	script	Shared benchmark/result types
README.md	file	Overview + schedule docs

VERCEL_AI_GATEWAY_API_KEY (scoped env var). HTTP app is read-only.