A public, read-only Val Town page that compares providers serving deepseek/deepseek-v4.1-flash through Vercel AI Gateway.

Every provider attempt is one row in the val-scoped benchmark_measurements SQLite table. The default provider comparison pools measurements from every completed run for the model. Each run keeps its benchmark version in SQLite so methodology changes remain auditable without removing older completed measurements from the aggregate.

Before each run, the worker fetches Vercel's live endpoint catalog. It benchmarks every returned provider and stores a normalized provider snapshot. Providers newly added by Vercel are included automatically. An interrupted run resumes against the exact provider set it started with.

Each provider receives the same fixed multi-turn message array four times. Attempts for one provider run sequentially to preserve the cache warmup signal, while different providers run concurrently. Gateway automatic caching is enabled and each request is restricted to the provider being measured.

Alongside the solo providers (only: [provider]), the run also benchmarks three synthetic "strategy" endpoints defined in benchmark/strategies.ts. All of them pass only: US_PROVIDERS and differ only in ordering:

Row	providerOptions.gateway
US ordered	only: US_PROVIDERS, order: US_PROVIDERS — the config to ship
US · sort by TTFT	only: US_PROVIDERS, sort: "ttft"
US · sort by throughput	only: US_PROVIDERS, sort: "tps"

Every strategy also sends zeroDataRetention: true and disallowPromptTraining: true, so the gateway only routes to providers with no-store / no-train agreements. Those flags are shown per provider in the No store / No train columns, read from has_zdr / has_no_training in Vercel's endpoint catalogue.

For strategy rows the table also shows which provider the gateway actually served from (providerMetadata.gateway.routing.finalProvider) and how many successful requests needed a failover (totalProviderAttemptCount > 1). TTFT for a strategy row therefore includes any retry time — the true cost of autorouting.

deepseek-v4.1-flash is a reasoning model and this benchmark measures it with reasoning on — the product default. Since v4 the request explicitly sends providerOptions.deepseek.thinking = enabled for every attempt: probing showed modal and wafer serve the model with thinking off by default at the gateway (46-token answers, no reasoning), while particle and novita default it on (~130 reasoning + ~50 text tokens). Without the explicit flag the benchmark was comparing providers doing different tasks.

Output is capped at 300 tokens to keep the benchmark cheap, so the final question is deliberately pure recall from the conversation: measured on particle/novita/deepseek it costs 80–160 reasoning tokens plus ~50 tokens of visible text, leaving comfortable headroom.

History: static-multiturn-v2 asked an open-ended "most important invariant" question that burned 100–370 reasoning tokens; every no_text error in that version had exactly 300 output tokens (budget spent thinking, no text emitted), which made reasoning-on providers look ~25 points less reliable. v3 switched to the recall question and subtracted usage-reported reasoning tokens from the tok/s numerator. v4 forces thinking on everywhere and, because modal streams reasoning but reports reasoningTokens: 0 (folded into outputTokens), derives a floor estimate from streamed reasoning characters for the tok/s subtraction. v5 pins reasoningEffort: "low" (the gateway's default "medium" is rejected by togetherai) and replaces the tok/s formula — v4's post-TTFT window was ~0.4 ms for providers that deliver all text in one chunk, producing 80k tok/s. Older version rows remain in SQLite but are not pooled.

The sortable provider table reports:

• Reliability across all attempts.
• TTFT p50 and p95 across successful responses.
• p50 and p5 observed output tokens per second after first text.
• Cache hits among eligible repeated successful responses.
• Observed cost per million input plus output tokens.

Tokens/sec is all output tokens (reasoning included) divided by the window from the first streamed reasoning/text delta to the end of the stream. Responses delivered in fewer than 3 chunks or a window under 100 ms were buffered rather than streamed and are excluded (null) instead of reported as absurdly fast. p50 is the median, TTFT p95 is the slow latency tail, and tokens/sec p5 is the slow throughput tail.

$/MTok is total observed cost divided by total input plus output tokens, multiplied by one million. Cached tokens remain part of input tokens, while their discount is reflected in observed cost.

The measurements page exposes Success and Error. Success is based on the complete consumed text stream, independent of the bounded private response preview. Error rows show a short, redacted message; timeouts include the configured 75 second limit. Raw response text and provider metadata remain private.

Since v5 the benchmark only covers US-based providers with ZDR + no-train agreements, in the fallback order we'd ship in an app (see US_PROVIDERS in benchmark/strategies.ts). The three gateway:us-* strategies all restrict routing to that set and differ only in ordering.

Costs roughly $0.017 per run → ~$70/month at a 10-minute cadence.

benchmark/scheduled.ts runs every 10 minutes. Requests have a 75 second timeout and automatic SDK retries are disabled.

A renewable SQLite lease prevents scheduled and manual starts from overlapping. The lease can be reclaimed after five minutes without a renewal, so an interrupted invocation does not block several later intervals. Individual attempts are checkpointed, and a later invocation resumes the same benchmark version without duplicating stored measurements.

To run manually, edit benchmark/config.ts if needed, open benchmark/scheduled.ts in Val Town, and click Run. Public visitors cannot start model requests from the HTTP app.

The Gateway credential is stored as the encrypted VERCEL_AI_GATEWAY_API_KEY environment variable. The HTTP app exposes only read-only routes.

• benchmark/prompt.ts — fixed messages and benchmark version.
• benchmark/attempt.ts — one streamed AI SDK request.
• benchmark/measurement.ts — pure observation-to-row conversion.
• benchmark/endpoints.ts — live Gateway provider discovery.
• benchmark/strategies.ts — synthetic routing-strategy endpoints.
• benchmark/lease.ts — renewable execution lease.
• benchmark/execute.ts — benchmark orchestration.
• db/ — val-scoped SQLite schema, writes, provider snapshots, and aggregation.
• frontend/ — React interface styled with Twind.
• shared/ — shared benchmark and result types.