A public, read-only Val Town page that compares providers serving zai/glm-5.3-flash through Vercel AI Gateway.

Every provider attempt is one row in the val-scoped benchmark_measurements SQLite table. The default provider comparison pools measurements from every completed run for the model. Each run keeps its benchmark version in SQLite so methodology changes remain auditable without removing older completed measurements from the aggregate.

Before each run, the worker fetches Vercel's live endpoint catalog. It benchmarks every returned provider and stores a normalized provider snapshot. Providers newly added by Vercel are included automatically. An interrupted run resumes against the exact provider set it started with.

Each provider receives the same fixed multi-turn message array four times. Attempts for one provider run sequentially to preserve the cache warmup signal, while different providers run concurrently. Gateway automatic caching is enabled and each request is restricted to the provider being measured.

The sortable provider table reports:

• Reliability across all attempts.
• TTFT p50 and p95 across successful responses.
• p50 and p5 observed output tokens per second after first text.
• Cache hits among eligible repeated successful responses.
• Observed cost per million input plus output tokens.

Tokens/sec is output tokens divided by duration - TTFT. It measures how quickly text reaches this client after the first text delta. Providers that buffer output into a few large chunks can therefore appear unusually fast. p50 is the median, TTFT p95 is the slow latency tail, and tokens/sec p5 is the slow throughput tail.

$/MTok is total observed cost divided by total input plus output tokens, multiplied by one million. Cached tokens remain part of input tokens, while their discount is reflected in observed cost.

The measurements page exposes Success and Error. Success is based on the complete consumed text stream, independent of the bounded private response preview. Error rows show a short, redacted message; timeouts include the configured 75 second limit. Raw response text and provider metadata remain private.

benchmark/scheduled.ts runs every 15 minutes in UTC. Requests have a 75 second timeout and automatic SDK retries are disabled.

A renewable SQLite lease prevents scheduled and manual starts from overlapping. The lease can be reclaimed after five minutes without a renewal, so an interrupted invocation does not block several later intervals. Individual attempts are checkpointed, and a later invocation resumes the same benchmark version without duplicating stored measurements.

To run manually, edit benchmark/config.ts if needed, open benchmark/scheduled.ts in Val Town, and click Run. Public visitors cannot start model requests from the HTTP app.

The Gateway credential is stored as the encrypted VERCEL_AI_GATEWAY_API_KEY environment variable. The HTTP app exposes only read-only routes.

• benchmark/prompt.ts — fixed messages and benchmark version.
• benchmark/attempt.ts — one streamed AI SDK request.
• benchmark/measurement.ts — pure observation-to-row conversion.
• benchmark/endpoints.ts — live Gateway provider discovery.
• benchmark/lease.ts — renewable execution lease.
• benchmark/execute.ts — benchmark orchestration.
• db/ — val-scoped SQLite schema, writes, provider snapshots, and aggregation.
• frontend/ — React interface styled with Twind.
• shared/ — shared benchmark and result types.