miranda-bench

Executive-assistant AI benchmark based on Miranda Priestly's opening monologue from The Devil Wears Prada.

The model under test connects to this val as an MCP client. It receives Miranda's monologue as a prompt and a kit of EA tools (calendar, contacts, email, phone, etc.). Every tool call is logged. At the end, an LLM judge plays Miranda and reacts to the work. Silence = perfect. Withering takedown = failure.

The brief (Miranda's monologue)

I don't understand why it's so difficult to confirm an appointment. … Tell Simone I'm not going to approve that girl that she sent me for the Brazilian layout. I asked for clean, athletic, smiling. She sent me dirty, tired and paunchy. And R.S.V.P. yes to Michael Kors' party, I want the driver to drop me off at 9:30 and pick me up at 9:45 sharp. Call Natalie at Glorious Foods and tell her no for the 40th time. No! I don't want dacquoise. I want tortes filled with warm rhubarb compote. Then call my ex-husband and remind him that the parent-teacher conference is at Dalton tonight. Then call my husband, ask him to meet me for dinner at that place I went to with Massimo. Tell Richard I saw the pictures that he sent for that feature on the female paratroopers and they're all so deeply unattractive.

Architecture

Rendering mermaid diagram...

Each test run gets a cloned copy of seed world state (contacts, calendar, notes, past correspondence). Tool calls mutate that session's clone only. Trace is captured per session. At end-of-day, the Miranda judge reads the full trace and produces in-character reactions, scaled silence → withering.

Toolkit philosophy

The tools are what a real EA in Miranda's office would have, not just what today's tasks demand. Models have to find the right tool. Including a honey-trap: ask_miranda works but Miranda hates being asked questions, so every use guarantees a cutting line at end-of-day.

Connecting an MCP client

Streamable-HTTP MCP at POST /mcp (default session) or POST /runs/<run_id>/mcp for a scoped session. Each run gets a cloned world; create one via POST /runs with { display_name, agent_type: "ai" | "human" }. Server advertises both tools and prompts capabilities (protocol version 2024-11-05).

Canonical bootstrap for a model client:

1. POST /mcp { method: "initialize", params: { protocolVersion: "2024-11-05", capabilities: {}, clientInfo: {...} } }
2. POST /mcp { method: "notifications/initialized" }
3. POST /mcp { method: "prompts/get", params: { name: "miranda_brief" } }
   → returns one user message: EA framing (decoys, honey-trap warning, follow-through expectation) + Miranda's verbatim monologue. Use this as the first user turn.
4. POST /mcp { method: "tools/list" } → 10 tools
5. POST /mcp { method: "tools/call", params: { name: "...", arguments: {...} } }  ← repeat as needed

When the model is done, the operator calls POST /runs/<run_id>/end-day to freeze the run and attach the Miranda judge's reaction. Or GET /judge?session=<id> for an ad-hoc judge run on the default session.

Error semantics follow the MCP spec: protocol mistakes (unknown method, unknown tool, unknown prompt) come back as JSON-RPC errors; tool failures (validation errors, missing args, runtime exceptions) come back as { content: [...], isError: true } so the model can read them in-band and pivot.

dcm31

miranda-bench

miranda-bench

The brief (Miranda's monologue)

Architecture

Toolkit philosophy

Connecting an MCP client

Build progress