cogs-v-clippies-2

We're going to make a game server for a physical board game where LLMs get to make moves, based on a photo of the game board. each piece that is LLM-controlled will move independently, but all at the same time. thus, we need a server that will take in the photo, and send it to a separate LLM for EACH piece. there can be multiple human players, who get to choose their pieces and give advice to each piece.

the server should be basically agnostic to the nature of the game! its role is to manage the parallelization of having all of the LLM-controlled pieces move based on purely the input of:

1. (already, in background) the general game rules
2. (already, in background) instructions given from the players
3. (every turn) a photo of the game board (we’ll need to use a multimodal LLM, obviously!)

The server does not enforce the game rules or events - the human players do. the server just inputs board state and outputs a description of intended behavior for each piece.

1. game manager loads homepage, which has two buttons: 1a) [new game] 1b) gameid [join game]
2. once the game has been created, it gets special urls using short gameids (5 letters, case-insensitive, check on the server to ensure uniqueness). no auth is required. this is for playtesting at a very small event so that’s long enough. 2aa) /game/:gameid/manage/:mgmtid, which allows editing these:
   • game_name
   • game_rules (common (and common knowledge) for all pieces etc)
   • [new piece type] button that lets you specify specific piece types. two properties:
     • piece_name: "red pawn" or "black disc"
     • piece_description: written in natural language, describes properties/abilities/rules/limits/personality 2b) /game/:gameid (join the game. have the user enter an alphanumeric username with no spaces. enforce uniqueness). redirects them to: 2c) /game/:gameid/play/:username (the url that the player will use to play the game) 2d) /game/:gameid/clone (makes a new game with the same rules)
3. on the /play/ page, the player sees: 3a) button: [new piece] 3b) pieces [list]. for each, the player can input instructions 3c) button: [take photo & play turn] (any player can select this; again, rules are not enforced by the server) 3d) moves to make: [output for each piece] (these get refreshed every time a turn is played)

The LLMs don’t receive any context on previous game events each time they play a move. They only know the current state.

You are playing as an autonomous piece in a board game.  The rules of the board game are:
> ${game_rules}

Your piece is ${piece_name} and the description of your piece is:
> ${piece_description}

Attached is a photo of the board state.

Think out loud as much as you want, then output MOVE: [description of all behavior your piece will perform this turn]

This document outlines the key architectural decisions for the LLM-based board game server.

1. Conflict Resolution: Human-Arbitrated

The server is fundamentally game-agnostic and will not enforce rules or resolve conflicts.

• Server Role: The server's sole responsibility is to make parallel calls to the LLMs for each piece and display their intended moves as text output.
• Human Role: The human players are the final arbiters. They are responsible for interpreting the intended moves, identifying any conflicts (e.g., two pieces wanting the same square), and applying the game rules to determine the outcome.

2. Information Flow & State Management

The server will manage game and piece state according to the following principles.

• A) Optional Piece Memory:

  • The server will save the last successfully executed move for each individual piece.
  • A game-level setting, "Include Previous Move Context," will be available on the /manage page and can be toggled mid-game.
  • If this setting is enabled, the server will add the piece's previously saved move into its prompt context for the current turn. If disabled, this information will be omitted.

• B) Turn State Snapshotting:

  • When any player initiates a turn by pressing "[take photo & play turn]," the server will immediately create an immutable "snapshot" of the current game state.
  • This snapshot includes the game_rules, all piece_description configurations, and all currently saved player instructions.
  • All LLM API calls for that specific turn will use only the data from this snapshot, ensuring that any real-time changes made by a manager do not affect a turn already in progress.

• C) Persistent Player Instructions:

  • Instructions provided by players for each piece are persistent. The server will store the latest submitted instruction for a piece and will automatically include it in that piece's prompt for all subsequent turns until a player explicitly clears or changes it.

• D) Database storage:

  • State will be stored in the SQLite database and retrieved using the appropriate syntax and object/row format.

3. Asynchronous Workflow & User Experience

The turn resolution process is designed to be asynchronous and transparent to the players.

• Parallel LLM Calls: Upon turn initiation, the server will make all LLM API calls for every piece simultaneously (in parallel), not sequentially.
• Live Updates via SSE: As each individual LLM call completes and returns a result, the server will immediately push that piece's move and thought process to all connected players' /play/ pages using server-sent events. This allows players to see results trickle in as they become available, rather than waiting for all pieces to finish processing.
• Output Display: The UI will display the full output from each LLM, structured for clarity:
  • Thinking Process: The "think out loud" portion of the LLM's response will be displayed.
  • Final Move: The final MOVE: description will be clearly and prominently displayed, visually distinct from the thinking process (e.g., via a larger font or bolding). This provides crucial feedback to players, helping them understand the LLM's reasoning and refine their instructions for future turns.

4. Townie-specific notes

For Townie, note:

• SQLite queries return an object with a rows array, not a direct array
• WebSockets aren’t available; use Server-Sent Events