We're going to make a game server for a physical board game where LLMs get to
make moves, based on a photo of the game board. each piece that is
LLM-controlled will move independently, but all at the same time. thus, we need
a server that will take in the photo, and send it to a separate LLM for EACH
piece. there can be multiple human players, who get to choose their pieces and
give advice to each piece.
the server should be basically agnostic to the nature of the game! its role is
to manage the parallelization of having all of the LLM-controlled pieces move
based on purely the input of:
- (already, in background) the general game rules
- (already, in background) instructions given from the players
- (every turn) a photo of the game board (we’ll need to use a multimodal LLM,
obviously!)
The server does not enforce the game rules or events - the human players do. the
server just inputs board state and outputs a description of intended behavior
for each piece.
- game manager loads homepage, which has two buttons: 1a) [new game] 1b)
gameid [join game]
- once the game has been created, it gets special urls using short gameids (5
letters, case-insensitive, check on the server to ensure uniqueness). no auth
is required. this is for playtesting at a very small event so that’s long
enough. 2aa) /game/:gameid/manage/:mgmtid, which allows editing these:
- game_name
- game_rules (common (and common knowledge) for all pieces etc)
- [new piece type] button that lets you specify specific piece types. two
properties:
- piece_name: "red pawn" or "black disc"
- piece_description: written in natural language, describes
properties/abilities/rules/limits/personality 2b) /game/:gameid (join the
game. have the user enter an alphanumeric username with no spaces.
enforce uniqueness). redirects them to: 2c) /game/:gameid/play/:username
(the url that the player will use to play the game) 2d)
/game/:gameid/clone (makes a new game with the same rules)
- on the /play/ page, the player sees: 3a) button: [new piece] 3b) pieces
[list]. for each, the player can input instructions 3c) button: [take photo &
play turn] (any player can select this; again, rules are not enforced by the
server) 3d) moves to make: [output for each piece] (these get refreshed every
time a turn is played)
The LLMs don’t receive any context on previous game events each time they play a
move. They only know the current state.
You are playing as an autonomous piece in a board game. The rules of the board game are:
> ${game_rules}
Your piece is ${piece_name} and the description of your piece is:
> ${piece_description}
Attached is a photo of the board state.
Think out loud as much as you want, then output MOVE: [description of all behavior your piece will perform this turn]
This document outlines the key architectural decisions for the LLM-based board
game server.
1. Conflict Resolution: Human-Arbitrated
The server is fundamentally game-agnostic and will not enforce rules or resolve
conflicts.
- Server Role: The server's sole responsibility is to make parallel calls to
the LLMs for each piece and display their intended moves as text output.
- Human Role: The human players are the final arbiters. They are responsible
for interpreting the intended moves, identifying any conflicts (e.g., two
pieces wanting the same square), and applying the game rules to determine the
outcome.
2. Information Flow & State Management
The server will manage game and piece state according to the following
principles.
-
A) Optional Piece Memory:
- The server will save the last successfully executed move for each individual
piece.
- A game-level setting, "Include Previous Move Context," will be available on
the
/manage page and can be toggled mid-game.
- If this setting is enabled, the server will add the piece's previously saved
move into its prompt context for the current turn. If disabled, this
information will be omitted.
-
B) Turn State Snapshotting:
- When any player initiates a turn by pressing "[take photo & play turn]," the
server will immediately create an immutable "snapshot" of the current game
state.
- This snapshot includes the
game_rules, all piece_description
configurations, and all currently saved player instructions.
- All LLM API calls for that specific turn will use only the data from this
snapshot, ensuring that any real-time changes made by a manager do not
affect a turn already in progress.
-
C) Persistent Player Instructions:
- Instructions provided by players for each piece are persistent. The server
will store the latest submitted instruction for a piece and will
automatically include it in that piece's prompt for all subsequent turns
until a player explicitly clears or changes it.
-
D) Database storage:
- State will be stored in the SQLite database and retrieved using the
appropriate syntax and object/row format.
3. Asynchronous Workflow & User Experience
The turn resolution process is designed to be asynchronous and transparent to
the players.
- Parallel LLM Calls: Upon turn initiation, the server will make all LLM API
calls for every piece simultaneously (in parallel), not sequentially.
- Live Updates via SSE: As each individual LLM call completes and returns a
result, the server will immediately push that piece's move and thought process
to all connected players'
/play/ pages using server-sent events. This allows
players to see results trickle in as they become available, rather than
waiting for all pieces to finish processing.
- Output Display: The UI will display the full output from each LLM,
structured for clarity:
- Thinking Process: The "think out loud" portion of the LLM's response
will be displayed.
- Final Move: The final
MOVE: description will be clearly and
prominently displayed, visually distinct from the thinking process (e.g.,
via a larger font or bolding). This provides crucial feedback to players,
helping them understand the LLM's reasoning and refine their instructions
for future turns.
4. Townie-specific notes
For Townie, note:
- SQLite queries return an object with a rows array, not a direct array
- WebSockets aren’t available; use Server-Sent Events