Track feature ideas and implementation status here. Mark items [x] when implemented.

---

Forward any email to the bot's Val Town email address. Claude reads the email, takes action (add to calendar, create task, store info), posts a summary to the Telegram group, and replies to the sender confirming what was done. The forwarder can add instructions at the top of the email body (e.g. "add this to calendar" or "just FYI") — these are parsed out and followed.

Files: email.ts (Val Town email type)

---

Tasks now track an owner (jeff/marisa/both) and an optional reminder time. A new 15-minute cron (reminder-cron.ts) fires a proactive Telegram ping — with a real @mention — when a task's reminder time arrives. Owner→Telegram-ID mapping is auto-learned from message.from on every inbound message (no config required). Timed tasks ping at the exact specified time; date-only tasks ping at 8am ET. The update_task tool lets you assign or reschedule after creation.

This delivers the owner-tracking half of the Responsibility Model backlog item.

---

When the bot completes a simple, unambiguous action (adding a task, setting a reminder, adding a calendar event, storing info), it now acknowledges with an emoji reaction on the triggering message instead of writing a full sentence. Questions, lookups, and anything that needs a conversational response still get text replies.

How it works: A react tool in tools.ts calls setMessageReaction (Telegram Bot API 7.0+), attaching the emoji directly to the user's message. If Telegram rejects the reaction (group settings, old message), it falls back to sending the emoji as a plain message. The LLM decides which mode fits based on guidance in the system prompt.

Files: tools.ts (react tool + ToolCtx type), main.ts (setMessageReaction helper, updated processMessage, updated system prompt)

---

Today the bot only replies in the group when @mentioned (unless RESPOND_TO_ALL=true, which is noisy). @mention is reliable but cumbersome. Add a cheap, fast per-message LLM "gate" that decides whether the bot should respond when it wasn't tagged — because the message is directed at it, is a reply to one of its messages, continues a conversation with it, or is something it can clearly help with. Posture is conservative: only chime in when fairly clearly warranted, to avoid butting into ordinary chatter. A third mode between "@mention only" and "respond to all."

Approach — two-stage gate for non-mention group messages:

1. Free deterministic short-circuit: a direct reply to a bot message (message.reply_to_message.from.is_bot) always gets a response — zero LLM cost.
2. Cheap LLM classifier: returns {respond, confidence, reason} via a forced tool call, gated by a confidence threshold (default 0.8 = conservative). Fail-safe: any error/parse failure ⇒ stay silent (matches today's default). Provider-flexible — reuses the existing isGptModel() branch so it works on a gpt-* or claude-* gate model.

Implementation steps:

1. llm.ts — add config exports:
   • SMART_RESPONSES = (Deno.env.get("SMART_RESPONSES") ?? "true") === "true"
   • GATE_MODEL = Deno.env.get("GATE_MODEL") ?? "claude-haiku-4-5" (cheapest; can point at any gpt-/claude- id)
   • GATE_CONFIDENCE_THRESHOLD = Number(Deno.env.get("GATE_CONFIDENCE_THRESHOLD") ?? "0.8")
2. llm.ts — add classifyShouldRespond({ text, history, botUsername }) => { respond, confidence, reason }:
   • Conservative system prompt describing capabilities (tasks/calendar/info/weather) + criteria; return respond: true only when clearly directed at the bot, clearly continuing a thread with it, or an unambiguous actionable request; quiet on chatter / messages aimed at the other person / ambiguity.
   • Structured output via a forced single tool decide (schema {respond: boolean, confidence: number, reason: string}):
     • Anthropic branch: tool_choice: {type: "tool", name: "decide"}, max_tokens: 128, parse tool_use.input.
     • GPT branch: mirror the OPENAI_TOOLS_RESPONSES shape, forced tool_choice, max_output_tokens: 128, reasoning: {effort: "low"}, parse function-call arguments.
   • Fail-safe default {respond: false, confidence: 0}; console.log the decision + reason.
   • Pass last ~8 messages; mark image-only messages as [photo] so the gate stays text-only/cheap.
3. main.ts — replace the gate at lines 784-788: if group && not RESPOND_TO_ALL && not @mentioned → respond free if replyToBot; else if SMART_RESPONSES run classifyShouldRespond and only proceed when respond && confidence >= GATE_CONFIDENCE_THRESHOLD; else legacy return new Response("OK"). Import the new symbols. Downstream (typing, photo download, processMessage) unchanged — the full model call only happens once a message clears the gate, same as @mentions today.
4. New env vars (all optional): SMART_RESPONSES (default true), GATE_MODEL (default claude-haiku-4-5), GATE_CONFIDENCE_THRESHOLD (default 0.8). Add to CLAUDE.md env table.
5. Changelog: minor release within the Blue era → Cerulean (v2.2). Update changelog.md + CHANGELOG_LATEST in main.ts. No Telegram announcement (minor).

Cost estimate: per check ≈ ~800 input + ~40 output tokens (model-independent — multiply by your gate model's rate). At the default Haiku 4.5 ($1 in / $5 out per MTok) ≈ $0.001/check:

Volume (non-mention group msgs)	Checks/mo	Cost/mo
Light (~20/day)	~600	~$0.60
Moderate (~50/day)	~1,500	~$1.50
Heavy (~150/day)	~4,500	~$4.50

Direct replies to the bot cost $0 (deterministic path). The gate is strictly new cost — it replaces silent drops with a ~$0.001 check; only passing messages hit the main model, same as @mentions today (~10–50× cheaper than a full response turn). Prompt caching won't help (stable prefix ~370 tokens is below the min cacheable prefix; volume is low). Note: MODEL is currently a GPT model whose current per-token rate is not quoted here — apply your OpenAI rate to the ~800-in/~40-out figures if keeping the gate on a gpt-* model.

Verification:

1. Deploy with SMART_RESPONSES=true; confirm the gate model's API key is set.
2. In group (not @mentioned): obvious request ("someone add milk to the list") → responds; direct reply to a bot message → responds (logs show no gate call); ordinary chatter ("running 5 min late") → silent.
3. Check Val Town logs for gate: respond=… conf=… — … to confirm/tune the threshold.
4. Confirm @mention and RESPOND_TO_ALL=true still work unchanged.

---

Store freeform text under a topic (e.g. "store these ideas for our 2027 ski trip"), distinct from the key/value fm_info table.

Implementation steps:

1. Add fm_notes table: id, topic, content, created_by, created_at
2. Add migration to initDb() for the new table
3. Add addNote(topic, content, createdBy) and searchNotes(query) db functions
4. Add two tools to TOOLS array in main.ts:
   • add_note(topic, content) — store freeform notes under a topic label
   • search_notes(query) — full-text search across topic and content
5. Add cases for both in the executeTool switch

---

Natural-language weather queries: "what's the weather in Boston this weekend?"

Implementation steps:

1. Add get_weather(location, days_ahead?) tool to TOOLS array
2. Implement getWeather(location, daysAhead) using Open-Meteo (no API key required):
   • Geocode location name: https://geocoding-api.open-meteo.com/v1/search
   • Fetch forecast: https://api.open-meteo.com/v1/forecast
3. Format response as a readable summary (high/low, condition, precipitation)
4. Add case in executeTool switch
5. No new env vars required

---

Update an upcoming event's title, start/end time, or description via natural language — "move the dentist appointment to 3pm" or "rename Mara's recital to Spring Concert".

Files: main.ts — findCalendarEventByQuery, updateCalendarEvent, update_calendar_event tool

---

Send a photo to the bot and Claude will describe it, extract text, read a receipt, parse a whiteboard, etc.

Implementation steps:

1. In the main handler (main.ts around line 468), detect message.photo in addition to message.text
2. Add downloadTelegramPhoto(fileId) function: calls getFile API then fetches the image bytes
3. Convert image to base64 and pass to Claude as an image content block in the user message
4. Claude responds naturally — no explicit tool needed; can still call existing tools (e.g. add_task items parsed from a photo of a handwritten list)
5. Handle caption text alongside the image if present (message.caption)

---

Lightweight regression testing for the LLM's tool-routing and response quality.

Approach: two layers

Layer 1 — Trace logging (production data, ~30min) Add an fm_evals SQLite table that captures every real LLM interaction: input message, tools called, and final response. Zero test-writing upfront; surfaces failure patterns from real traffic. Add a /evals command or HTTP endpoint to browse the log.

Layer 2 — Golden test val (~1h) Create evals.ts as a script val with ~8–10 fixed test cases that verify tool routing:

const cases = [
  { input: "Jeff: add milk to the grocery list", expect_tools: ["add_task"] },
  { input: "what's on the calendar this week?", expect_tools: ["get_calendar_events"] },
  { input: "remember the wifi password is Hunter2", expect_tools: ["store_info"] },
  { input: "mark task 5 as done", expect_tools: ["complete_task"] },
  { input: "hey how's everyone doing", expect_tools: [] }, // should NOT call tools
];
// For each: call Anthropic with the same system prompt + test input,
// assert tool_use blocks match expected tools, log PASS/FAIL to console + SQLite

Run manually or on a cron; results visible in Val Town logs.

Implementation steps:

1. Add fm_evals table: id, input, tools_called, response, created_at, passed
2. In processMessage, after each completed agentic loop, insert a row into fm_evals
3. Create evals.ts script val importing the same TOOLS and system prompt from main.ts
4. Write 8–10 golden test cases covering each tool + the no-tool case
5. Run each case, compare actual tool_use blocks to expected, write results to fm_evals
6. Optionally add a /evals slash command in main.ts to show recent pass/fail counts

---

Extend existing vision to automatically extract structured events and todos from photos — school flyers, birthday invitations, sports schedules, camp confirmation emails, etc.

The current image parsing describes what's in a photo; this goes further: Claude identifies dates, times, places, and required actions, then proposes adding them to the calendar or task list in a single step. The chat becomes the family inbox — you shouldn't need to decide whether something goes on the calendar or the task list.

Implementation steps:

1. Extend the image system prompt to explicitly ask Claude: "If this image contains dates, events, deadlines, or action items, propose adding them using add_calendar_event, add_task, or add_note"
2. Add optional confirm_before_adding flag in the message handler so Claude can ask "Found: Mara's fall concert Nov 14 at 7pm. Add it?" before writing
3. Handle caption text as additional context ("this is Cass's soccer schedule")
4. Test against: school flyers, party invitations, prescription bottles, forms with due dates

---

Track things that need figuring out but have no date attached — distinct from tasks ("do this") and calendar events ("be there at X"). Examples: "we need to figure out summer camp", "decide on Mara's birthday party venue", "sort out ski rentals for Vermont trip".

Open loops persist until someone explicitly closes them. The bot surfaces them in the Sunday weekly briefing and when contextually relevant (e.g., upcoming trip → check open loops for that trip).

Implementation steps:

1. Add fm_loops table: id, title, context, status (open/closed), created_by, created_at, closed_at
2. Add tools:
   • add_open_loop(title, context?) — create an unresolved item
   • close_loop(id_or_title) — mark resolved
   • list_open_loops() — show all unresolved items
3. Claude should recognize loop-shaped language ("we need to figure out", "still need to decide", "haven't sorted") and proactively offer to add a loop
4. Include open loops in the Sunday briefing output (see Proactive Briefings feature)

---

Two recurring messages that turn the bot from reactive to proactive:

Morning briefing (daily, ~7am): "Today: Mara school 8:15; Cass daycare; Jeff has late meeting until 5:30; rain at 3 — bring raincoats. Mara needs her library book today."

Sunday week-ahead (~8am): Reconciles both calendars + memory + open loops. Doesn't just list events — answers "what does this week require us to do?" Example output: "Mara's birthday party Saturday → gift not bought yet; Jeff out Thursday evening → Marisa solo bedtime; ski weekend coming → Cass rentals unresolved."

Implementation steps:

1. Create cron.ts (or extend existing) with two interval jobs
2. Morning briefing: fetch today's calendar events, check weather, query open tasks with today's due date, surface any prep notes from fm_info/fm_notes
3. Sunday briefing: fetch full week's events, run Claude with the prompt "what does this week require the family to prepare, decide, or do?" — pulling from calendar + tasks + open loops + family memory
4. Send to the family Telegram chat via sendTelegramMessage
5. Make briefing opt-in via /briefing on|off slash command stored in fm_info

---

A richer layer on top of fm_info for the facts family life constantly needs: kids' clothing/shoe sizes, school schedules, babysitter contact info, gift ideas, activities the kids love, dietary restrictions, packing preferences, places you've stayed, things Marisa or Jeff have mentioned wanting.

The goal: natural queries work years later. "What size shoes does Mara wear?" "What was that restaurant in Rotterdam the kids loved?" "What are gift ideas for Cass?"

Implementation steps:

1. Add fm_memory table: id, person, category, key, value, created_at, updated_at
   • person: "mara", "cass", "jeff", "marisa", "family"
   • category: "sizes", "gifts", "activities", "places", "contacts", "preferences"
2. Add tools:
   • remember(person, category, key, value) — store a memory
   • recall(person?, category?, query) — retrieve matching memories
3. Claude should recognize memory-worthy statements ("Cass is obsessed with trains", "Mara said she wants a telescope") and proactively store them
4. Memories feed into trip planning, outing suggestions, and gift tracking
5. Supersedes some ad-hoc fm_info usage — migrate key entries on first run

---

Maintain a "trip object" in memory for each upcoming family trip. Tracks logistics as a checklist: lodging, tickets/passes, rentals, restaurant reservations, packing list. A few days before departure, auto-generates a packing list based on destination + weather forecast + planned activities + family ages.

Implementation steps:

1. Add fm_trips table: id, name, destination, start_date, end_date, status, notes
2. Add fm_trip_items table: id, trip_id, category (lodging/transport/activity/rental/other), description, status (open/done), owner
3. Add tools:
   • create_trip(name, destination, start_date, end_date) — create trip record
   • add_trip_item(trip_name, category, description, owner?) — add a checklist item
   • get_trip_status(trip_name) — show full trip with open/done items
4. 3 days before start_date: auto-send trip status + generated packing list (via cron) using weather forecast for the destination
5. Open loops referencing a trip (e.g., "Cass rentals — Vermont") should link to the trip object

---

Share any URL and the bot reads it — school newsletters, event pages, Google Docs, Google Sheets. Claude fetches the content and proactively extracts dates, tasks, and important info.

Files: tools.ts (fetch_url tool + fetchUrl + stripHtml), main.ts (system prompt, /start, /help updated)

---

Let the bot look things up: local events, restaurant hours, camp registration dates, activity options, nearby attractions. Turns "remember and coordinate" into "figure this out for us."

This is the unlock for: "find a good Italian restaurant near us for Friday", "when does [summer camp] registration open?", "what's open on Columbus Day weekend?", "find something fun to do Saturday with a 5-year-old."

Implementation steps:

1. Add web_search(query) tool using a search API (Brave Search or Tavily — both have free tiers; store key in Val Town env)
2. Claude decides when a query needs live information vs. stored memory
3. Combine with weather and family memory for proactive suggestions: "Saturday is sunny and free — here are 3 ideas based on what the kids like"
4. Add BRAVE_SEARCH_API_KEY or TAVILY_API_KEY to Val Town env vars

---

On Friday afternoons when the weekend has unscheduled time + good weather, the bot proactively sends outing ideas — drawing on family memory (what activities the kids like, places you've been) + weather + calendar gaps.

Also: flag weather-sensitive plans ("rain forecast Saturday — your park plan might need a backup"), first snow ("sledding Sunday?"), or local events (concert, festival) matching the family's interests.

Implementation steps:

1. Add a Friday ~3pm cron job that checks: Saturday/Sunday calendar for free blocks of 2+ hours; weather forecast for the weekend
2. If a free block + decent weather exists, call Claude with: free times, weather, family memory (activities, kids' ages, recent outings), and ask for 2–3 suggestions
3. Send to family chat: "Saturday 10am–2pm looks free and sunny. A few ideas: [...]"
4. Requires Web Search tool for event discovery beyond stored memory
5. Track "recently done" to avoid repeating suggestions (fm_notes topic: "recent outings")

---

Track who owns what for a given event: preparation, transportation, and follow-through. When both parents are busy at the same time, the bot flags the conflict rather than leaving it implicit.

Example: "Mara's birthday party Saturday 2pm — Jeff driving, gift still needed, card still needed." Or: "Jeff is out Thursday evening → Marisa is solo bedtime — confirm?"

Implementation steps:

1. Add owner field to calendar events and tasks (values: "jeff", "marisa", "both", "tbd")
2. Add tool: assign_owner(event_or_task_id, owner) — set responsibility
3. In weekly briefing, for each event/task with owner = tbd or no owner, flag it explicitly
4. When a new event is added that overlaps with an existing commitment for the same owner, surface the conflict: "Jeff already has a meeting 5–6:30pm Thursday — who handles pickup?"
5. Eventually: "I can do pickup Wednesday" → Claude parses speaker identity (from Telegram user) and updates the owner field automatically