cl-triage

${esc(heading)}

\n${body}\n`,\n { status, headers: { \"content-type\": \"text/html; charset=utf-8\" } },\n );\n}\n\n/**\n * GET /status with `Authorization: Bearer `.\n * Val Town's logs are not reachable from the API and `vt tail` is unreliable, so this is how\n * the tables get inspected from a terminal. The bearer is verified by asking api.val.town who it\n * belongs to and comparing with the owner named in VALTOWN_ENTRYPOINT; no second secret needed.\n */\nasync function status(req: Request): Promise {\n const json = (body: unknown, code = 200) =>\n new Response(JSON.stringify(body, null, 1), {\n status: code,\n headers: { \"content-type\": \"application/json; charset=utf-8\" },\n });\n const auth = req.headers.get(\"authorization\") ?? \"\";\n const owner = (Deno.env.get(\"VALTOWN_ENTRYPOINT\") ?? \"\").match(/esm\\.town\\/v\\/([^/]+)\\//)?.[1];\n if (!auth.startsWith(\"Bearer \") || !owner) return json({ error: \"unauthorized\" }, 401);\n const me = await fetch(\"https://api.val.town/v1/me\", {\n headers: { authorization: auth },\n signal: AbortSignal.timeout(8000),\n });\n if (!me.ok || (await me.json())?.username !== owner) return json({ error: \"unauthorized\" }, 401);\n\n await db.init();\n const limit = Number(new URL(req.url).searchParams.get(\"limit\") ?? \"20\");\n const t = await db.dumpTables(Math.min(Math.max(limit, 1), 200));\n const posts = t.posts.map((r) => ({\n uuid: r.uuid, search: r.search, title: r.title, price: r.price, status: r.status,\n verdict: r.verdict, score: r.score, reason: r.reason, flags: r.flags, dup_of: r.dup_of,\n attempts: r.attempts, queued_at: r.queued_at, judged_at: r.judged_at, emailed_at: r.emailed_at,\n }));\n return json({ posts, blocked: t.blocked.length, feedback: t.feedback });\n}\n",[],[214],"d2b98150-26bd-4fef-bb6e-2787fff9fef9","deno.json","e8bf8620-ace8-11f1-b2b3-1607ee4eb77e",58,["D",1789030157954],"{\n \"tasks\": {\n \"test\": \"deno test --allow-read=test/fixtures test/\",\n \"check\": \"deno check --allow-import=esm.town:443 lib/ inbox.email.tsx judge.cron.tsx feedback.http.tsx\",\n \"fmt\": \"deno fmt\",\n \"lint\": \"deno lint\",\n \"criteria\": \"deno run --allow-read --allow-env=HOME --allow-net=api.val.town scripts/push-criteria.ts\"\n },\n \"fmt\": {\n \"lineWidth\": 100,\n \"exclude\": [\n \"*.md\",\n \"research/\",\n \"test/fixtures/\"\n ]\n },\n \"compilerOptions\": {\n \"strict\": true\n }\n}\n",[],[222],"c80a41e1-c296-47b5-b399-1191650496cf","README.md","2e4ab7f4-acea-11f1-a894-1607ee4eb77e",53,["D",1789029306692],"# cl-triage\n\nCraigslist saved-search alerts → an LLM judge → one email with only the listings worth looking\nat. Runs on Val Town's free tier.\n\nCraigslist emails a saved-search alert whenever matching posts appear. Those alerts re-fire on\nevery renew, repost and edit, so the same item arrives over and over, and most of what arrives\nis not what you meant. This val parses each alert, fetches the real posting and its photos,\nasks Claude whether it matches criteria you wrote in plain English, and emails you only the\ngood ones — with a **nope** button that suppresses that listing (and its reposts and\ncrossposts) forever and teaches the judge what you don't want.\n\n## How it works\n\n```\ncraigslist alert -> inbox.email.tsx parse, cheap dedup, enqueue, then judge inline\n | SQLite: posts(uuid, pid, search, title, ..., status)\n |\n +--------> lib/drain.ts claim a row -> detail API -> dedup ladder\n | -> images -> one LLM call -> verdict\n | -> one email per drain\n v\n judge.cron.tsx the same drain, for the overflow: the rest of a\n | multi-result alert, backlog, retries\n v\n feedback.http.tsx the nope / more-like-this links, HMAC signed\n```\n\nAn alert is judged the moment it arrives — most alerts are a single result, and waiting for the\nnext cron tick would be latency for nothing. That is safe because the drain measures its\n40-second deadline from the *email handler's* start, so parsing counts against the same budget,\nand it will not begin a post without 38 seconds still in hand. Val Town's free tier kills a run\nat 60 seconds and *repeated* overruns temporarily pause the val, so the rule is that the inline\npass degrades by doing less, never by running longer: whatever it declines to start, the cron\ntakes. Because two drains can overlap, each post is claimed with a conditional UPDATE and only\none of them wins it.\n\nDuplicates are caught by a ladder of keys, strongest first: the numeric post id (stable across\nrenew), `repostOf` (craigslist records repost lineage itself), the sorted set of image ids\n(catches crossposts under different ids and different coordinates), and finally\ntitle + price + lat/lon rounded to three decimals. A duplicate inherits the original's verdict\nand costs no model call.\n\n## Environment variables\n\nAll configuration is in env vars, because a free-tier val is public and its source is\nworld-readable.\n\n| variable | required | what it is |\n|---|---|---|\n| `CRITERIA_JSON` | no (but the point) | `{\"default\": \"\", \"searches\": {\"\": \"\"}}` — what you actually want, per saved search |\n| `ANTHROPIC_API_KEY` | no | Without it the val runs in DRY_RUN: everything except the model call, verdict stored as `dry` |\n| `JUDGE_MODEL` | no | Defaults to `claude-sonnet-5` |\n| `FEEDBACK_SECRET` | yes for the buttons | HMAC key for the nope / more-like-this links. Without it the feedback endpoint answers 503 and the digest ships without buttons |\n| `FEEDBACK_URL` | no | The public URL of `feedback.http.tsx`. Discovered automatically at runtime if unset |\n| `SEND_IN_DRY_RUN` | no | `1` makes DRY_RUN still send the digest, for testing the email path |\n\n## Setup\n\n1. Set `CRITERIA_JSON`, `ANTHROPIC_API_KEY` and `FEEDBACK_SECRET` in the val's environment\n variables (sidebar in the Val Town editor), and set the cron interval on `judge.cron.tsx`\n (15 minutes is the free-tier minimum).\n2. Turn on the alert checkbox for each saved search in your craigslist account.\n3. In Gmail: Settings → Forwarding → add this val's inbound address (shown on\n `inbox.email.tsx` in the Val Town editor). Gmail sends a verification code to it; the\n handler stores that mail in blob `last-other-email`, so read the code with\n `blob.getJSON(\"last-other-email\")`.\n4. Add a Gmail filter `from:alerts@alerts.craigslist.org` → *Forward it to* that address, and\n probably *Skip the inbox* so only judged results land in front of you.\n5. Forward one alert by hand first and check the log line `ALERT {...}`.\n\n## Adding a search\n\nTick the alert box on the new saved search in craigslist. Nothing here needs to change: the\nalert subject is `(N new results) ` and that name is the key. Add an entry\nunder `searches` in `CRITERIA_JSON` when you want it judged on its own terms — until you do, it\nfalls back to `default` and logs `CRITERIA_FALLBACK`, so a new search is never silently dropped.\n\n## Being a good citizen\n\nRoughly one 2 KB JSON call and at most five cached images per alerted post — three orders of\nmagnitude under the 1,000 pages per 24 hours craigslist's own terms name. Browser-like\nUser-Agent, GET only, never `?format=rss`, never `/reply` or anything else robots.txt\ndisallows. If craigslist ever answers with its block page, the run logs the blockID, marks the\npost `blocked_by_cl` and stops. Do not retry it and do not move it to another IP.\n\n## Development\n\n```\ndeno task test # unit tests for the email parser, the dedup ladder and the link signing\ndeno task check # typecheck, including the esm.town imports\nvt push # deploy\n```\n",[],[230],"c918f213-b092-41f8-8345-6ba15e91e6ad",".vtignore","e7eae14a-ace8-11f1-8c88-1607ee4eb77e",57,["D",1789030157889],"# Never sync to Val Town. The val is PUBLIC (free tier = zero private vals), so\n# anything here is world-readable. Keep research notes, fixtures and secrets out.\n.env\n.env.*\n*.local\n\nPLAN.md\nCLAUDE.md\nresearch/\ntest/\nscratch/\n\n.git/\n\n# Lockfile is a local build artifact; Val Town resolves imports itself.\ndeno.lock\ncriteria.json\nscripts/\n",[],[238],"4afb64de-e43d-4e3a-85c7-c5448521df3a",".gitignore","e7f8b5a4-ace8-11f1-81ba-1607ee4eb77e",4,["D",1789025174910],".env\n.env.*\n*.local\n.vt/\nscratch/\n",[],[246],"77b700c6-5186-4dfd-a699-429343c6dbcf","sign.ts","e83bfcf6-ace8-11f1-8265-1607ee4eb77e",11,["D",1789025175351],"/**\n * HMAC-SHA256 over `|`, hex encoded, via Web Crypto (available in Deno and\n * on Val Town). The token is the only authentication the feedback endpoint has: the val is\n * public and the links go out in email, so the secret must never appear in source.\n */\n\nexport type FeedbackKind = \"nope\" | \"yes\";\n\nasync function hmacHex(secret: string, msg: string): Promise {\n const key = await crypto.subtle.importKey(\n \"raw\",\n new TextEncoder().encode(secret),\n { name: \"HMAC\", hash: \"SHA-256\" },\n false,\n [\"sign\"],\n );\n const sig = await crypto.subtle.sign(\"HMAC\", key, new TextEncoder().encode(msg));\n return Array.from(new Uint8Array(sig)).map((b) => b.toString(16).padStart(2, \"0\")).join(\"\");\n}\n\nexport function sign(secret: string, uuid: string, kind: FeedbackKind | string): Promise {\n return hmacHex(secret, `${uuid}|${kind}`);\n}\n\nexport async function verify(\n secret: string,\n uuid: string,\n kind: string,\n token: string | null | undefined,\n): Promise {\n if (!token) return false;\n const expected = await sign(secret, uuid, kind);\n if (token.length !== expected.length) return false;\n // Constant time over the hex string.\n let diff = 0;\n for (let i = 0; i < expected.length; i++) diff |= expected.charCodeAt(i) ^ token.charCodeAt(i);\n return diff === 0;\n}\n",[],[164,254],"d25bbea1-c393-40ff-9c29-4c9e6b5da3c4","parse.ts","e824aae2-ace8-11f1-a6b9-1607ee4eb77e",34,["D",1789026338778],"/**\n * Parsing of Craigslist saved-search alert emails. Pure: no I/O, no Val Town imports,\n * so it is unit-testable against the fixtures in test/fixtures/.\n *\n * Shape of the plain-text part (the authoritative one — it survives Gmail's forward\n * wrapping better than the HTML):\n *\n * =========================================================== \n * <post url>\n * - $10 (santa rosa)\n *\n * The price line is optional (free items have none) and the parenthesised neighbourhood\n * is optional too; a free item may carry only the parenthesised location.\n *\n * Two URL shapes exist. Craigslist switched on 2026-07-02:\n * new: https://www.craigslist.org/view/d/<slug>/<uuid>\n * legacy: https://sfbay.craigslist.org/eby/hsh/d/<slug>/<pid>.html\n * Legacy rows carry a numeric post id and no uuid; the judge resolves the uuid later by\n * following the 301 on the old URL.\n */\n\nexport interface ParsedResult {\n uuid: string | null;\n pid: number | null;\n slug: string;\n title: string;\n price: number | null;\n hood: string | null;\n /** The 11-char image identity from the alert's _300x300 thumbnail, when there is one. */\n thumbImgId: string | null;\n url: string;\n}\n\nexport interface ParsedAlert {\n /** Saved-search name, lowercased — this is the key criteria are stored under. */\n search: string;\n /** N from \"(N new results)\", or null if the subject did not parse. */\n count: number | null;\n results: ParsedResult[];\n}\n\nconst NEW_URL =\n /https?:\\/\\/(?:www\\.)?craigslist\\.org\\/view\\/d\\/([^/\\s\"'<>]+)\\/([A-Za-z0-9]{15,30})/;\nconst LEGACY_URL =\n /https?:\\/\\/[a-z0-9.-]*craigslist\\.org\\/(?:[a-z0-9]+\\/)*d\\/([^/\\s\"'<>]+)\\/(\\d{6,})\\.html/;\n/** \"- $10 (santa rosa)\", \"- $4,500.00 (rockridge)\", \"- $20\". Cents are real: alerts carry them. */\nconst PRICE_LINE = /^-\\s*\\$([\\d,]+(?:\\.\\d+)?)(?:\\s*\$(.+)\$)?\\s*$/;\nconst THUMB = /images\\.craigslist\\.org\\/[0-9A-Za-z]+_([0-9A-Za-z]{8,14})_[0-9A-Za-z]+_\\d+x\\d+\\.jpg/;\n\n/** \"(2 new results) some saved search\" -> {count: 2, search: \"some saved search\"} */\nexport function parseSubject(\n subject: string | undefined,\n): { count: number; search: string } | null {\n if (!subject) return null;\n const m = subject.match(/\$(\\d+)\\s+new\\s+results?\$\\s*(.+)$/i);\n if (!m) return null;\n const search = m[2].trim().toLowerCase();\n if (!search) return null;\n return { count: Number(m[1]), search };\n}\n\n/** Identity key for a result: the uuid when we have one, otherwise \"pid:<n>\". */\nexport function resultKey(r: { uuid: string | null; pid: number | null }): string {\n return r.uuid ?? `pid:${r.pid}`;\n}\n\nexport function parseAlert(\n subject: string | undefined,\n text: string | undefined,\n html: string | undefined,\n): ParsedAlert {\n const sub = parseSubject(subject);\n // The text part is the one to trust — it is simpler and survives forward-wrapping better.\n // The HTML part carries the same links, titles and prices, so it is the fallback for a\n // forward that dropped or mangled the text alternative.\n let results = text ? parseTextPart(text) : [];\n if (!results.length && html) results = parseHtmlPart(html);\n const thumbs = html ? thumbnailsByKey(html) : new Map<string, string>();\n for (const r of results) {\n r.thumbImgId = thumbs.get(resultKey(r)) ?? null;\n }\n return { search: sub?.search ?? \"\", count: sub?.count ?? null, results };\n}\n\nfunction parseTextPart(text: string): ParsedResult[] {\n const out: ParsedResult[] = [];\n const seen = new Set<string>();\n let title: string | null = null;\n let block: string[] = [];\n\n const flush = () => {\n if (title) {\n const r = blockToResult(title, block);\n if (r && !seen.has(resultKey(r))) {\n seen.add(resultKey(r));\n out.push(r);\n }\n }\n title = null;\n block = [];\n };\n\n for (const raw of text.split(/\\r?\\n/)) {\n const sep = raw.match(/^={5,}\\s*(.*)$/);\n if (sep) {\n flush();\n title = sep[1].trim() || null;\n } else if (title) {\n block.push(raw.trim());\n }\n }\n flush();\n return out;\n}\n\nfunction blockToResult(title: string, lines: string[]): ParsedResult | null {\n let uuid: string | null = null;\n let pid: number | null = null;\n let slug = \"\";\n let url = \"\";\n\n for (const line of lines) {\n const nu = line.match(NEW_URL);\n if (nu) {\n slug = nu[1];\n uuid = nu[2];\n url = nu[0];\n break;\n }\n const lu = line.match(LEGACY_URL);\n if (lu) {\n slug = lu[1];\n pid = Number(lu[2]);\n url = lu[0];\n break;\n }\n }\n if (!url) return null;\n\n let price: number | null = null;\n let hood: string | null = null;\n for (const line of lines) {\n const p = line.match(PRICE_LINE);\n if (p) {\n price = Number(p[1].replace(/,/g, \"\"));\n hood = p[2]?.trim() || null;\n break;\n }\n const h = line.match(/^\$(.+)\$\\s*$/);\n if (h) {\n hood = h[1].trim();\n break;\n }\n }\n\n return { uuid, pid, slug, title, price, hood, thumbImgId: null, url };\n}\n\n/**\n * Thumbnail image ids from the HTML part, matched to results by document order: each\n * <img> belongs to the most recent post link above it, which is how the alert is laid out\n * (the thumbnail is wrapped in a second anchor to the same post).\n */\nfunction thumbnailsByKey(html: string): Map<string, string> {\n const combined = new RegExp(\n `(?:${NEW_URL.source})|(?:${LEGACY_URL.source})|(?:${THUMB.source})`,\n \"g\",\n );\n const map = new Map<string, string>();\n let current: string | null = null;\n for (const m of html.matchAll(combined)) {\n // groups: 1,2 = new slug/uuid 3,4 = legacy slug/pid 5 = thumbnail image id\n if (m[2]) current = m[2];\n else if (m[4]) current = `pid:${Number(m[4])}`;\n else if (m[5] && current && !map.has(current)) map.set(current, m[5]);\n }\n return map;\n}\n\n/**\n * Fallback parse of the HTML part. One result per post anchor:\n * <a href=\"<post url>\">title</a> - $10 <small>(santa rosa)</small>\n * The thumbnail below it is a second anchor to the same post whose content is an <img>, so it\n * yields no title and is dropped by the first-wins dedup.\n */\nfunction parseHtmlPart(html: string): ParsedResult[] {\n const anchor = /<a\\s[^>]*href=\"([^\"]+)\"[^>]*>([\\s\\S]*?)<\\/a>/gi;\n const matches = [...html.matchAll(anchor)];\n const out: ParsedResult[] = [];\n const seen = new Set<string>();\n\n for (let i = 0; i < matches.length; i++) {\n const m = matches[i];\n const href = m[1];\n const title = stripTags(m[2]);\n if (!title) continue;\n\n let uuid: string | null = null;\n let pid: number | null = null;\n let slug = \"\";\n const nu = href.match(NEW_URL);\n const lu = nu ? null : href.match(LEGACY_URL);\n if (nu) {\n slug = nu[1];\n uuid = nu[2];\n } else if (lu) {\n slug = lu[1];\n pid = Number(lu[2]);\n } else {\n continue;\n }\n\n // Price and neighbourhood sit between this anchor and the next one.\n const from = m.index! + m[0].length;\n const to = i + 1 < matches.length ? matches[i + 1].index! : html.length;\n const between = html.slice(from, to);\n const pm = between.match(/-\\s*\\$([\\d,]+(?:\\.\\d+)?)/);\n const hm = between.match(/<small>\\s*\$([^)]*)\$\\s*<\\/small>/i) ??\n between.match(/\$([^)<]+)\$/);\n\n const r: ParsedResult = {\n uuid,\n pid,\n slug,\n title,\n price: pm ? Number(pm[1].replace(/,/g, \"\")) : null,\n hood: hm ? hm[1].replace(/\\s+/g, \" \").trim() || null : null,\n thumbImgId: null,\n url: href,\n };\n if (seen.has(resultKey(r))) continue;\n seen.add(resultKey(r));\n out.push(r);\n }\n return out;\n}\n\nfunction stripTags(s: string): string {\n return s.replace(/<[^>]*>/g, \"\").replace(/&/g, \"&\").replace(/\\s+/g, \" \").trim();\n}\n",[],[164,262],"826c2430-3a31-4d09-af53-d9cc981dbcb4","mail.ts","e8045df0-ace8-11f1-8828-1607ee4eb77e",["D",1789025174997],"/**\n * The outgoing digest. One email per cron run, carrying every 'good' verdict from that run.\n *\n * Gmail rules (learned the hard way in the sibling calendar-val project):\n * - <style> is honoured only inside <head>; a <style> in the body is silently dropped.\n * This email therefore uses inline styles only and has no <style> block at all.\n * - ::before / ::after are stripped, so nothing decorative may live in CSS.\n * - messages over ~102 KB get clipped. At <=4 posts per run this is a couple of KB.\n * Free tier can only send to the account owner, so `to` is omitted entirely — passing an\n * explicit address is refused by std/email.\n */\nimport { email } from \"https://esm.town/v/std/email\";\n\nexport interface DigestItem {\n uuid: string;\n title: string;\n priceString: string | null;\n hood: string | null;\n score: number;\n reason: string;\n flags: string[];\n url: string;\n imageUrl: string | null;\n nopeUrl: string | null;\n yesUrl: string | null;\n}\n\nconst esc = (s: string) =>\n s.replace(/&/g, \"&\").replace(/</g, \"<\").replace(/>/g, \">\").replace(/\"/g, \""\");\n\nexport function feedbackLink(base: string, uuid: string, kind: string, token: string): string {\n const u = new URL(base);\n u.searchParams.set(\"u\", uuid);\n u.searchParams.set(\"v\", kind);\n u.searchParams.set(\"t\", token);\n return u.toString();\n}\n\nexport function buildDigest(items: DigestItem[]): { subject: string; html: string; text: string } {\n const subject = items.length === 1\n ? `craigslist: ${items[0].title}`\n : `craigslist: ${items.length} worth a look`;\n\n const blocks = items.map((it) => {\n const meta = [it.priceString, it.hood, `score ${it.score}/10`].filter(Boolean).join(\" · \");\n const img = it.imageUrl\n ? `<a href=\"${esc(it.url)}\"><img src=\"${esc(it.imageUrl)}\" alt=\"\" width=\"300\" ` +\n `style=\"max-width:300px;border:0;border-radius:4px;display:block;margin:8px 0\"></a>`\n : \"\";\n const flags = it.flags.length\n ? `<div style=\"color:#888;font-size:12px\">${esc(it.flags.join(\" · \"))}</div>`\n : \"\";\n const buttons = (it.nopeUrl && it.yesUrl)\n ? `<div style=\"margin-top:10px;font-size:13px\">` +\n `<a href=\"${esc(it.nopeUrl)}\" style=\"padding:5px 10px;border:1px solid #c33;` +\n `border-radius:4px;color:#c33;text-decoration:none\">nope</a>` +\n `<a href=\"${\n esc(it.yesUrl)\n }\" style=\"margin-left:8px;padding:5px 10px;border:1px solid #383;` +\n `border-radius:4px;color:#383;text-decoration:none\">more like this</a></div>`\n : `<div style=\"margin-top:10px;font-size:12px;color:#888\">` +\n `(feedback links unavailable — FEEDBACK_SECRET or FEEDBACK_URL not set)</div>`;\n return `<div style=\"margin:0 0 26px 0;padding:0 0 20px 0;border-bottom:1px solid #eee\">\n <div style=\"font-size:16px;font-weight:600\"><a href=\"${\n esc(it.url)\n }\" style=\"color:#0a58ca;text-decoration:none\">${esc(it.title)}</a></div>\n <div style=\"color:#555;font-size:13px;margin-top:2px\">${esc(meta)}</div>\n <div style=\"margin-top:6px\">${esc(it.reason)}</div>\n ${flags}\n ${img}\n ${buttons}\n</div>`;\n });\n\n const html = `<html><head><meta charset=\"utf-8\"></head>\n<body style=\"margin:0;padding:16px;font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;color:#222\">\n${blocks.join(\"\\n\")}\n<div style=\"color:#999;font-size:11px\">cl-triage</div>\n</body></html>`;\n\n const text = items.map((it) =>\n [\n it.title,\n [it.priceString, it.hood, `score ${it.score}/10`].filter(Boolean).join(\" · \"),\n it.reason,\n it.url,\n it.nopeUrl ? `nope: ${it.nopeUrl}` : \"\",\n it.yesUrl ? `more like this: ${it.yesUrl}` : \"\",\n ].filter(Boolean).join(\"\\n\")\n ).join(\"\\n\\n---\\n\\n\");\n\n return { subject, html, text };\n}\n\n/** Sends to the account owner. `to` must stay omitted: free tier refuses any other recipient. */\nexport async function sendDigest(items: DigestItem[]): Promise<void> {\n const { subject, html, text } = buildDigest(items);\n await email({ subject, html, text });\n}\n",[],[164,270],"4cbed68b-2609-44f0-9479-60babc41c946","llm.ts","e84fd7c6-ace8-11f1-8c92-1607ee4eb77e",33,["D",1789026338611],"/**\n * The judge call. One request per post: system prompt = the criteria for that saved search\n * plus the feedback he has given on that search; user content = the listing plus up to five\n * images as base64 vision blocks.\n *\n * Structured output uses output_config.format with a raw JSON schema (no zod dependency to\n * ship to Val Town). The schema language has no numeric or string constraints, so score is a\n * plain integer and is clamped here.\n */\nimport Anthropic from \"npm:@anthropic-ai/sdk@0.122.0\";\n\n/**\n * A past decision of his, used as a few-shot example. Declared here rather than imported from\n * db.ts so that this module (and its tests) never pull in the Val Town SQLite client;\n * db.feedbackExamples() returns exactly this shape.\n */\nexport interface Example {\n title: string;\n price: number | null;\n reason: string | null;\n}\n\nexport const DEFAULT_MODEL = \"claude-sonnet-5\";\n\nexport interface Criteria {\n default: string;\n searches: Record<string, string>;\n}\n\n/**\n * A deliberately generic stand-in used when CRITERIA_JSON is unset. Nothing personal may be\n * hardcoded here: this val is public.\n */\nexport const BUILTIN_DEFAULT_CRITERIA = [\n \"You are triaging Craigslist for-sale listings for the person who saved this search.\",\n \"No criteria have been configured for this search, so judge on general signals only:\",\n \"does the listing match its own title, is the item complete and usable, are the photos\",\n \"real and informative, is the asking price plausible, and does anything read like a scam,\",\n \"a bulk dealer relisting, or a parts-only item sold as working?\",\n 'Be skeptical. When you cannot tell, the answer is \"meh\".',\n].join(\" \");\n\nexport function loadCriteria(raw: string | undefined): Criteria {\n if (!raw) return { default: BUILTIN_DEFAULT_CRITERIA, searches: {} };\n try {\n const j = JSON.parse(raw);\n return {\n default: typeof j.default === \"string\" && j.default ? j.default : BUILTIN_DEFAULT_CRITERIA,\n searches: (j.searches && typeof j.searches === \"object\") ? j.searches : {},\n };\n } catch (e) {\n console.error(\"CRITERIA_JSON is not valid JSON, using the built-in default:\", String(e));\n return { default: BUILTIN_DEFAULT_CRITERIA, searches: {} };\n }\n}\n\n/** Criteria are keyed by the lowercased saved-search name; unknown names fall back and log. */\nexport function criteriaFor(c: Criteria, search: string): { prompt: string; fellBack: boolean } {\n const hit = c.searches[search.trim().toLowerCase()];\n if (hit) return { prompt: hit, fellBack: false };\n return { prompt: c.default, fellBack: true };\n}\n\nexport interface Verdict {\n verdict: \"good\" | \"meh\" | \"bad\";\n score: number;\n reason: string;\n flags: string[];\n}\n\nconst SCHEMA = {\n type: \"object\",\n properties: {\n verdict: { type: \"string\", enum: [\"good\", \"meh\", \"bad\"] },\n score: { type: \"integer\", description: \"0-10, where 10 is a listing he should go look at now\" },\n reason: { type: \"string\", description: \"One line, at most 200 characters.\" },\n flags: {\n type: \"array\",\n items: { type: \"string\" },\n description: \"Short tags for anything notable: scam, parts-only, dealer, far, overpriced.\",\n },\n },\n required: [\"verdict\", \"score\", \"reason\", \"flags\"],\n additionalProperties: false,\n} as const;\n\nexport function buildSystem(criteria: string, nopes: Example[], yeses: Example[]): string {\n const parts = [criteria.trim()];\n const fmt = (e: Example) =>\n `- ${e.title}${e.price === null ? \"\" : ` ($${e.price})`}${e.reason ? ` — ${e.reason}` : \"\"}`;\n if (yeses.length) {\n parts.push(\n \"Listings he asked for more like these:\\n\" + yeses.map(fmt).join(\"\\n\"),\n );\n }\n if (nopes.length) {\n parts.push(\n \"Listings he explicitly rejected. Treat close matches as bad:\\n\" + nopes.map(fmt).join(\"\\n\"),\n );\n }\n parts.push(\n \"Reply with the JSON object only. reason is one line, at most 200 characters, written to \" +\n \"him, saying the single thing that decides it.\",\n );\n return parts.join(\"\\n\\n\");\n}\n\nexport interface PostForJudge {\n title: string;\n priceString: string | null;\n hood: string | null;\n body: string;\n attributes: string[];\n category: string | null;\n postedIso: string | null;\n createdIso: string | null;\n search: string;\n}\n\nexport function buildUserText(p: PostForJudge): string {\n const lines = [\n `saved search: ${p.search}`,\n `title: ${p.title}`,\n `price: ${p.priceString ?? \"(none listed)\"}`,\n `location: ${p.hood ?? \"(not given)\"}`,\n `category: ${p.category ?? \"(unknown)\"}`,\n ];\n if (p.attributes.length) lines.push(`attributes: ${p.attributes.join(\"; \")}`);\n if (p.createdIso) lines.push(`first posted: ${p.createdIso}`);\n if (p.postedIso) lines.push(`last renewed: ${p.postedIso}`);\n lines.push(\"\", \"body:\", p.body || \"(empty)\");\n return lines.join(\"\\n\");\n}\n\nexport interface JudgeInput {\n apiKey: string;\n model: string;\n system: string;\n userText: string;\n images: { mediaType: string; data: string }[];\n timeoutMs: number;\n}\n\nfunction clampVerdict(o: Record<string, unknown>): Verdict {\n const v = o.verdict === \"good\" || o.verdict === \"meh\"\n ? o.verdict\n : o.verdict === \"bad\"\n ? \"bad\"\n : \"meh\";\n const n = Number(o.score);\n return {\n verdict: v as Verdict[\"verdict\"],\n score: Number.isFinite(n) ? Math.max(0, Math.min(10, Math.round(n))) : 0,\n reason: String(o.reason ?? \"\").slice(0, 200),\n flags: Array.isArray(o.flags) ? o.flags.map(String).slice(0, 10) : [],\n };\n}\n\nexport async function judge(input: JudgeInput): Promise<Verdict> {\n // maxRetries 0 on purpose: a retry costs another full timeout, and two of those overrun the\n // 60 s cap that takes the val out of service. A transient failure goes back on the queue.\n const client = new Anthropic({ apiKey: input.apiKey, timeout: input.timeoutMs, maxRetries: 0 });\n const content: Anthropic.ContentBlockParam[] = input.images.map((img) => ({\n type: \"image\" as const,\n source: { type: \"base64\" as const, media_type: img.mediaType as \"image/jpeg\", data: img.data },\n }));\n content.push({ type: \"text\", text: input.userText });\n\n const res = await client.messages.create({\n model: input.model,\n max_tokens: 2000,\n system: input.system,\n output_config: { effort: \"low\", format: { type: \"json_schema\", schema: SCHEMA } },\n messages: [{ role: \"user\", content }],\n });\n\n if (res.stop_reason === \"refusal\") throw new Error(\"judge refused this listing\");\n const text = res.content.find((b) => b.type === \"text\");\n if (!text || text.type !== \"text\") throw new Error(`no text block (stop=${res.stop_reason})`);\n return clampVerdict(JSON.parse(text.text));\n}\n",[],[164,277],"64a5cc6f-8e19-4d84-9e4d-b3d9e5beb367","drain.ts","e28f72de-acf1-11f1-ac1c-1607ee4eb77e",["D",1789030959109],"/**\n * The drain loop: take queued posts, judge them, send one digest.\n *\n * Called from two places and it must behave identically in both: `inbox.email.tsx` runs it\n * inline the moment an alert lands (his alerts are usually one result, and waiting up to\n * fifteen minutes for a cron to notice is latency for nothing), and `judge.cron.tsx` runs it\n * every interval to pick up the overflow from a burst, the first-activation backlog, and\n * anything that failed and went back on the queue.\n *\n * Budget: Val Town's free tier kills a run at 60 s, and *repeated* overruns temporarily pause\n * the val, so every path here has to fit inside a self-imposed 40 s.\n *\n * A post is started only with >= 25 s of the 40 s left. Worst case from that start:\n * legacy uuid resolution 8 s (only for pid-only rows)\n * + sapi detail 8 s\n * + images none is started with < 10 s left, 5 s timeout each\n * + the model call timeout is left() - 4 s, floor 5 s\n * + the digest email ~1 s\n * = ends about 1-6 s past the 40 s self-deadline in the very worst case, against a 60 s\n * hard cap. The SDK gets maxRetries 0: a retry doubles the timeout, which is what would blow\n * the cap. A real judgement measures ~6 s, so a drain normally judges two or three posts and\n * an inline drain after a 20-result alert (which costs ~2-3 s of SQLite round-trips to\n * enqueue) still judges something instead of tripping the start guard immediately.\n *\n * Per post: one sapi detail call, the dedup ladder, then (only if it is genuinely new) up to\n * five images and exactly one LLM call. Good verdicts from one run go out as a single email.\n */\nimport {\n BlockedError,\n bodyText,\n fetchDetail,\n fetchImageBase64,\n imageUrl,\n resolveUuid,\n} from \"./cl.ts\";\nimport * as db from \"./db.ts\";\nimport { type DedupRow, findDuplicate, imageIdentity } from \"./dedup.ts\";\nimport {\n buildSystem,\n buildUserText,\n criteriaFor,\n DEFAULT_MODEL,\n judge,\n loadCriteria,\n type Verdict,\n} from \"./llm.ts\";\nimport { type DigestItem, feedbackLink, sendDigest } from \"./mail.ts\";\nimport { sign } from \"./sign.ts\";\n\nconst RUN_DEADLINE_MS = 40_000;\n/** A post is only started with almost the whole budget in hand — see the worst case above. */\nconst START_MIN_LEFT_MS = 25_000;\nconst RESOLVE_TIMEOUT_MS = 8_000;\nconst DETAIL_TIMEOUT_MS = 8_000;\nconst IMAGE_TIMEOUT_MS = 5_000;\n/** Never begin another image fetch this close to the deadline. */\nconst IMAGE_MIN_LEFT_MS = 10_000;\n/** Wall clock kept back from the model call for storing the verdict and sending the digest. */\nconst JUDGE_RESERVE_MS = 4_000;\nconst MAX_ATTEMPTS = 3;\n/** A claim older than this is assumed to belong to an isolate that died. */\nconst CLAIM_TTL_MS = 120_000;\nconst MAX_IMAGES = 5;\nconst NOPE_EXAMPLES = 10;\nconst YES_EXAMPLES = 5;\n\nexport interface DrainOpts {\n /**\n * When the *caller* started, not when drain was called. The email handler spends time\n * parsing and enqueueing first, and that time comes out of the same 40 s budget.\n */\n started: number;\n maxPosts: number;\n}\n\nexport interface DrainResult {\n picked: number;\n judged: number;\n dry: number;\n skippedClaimed: number;\n errors: number;\n emailed: number;\n ms: number;\n}\n\nexport async function drain(opts: DrainOpts): Promise<DrainResult> {\n const { started, maxPosts } = opts;\n const left = () => RUN_DEADLINE_MS - (Date.now() - started);\n const result: DrainResult = {\n picked: 0,\n judged: 0,\n dry: 0,\n skippedClaimed: 0,\n errors: 0,\n emailed: 0,\n ms: 0,\n };\n\n await db.init();\n const apiKey = Deno.env.get(\"ANTHROPIC_API_KEY\") ?? \"\";\n const dryRun = !apiKey;\n\n // A dry run leaves rows queued but must not rehearse the same ones every cron — see\n // listQueued. The moment a key is set, those rehearsed rows are picked up for real.\n // An isolate killed mid-post leaves its row claimed. Two minutes is far longer than the\n // 40 s a drain can take, so anything older than that is certainly abandoned.\n const freed = await db.releaseAbandonedClaims(CLAIM_TTL_MS);\n if (freed) console.log(\"UNCLAIMED\", JSON.stringify({ rows: freed }));\n\n const queue = await db.listQueued(maxPosts, !dryRun);\n // msLeft is worth logging: if init ever gets slow enough to eat the start margin,\n // the run would silently do nothing and only this line would say why.\n console.log(\n \"RUN\",\n JSON.stringify({ queued: queue.length, max: maxPosts, dryRun, msLeft: left() }),\n );\n if (!queue.length) {\n result.ms = Date.now() - started;\n return result;\n }\n\n const sendInDryRun = Deno.env.get(\"SEND_IN_DRY_RUN\") === \"1\";\n const model = Deno.env.get(\"JUDGE_MODEL\") || DEFAULT_MODEL;\n const criteria = loadCriteria(Deno.env.get(\"CRITERIA_JSON\"));\n if (!Deno.env.get(\"CRITERIA_JSON\")) console.log(\"CRITERIA_JSON unset — built-in default only\");\n if (dryRun) console.log(\"DRY_RUN: ANTHROPIC_API_KEY unset, no LLM call will be made\");\n\n const digest: DigestItem[] = [];\n\n for (const row of queue) {\n if (left() < START_MIN_LEFT_MS) {\n console.log(\"DEADLINE\", JSON.stringify({ msLeft: left(), stoppedBefore: row.uuid }));\n break;\n }\n // Claim the row before touching it. An email arriving while the cron is mid-run — or two\n // alerts seconds apart — would otherwise both pick the same row and judge it twice. The\n // UPDATE ... WHERE status='queued' is the whole lock: exactly one caller sees rowsAffected 1.\n if (!(await db.claimPost(String(row.uuid)))) {\n result.skippedClaimed++;\n console.log(\"CLAIMED_ELSEWHERE\", JSON.stringify({ uuid: row.uuid }));\n continue;\n }\n result.picked++;\n\n // handleOne renames the row when a legacy pid resolves to a uuid, so the error handler\n // has to write to whatever the row is called by then, not what it was called at the start.\n const ref = { uuid: String(row.uuid) };\n try {\n const item = await handleOne(row, ref, {\n apiKey,\n dryRun,\n sendInDryRun,\n model,\n criteria,\n left,\n counts: result,\n });\n if (item) digest.push(item);\n } catch (err) {\n if (err instanceof BlockedError) {\n // Craigslist blocked us. Stop the run. Do not retry and do not change egress:\n // evading a block is the fact that turns scraping into a legal problem.\n console.error(\"BLOCKED_BY_CL\", JSON.stringify({ blockID: err.blockID, url: err.url }));\n await db.updatePost(ref.uuid, {\n status: \"blocked_by_cl\",\n reason: `blockID=${err.blockID}`,\n });\n break;\n }\n // Everything else is assumed transient — a 429 or 529 from the model, a sapi 5xx, a\n // timeout — and goes back on the queue. Only after MAX_ATTEMPTS does it become 'error',\n // because an 'error' row is never looked at again and later reposts of it would inherit\n // nothing.\n result.errors++;\n const attempts = Number(row.attempts ?? 0) + 1;\n const giveUp = attempts >= MAX_ATTEMPTS;\n console.error(\n \"POST_ERROR\",\n JSON.stringify({ uuid: ref.uuid, attempts, giveUp, err: String(err).slice(0, 200) }),\n );\n await db.updatePost(ref.uuid, {\n status: giveUp ? \"error\" : \"queued\",\n attempts,\n reason: String(err).slice(0, 200),\n });\n }\n }\n\n if (digest.length) {\n await sendDigest(digest);\n for (const it of digest) await db.updatePost(it.uuid, { emailed_at: db.now() });\n result.emailed = digest.length;\n console.log(\n \"EMAILED\",\n JSON.stringify({ count: digest.length, uuids: digest.map((d) => d.uuid) }),\n );\n }\n result.ms = Date.now() - started;\n console.log(\"DONE\", JSON.stringify(result));\n return result;\n}\n\ninterface Cfg {\n apiKey: string;\n dryRun: boolean;\n sendInDryRun: boolean;\n model: string;\n criteria: ReturnType<typeof loadCriteria>;\n /** Milliseconds remaining in this run's self-imposed deadline. */\n left: () => number;\n /** Tallies for the DONE line; handleOne knows which outcome it reached. */\n counts: DrainResult;\n}\n\nasync function handleOne(\n row: db.Row,\n ref: { uuid: string },\n cfg: Cfg,\n): Promise<DigestItem | null> {\n let uuid = String(row.uuid);\n const search = String(row.search ?? \"\");\n\n // Legacy pid-only rows: the old numeric URL 301s to the /view/d/<slug>/<uuid> form, which\n // is the only pid -> uuid oracle there is (the uuid is a random 128-bit token).\n if (uuid.startsWith(\"pid:\")) {\n const res = await resolveUuid(String(row.url), RESOLVE_TIMEOUT_MS);\n if (res.status === 404) {\n await db.updatePost(uuid, { status: \"gone\", judged_at: db.now() });\n console.log(\"GONE\", JSON.stringify({ key: uuid, via: \"legacy-404\" }));\n return null;\n }\n if (!res.uuid) throw new Error(`no uuid from redirect (status ${res.status})`);\n const existing = await db.getPost(res.uuid);\n if (existing) {\n await db.updatePost(uuid, {\n status: \"dup\",\n dup_of: res.uuid,\n verdict: existing.verdict,\n score: existing.score,\n reason: existing.reason,\n judged_at: db.now(),\n });\n console.log(\"DUP\", JSON.stringify({ key: uuid, dupOf: res.uuid, rule: \"resolved-uuid\" }));\n return null;\n }\n await db.renameUuid(uuid, res.uuid);\n console.log(\"RESOLVED\", JSON.stringify({ from: uuid, to: res.uuid }));\n uuid = res.uuid;\n ref.uuid = res.uuid;\n }\n\n const { status, item } = await fetchDetail(uuid, DETAIL_TIMEOUT_MS);\n if (status === 404 || !item) {\n if (status === 404) {\n await db.updatePost(uuid, { status: \"gone\", judged_at: db.now() });\n console.log(\"GONE\", JSON.stringify({ uuid }));\n return null;\n }\n throw new Error(`sapi status ${status}`);\n }\n if (item.notices?.length) {\n console.log(\"NOTICES\", JSON.stringify({ uuid, notices: item.notices }));\n }\n\n const images = item.images ?? [];\n const lat = item.location?.lat ?? null;\n const lon = item.location?.lon ?? null;\n const hood = item.location?.neighborhood || (row.hood === null ? null : String(row.hood));\n\n await db.updatePost(uuid, {\n pid: item.postingId ?? null,\n repost_of: item.repostOf ?? null,\n title: item.title ?? String(row.title ?? \"\"),\n price: item.price ?? (row.price === null ? null : Number(row.price)),\n hood,\n lat,\n lon,\n image_ids: images.join(\",\"),\n record_created: item.recordCreated ?? null,\n url: item.url ?? String(row.url ?? \"\"),\n // status stays 'working' on purpose: if the isolate is killed between here and the final\n // verdict, releaseAbandonedClaims() puts the row back on the queue. An intermediate status\n // would strand it where neither the queue nor the sweep looks.\n });\n\n // Anything he has already said no to, in any of its forms.\n const blockKeys = [\n `pid:${item.postingId}`,\n ...(item.repostOf ? [`pid:${item.repostOf}`] : []),\n ...images.map((i) => `img:${imageIdentity(i)}`),\n ];\n const blockedBy = await db.isBlocked(blockKeys);\n if (blockedBy) {\n await db.updatePost(uuid, {\n status: \"rejected\",\n verdict: \"bad\",\n score: 0,\n reason: `blocked by earlier nope (${blockedBy})`,\n judged_at: db.now(),\n });\n console.log(\"BLOCKED\", JSON.stringify({ uuid, key: blockedBy }));\n return null;\n }\n\n const cand: DedupRow = {\n uuid,\n pid: item.postingId ?? null,\n repostOf: item.repostOf ?? null,\n imageIds: images,\n title: item.title ?? \"\",\n price: item.price ?? null,\n lat,\n lon,\n };\n const known = (await db.dedupRows()).map(toDedupRow);\n const dup = findDuplicate(cand, known);\n if (dup) {\n // Follow any dup_of chain to the row that actually carries the verdict. Without this a\n // post can be marked a duplicate of a row that is itself a duplicate of this post.\n const canonical = await canonicalUuid(dup.uuid, uuid);\n if (canonical === uuid) {\n console.log(\"DUP_SELF\", JSON.stringify({ uuid, matched: dup.uuid, rule: dup.rule }));\n } else {\n return await markDup(uuid, canonical, dup.rule);\n }\n }\n\n // Only now, for a post we have never judged, do we spend images and a model call.\n const picked = images.slice(0, MAX_IMAGES);\n const fetched: { mediaType: string; data: string }[] = [];\n for (const raw of picked) {\n if (cfg.left() < IMAGE_MIN_LEFT_MS) {\n console.log(\n \"IMAGE_DEADLINE\",\n JSON.stringify({ uuid, got: fetched.length, msLeft: cfg.left() }),\n );\n break;\n }\n const img = await fetchImageBase64(raw, \"600x450\", IMAGE_TIMEOUT_MS);\n if (img) fetched.push(img);\n }\n\n const { prompt, fellBack } = criteriaFor(cfg.criteria, search);\n if (fellBack) console.log(\"CRITERIA_FALLBACK\", JSON.stringify({ search, uuid }));\n const nopes = await db.feedbackExamples(search, \"nope\", NOPE_EXAMPLES);\n const yeses = await db.feedbackExamples(search, \"yes\", YES_EXAMPLES);\n const system = buildSystem(prompt, nopes, yeses);\n const userText = buildUserText({\n title: item.title ?? \"\",\n priceString: item.priceString ?? null,\n hood,\n body: bodyText(item.body),\n attributes: (item.attributes ?? []).map((a) => `${a.label ?? \"\"}: ${a.value ?? \"\"}`),\n category: item.category ?? null,\n postedIso: iso(item.postedDate),\n createdIso: iso(item.recordCreated),\n search,\n });\n\n let verdict: Verdict;\n if (cfg.dryRun) {\n const promptLen = system.length + userText.length;\n verdict = {\n verdict: \"meh\",\n score: 0,\n reason: `DRY_RUN: no ANTHROPIC_API_KEY (prompt ${promptLen} chars, ${fetched.length} images)`,\n flags: [\"dry\"],\n };\n // Deliberately stays 'queued'. A dry run is a rehearsal, not a decision: the moment a key\n // is set, the post must still be judged for real.\n await db.updatePost(uuid, {\n status: \"queued\",\n verdict: \"dry\",\n score: 0,\n reason: verdict.reason,\n flags: \"dry\",\n prompt_len: promptLen,\n judged_at: db.now(),\n });\n cfg.counts.dry++;\n console.log(\n \"DRY\",\n JSON.stringify({ uuid, promptLen, images: fetched.length, title: item.title }),\n );\n if (!cfg.sendInDryRun) return null;\n } else {\n verdict = await judge({\n apiKey: cfg.apiKey,\n model: cfg.model,\n system,\n userText,\n images: fetched,\n timeoutMs: Math.max(5_000, cfg.left() - JUDGE_RESERVE_MS),\n });\n await db.updatePost(uuid, {\n status: \"judged\",\n verdict: verdict.verdict,\n score: verdict.score,\n reason: verdict.reason,\n flags: verdict.flags.join(\",\"),\n prompt_len: system.length + userText.length,\n judged_at: db.now(),\n });\n cfg.counts.judged++;\n console.log(\n \"JUDGED\",\n JSON.stringify({\n uuid,\n verdict: verdict.verdict,\n score: verdict.score,\n images: fetched.length,\n title: item.title,\n }),\n );\n if (verdict.verdict !== \"good\") return null;\n }\n\n return await digestItem(uuid, item, hood, verdict, picked[0] ?? null);\n}\n\n/**\n * Walk dup_of to the row that actually holds the verdict. `self` is the post being judged:\n * if the chain leads back to it, the match is circular and there is no duplicate.\n */\nasync function canonicalUuid(start: string, self: string): Promise<string> {\n let cur = start;\n for (let i = 0; i < 4; i++) {\n const row = await db.getPost(cur);\n const next = row?.dup_of ? String(row.dup_of) : null;\n if (!next || next === cur) return cur;\n if (next === self) return self;\n cur = next;\n }\n return cur;\n}\n\nasync function markDup(uuid: string, dupOf: string, rule: string): Promise<null> {\n const orig = await db.getPost(dupOf);\n await db.updatePost(uuid, {\n status: \"dup\",\n dup_of: dupOf,\n verdict: orig?.verdict ?? null,\n score: orig?.score ?? null,\n reason: `duplicate of ${dupOf} (${rule})`,\n judged_at: db.now(),\n });\n console.log(\"DUP\", JSON.stringify({ uuid, dupOf, rule }));\n return null;\n}\n\nasync function digestItem(\n uuid: string,\n item: { title?: string; priceString?: string; url?: string },\n hood: string | null,\n verdict: Verdict,\n firstImage: string | null,\n): Promise<DigestItem> {\n const secret = Deno.env.get(\"FEEDBACK_SECRET\") ?? \"\";\n const base = await feedbackBase();\n let nopeUrl: string | null = null;\n let yesUrl: string | null = null;\n if (secret && base) {\n nopeUrl = feedbackLink(base, uuid, \"nope\", await sign(secret, uuid, \"nope\"));\n yesUrl = feedbackLink(base, uuid, \"yes\", await sign(secret, uuid, \"yes\"));\n } else {\n console.log(\"NO_FEEDBACK_LINKS\", JSON.stringify({ hasSecret: !!secret, base }));\n }\n return {\n uuid,\n title: item.title ?? \"\",\n priceString: item.priceString ?? null,\n hood,\n score: verdict.score,\n reason: verdict.reason,\n flags: verdict.flags,\n url: item.url ?? `https://www.craigslist.org/view/d/x/${uuid}`,\n imageUrl: firstImage ? imageUrl(firstImage, \"600x450\") : null,\n nopeUrl,\n yesUrl,\n };\n}\n\n/**\n * The feedback endpoint's own URL, for the buttons in the digest.\n *\n * Zero-config by default: Val Town injects VALTOWN_ENTRYPOINT (which names the user and the\n * val) and an API token, so the endpoint can be looked up at runtime and cached for the life\n * of the isolate. Set FEEDBACK_URL to skip the lookup. There is no VAL_URL variable in the\n * runtime — checked, the injected names are VALTOWN_ENTRYPOINT / VALTOWN_API_URL / valtown.\n */\nlet cachedBase: string | null | undefined;\n\nasync function feedbackBase(): Promise<string | null> {\n const explicit = Deno.env.get(\"FEEDBACK_URL\");\n if (explicit) return explicit.replace(/\\/$/, \"\");\n if (cachedBase !== undefined) return cachedBase;\n cachedBase = await discoverFeedbackUrl();\n return cachedBase;\n}\n\nasync function discoverFeedbackUrl(): Promise<string | null> {\n const token = Deno.env.get(\"valtown\") ?? Deno.env.get(\"VAL_TOWN_API_KEY\");\n const entry = Deno.env.get(\"VALTOWN_ENTRYPOINT\") ?? \"\";\n const m = entry.match(/esm\\.town\\/v\\/([^/]+)\\/([^/@]+)/);\n if (!token || !m) return null;\n const auth = { headers: { authorization: `Bearer ${token}` }, signal: AbortSignal.timeout(8000) };\n try {\n const valRes = await fetch(`https://api.val.town/v2/alias/vals/${m[1]}/${m[2]}`, auth);\n if (!valRes.ok) return null;\n const valId = (await valRes.json())?.id;\n if (!valId) return null;\n const fileRes = await fetch(\n `https://api.val.town/v2/vals/${valId}/files?path=feedback.http.tsx`,\n auth,\n );\n if (!fileRes.ok) return null;\n const endpoint = (await fileRes.json())?.data?.[0]?.links?.endpoint;\n return typeof endpoint === \"string\" ? endpoint : null;\n } catch (e) {\n console.error(\"FEEDBACK_URL_DISCOVERY_FAILED\", String(e));\n return null;\n }\n}\n\nfunction iso(epoch: number | undefined): string | null {\n return epoch ? new Date(epoch * 1000).toISOString().slice(0, 16).replace(\"T\", \" \") : null;\n}\n\nfunction toDedupRow(r: db.Row): DedupRow {\n return {\n uuid: String(r.uuid),\n pid: r.pid === null ? null : Number(r.pid),\n repostOf: r.repost_of === null ? null : Number(r.repost_of),\n imageIds: String(r.image_ids ?? \"\").split(\",\").filter(Boolean),\n title: String(r.title ?? \"\"),\n price: r.price === null ? null : Number(r.price),\n lat: r.lat === null ? null : Number(r.lat),\n lon: r.lon === null ? null : Number(r.lon),\n };\n}\n",[],[164,285],["D",1789025748356],[],[164,64],"7f6a03a8-6d6b-4ba3-a21b-7abd5535415a","db.ts","e881e32e-ace8-11f1-9915-1607ee4eb77e",60,["D",1789030162978],"/**\n * Val-scoped SQLite. Note the import path ends in /main.ts — the legacy\n * https://esm.town/v/std/sqlite module is account-scoped and returns rows as arrays\n * instead of objects. Do not mix the two.\n *\n * Quota is 10 MB shared between SQLite and blob for the whole account, so this stores ids,\n * verdicts and short reasons only. Never image bytes.\n */\nimport { sqlite } from \"https://esm.town/v/std/sqlite/main.ts\";\n\nexport type Row = Record<string, string | number | null>;\n/** What the libsql client accepts as a bound parameter. */\nexport type Arg = string | number | bigint | boolean | null | Uint8Array;\n\nasync function run(sql: string, args: Arg[] = []): Promise<Row[]> {\n const res = await sqlite.execute({ sql, args });\n return (res.rows ?? []) as unknown as Row[];\n}\n\n/** Same, but for writes where the number of rows touched is the answer we need. */\nasync function write(sql: string, args: Arg[] = []): Promise<number> {\n const res = await sqlite.execute({ sql, args });\n return Number(res.rowsAffected ?? 0);\n}\n\n/** Migrations are CREATE ... IF NOT EXISTS only; every entry point calls this first. */\nexport async function init(): Promise<void> {\n await run(`CREATE TABLE IF NOT EXISTS posts (\n uuid TEXT PRIMARY KEY,\n pid INTEGER,\n repost_of INTEGER,\n search TEXT,\n title TEXT,\n price INTEGER,\n hood TEXT,\n lat REAL,\n lon REAL,\n image_ids TEXT,\n thumb_img_id TEXT,\n url TEXT,\n record_created INTEGER,\n status TEXT,\n verdict TEXT,\n score INTEGER,\n reason TEXT,\n flags TEXT,\n dup_of TEXT,\n prompt_len INTEGER,\n queued_at TEXT,\n judged_at TEXT,\n emailed_at TEXT\n )`);\n await run(`CREATE INDEX IF NOT EXISTS posts_pid ON posts (pid)`);\n await run(`CREATE INDEX IF NOT EXISTS posts_repost_of ON posts (repost_of)`);\n await run(`CREATE INDEX IF NOT EXISTS posts_status ON posts (status)`);\n await run(`CREATE TABLE IF NOT EXISTS blocked (\n key TEXT PRIMARY KEY, from_uuid TEXT, reason TEXT, at TEXT)`);\n await run(`CREATE TABLE IF NOT EXISTS feedback (\n uuid TEXT, kind TEXT, reason TEXT, at TEXT)`);\n await run(`CREATE INDEX IF NOT EXISTS feedback_uuid ON feedback (uuid)`);\n\n // Added after the table already existed on platform, so it cannot go in the CREATE above.\n // ALTER TABLE ADD COLUMN errors if the column is there, so ask first rather than swallowing.\n const cols = await run(`PRAGMA table_info(posts)`);\n const have = new Set(cols.map((c) => String(c.name)));\n if (!have.has(\"attempts\")) {\n await run(`ALTER TABLE posts ADD COLUMN attempts INTEGER DEFAULT 0`);\n console.log(\"MIGRATION posts.attempts added\");\n }\n if (!have.has(\"claim_at\")) {\n await run(`ALTER TABLE posts ADD COLUMN claim_at TEXT`);\n console.log(\"MIGRATION posts.claim_at added\");\n }\n}\n\nexport const now = () => new Date().toISOString();\n\nexport interface QueuedInsert {\n uuid: string;\n pid: number | null;\n search: string;\n title: string;\n price: number | null;\n hood: string | null;\n thumbImgId: string | null;\n url: string;\n}\n\nexport async function insertQueued(p: QueuedInsert): Promise<void> {\n await run(\n `INSERT OR IGNORE INTO posts\n (uuid, pid, search, title, price, hood, thumb_img_id, url, status, queued_at)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'queued', ?)`,\n [p.uuid, p.pid, p.search, p.title, p.price, p.hood, p.thumbImgId, p.url, now()],\n );\n}\n\nexport async function getPost(uuid: string): Promise<Row | null> {\n const rows = await run(`SELECT * FROM posts WHERE uuid = ?`, [uuid]);\n return rows[0] ?? null;\n}\n\nexport async function existsPid(pid: number): Promise<string | null> {\n const rows = await run(`SELECT uuid FROM posts WHERE pid = ? LIMIT 1`, [pid]);\n return rows[0] ? String(rows[0].uuid) : null;\n}\n\n/** Cheap pre-judge check: has this thumbnail already been seen on a post we know about? */\nexport async function existsImage(imgId: string): Promise<string | null> {\n const rows = await run(\n `SELECT uuid FROM posts\n WHERE thumb_img_id = ? OR (image_ids IS NOT NULL AND instr(image_ids, ?) > 0)\n LIMIT 1`,\n [imgId, imgId],\n );\n return rows[0] ? String(rows[0].uuid) : null;\n}\n\nexport async function isBlocked(keys: string[]): Promise<string | null> {\n for (const k of keys) {\n const rows = await run(`SELECT key FROM blocked WHERE key = ?`, [k]);\n if (rows[0]) return k;\n }\n return null;\n}\n\nexport async function addBlocked(keys: string[], fromUuid: string, reason: string | null) {\n for (const k of keys) {\n await run(\n `INSERT INTO blocked (key, from_uuid, reason, at) VALUES (?, ?, ?, ?)\n ON CONFLICT(key) DO UPDATE SET from_uuid = excluded.from_uuid, at = excluded.at`,\n [k, fromUuid, reason, now()],\n );\n }\n}\n\n/**\n * Attempts first, then arrival order: a post that has already failed twice must not keep\n * elbowing ahead of posts that have never been tried.\n *\n * `includeDry` is what stops a dry run from burning craigslist requests forever. A dry row\n * stays 'queued' on purpose (it has not been judged), but re-rehearsing it every 15 minutes\n * would cost a detail call plus five image fetches per row per cron — four rows left overnight\n * is ~2,300 requests a day, over craigslist's own written 1,000-page budget and for nothing.\n * So: dry run skips rows already rehearsed once; a run with a real key takes them.\n */\nexport async function listQueued(limit: number, includeDry: boolean): Promise<Row[]> {\n const dryClause = includeDry ? \"\" : ` AND COALESCE(verdict, '') != 'dry'`;\n return await run(\n `SELECT * FROM posts WHERE status = 'queued'${dryClause}\n ORDER BY COALESCE(attempts, 0) ASC, queued_at ASC LIMIT ?`,\n [limit],\n );\n}\n\n/**\n * The columns the dedup ladder needs, for every post we have already resolved.\n * 'working' rows are mid-judgement in some other isolate and 'gone' rows are left out on\n * purpose: they were never judged, and an item that comes back\n * after its earlier post expired deserves a fresh look rather than an inherited nothing.\n * DRY_RUN rows are left out for the same reason — 'dry' is not a verdict, and a repost must\n * not inherit it.\n */\nexport async function dedupRows(limit = 5000): Promise<Row[]> {\n return await run(\n `SELECT uuid, pid, repost_of, image_ids, title, price, lat, lon, verdict, status, dup_of\n FROM posts\n WHERE status NOT IN ('queued', 'working', 'gone') AND COALESCE(verdict, '') != 'dry'\n ORDER BY rowid DESC LIMIT ?`,\n [limit],\n );\n}\n\n/**\n * Take exclusive ownership of a queued row. The email handler now judges inline, so a cron run\n * and an inbound alert (or two alerts seconds apart) can be in the loop at the same time and\n * would otherwise both pick the same row. The `WHERE status = 'queued'` is the entire lock:\n * SQLite applies the UPDATE to one caller and reports 0 rows affected to the other.\n */\nexport async function claimPost(uuid: string): Promise<boolean> {\n const n = await write(\n `UPDATE posts SET status = 'working', claim_at = ? WHERE uuid = ? AND status = 'queued'`,\n [now(), uuid],\n );\n return n === 1;\n}\n\n/**\n * Put back rows whose claim is older than the drain could possibly be — the isolate that\n * claimed them was killed mid-post. Returns how many were freed.\n */\nexport async function releaseAbandonedClaims(ttlMs: number): Promise<number> {\n const cutoff = new Date(Date.now() - ttlMs).toISOString();\n return await write(\n `UPDATE posts SET status = 'queued'\n WHERE status = 'working' AND (claim_at IS NULL OR claim_at < ?)`,\n [cutoff],\n );\n}\n\nexport async function updatePost(uuid: string, fields: Record<string, Arg>): Promise<void> {\n const keys = Object.keys(fields);\n if (!keys.length) return;\n await run(\n `UPDATE posts SET ${keys.map((k) => `${k} = ?`).join(\", \")} WHERE uuid = ?`,\n [...keys.map((k) => fields[k]), uuid],\n );\n}\n\n/** A legacy pid-only row becomes a normal row once the 301 gives us its uuid. */\nexport async function renameUuid(oldUuid: string, newUuid: string): Promise<void> {\n await run(`UPDATE posts SET uuid = ? WHERE uuid = ?`, [newUuid, oldUuid]);\n}\n\nexport async function addFeedback(uuid: string, kind: string, reason: string | null) {\n await run(`INSERT INTO feedback (uuid, kind, reason, at) VALUES (?, ?, ?, ?)`, [\n uuid,\n kind,\n reason,\n now(),\n ]);\n}\n\nexport interface Example {\n title: string;\n price: number | null;\n reason: string | null;\n}\n\n/** Few-shot examples for the judge: what he rejected (or liked) on this saved search. */\nexport async function feedbackExamples(\n search: string,\n kind: \"nope\" | \"yes\",\n limit: number,\n): Promise<Example[]> {\n // For a rejection the only honest reason is the one he typed. p.reason is the judge's own\n // argument for why the listing was GOOD — handing that back as the explanation for a \"nope\"\n // teaches exactly the wrong thing. A 'yes' may fall back to it: there it agrees with him.\n const reasonExpr = kind === \"nope\" ? \"f.reason\" : \"COALESCE(f.reason, p.reason)\";\n const rows = await run(\n `SELECT p.title AS title, p.price AS price, ${reasonExpr} AS reason\n FROM feedback f JOIN posts p ON p.uuid = f.uuid\n WHERE f.kind = ? AND p.search = ?\n ORDER BY f.at DESC LIMIT ?`,\n [kind, search, limit],\n );\n return rows.map((r) => ({\n title: String(r.title ?? \"\"),\n price: r.price === null ? null : Number(r.price),\n reason: r.reason === null ? null : String(r.reason),\n }));\n}\n\nexport async function dumpTables(limit = 50): Promise<Record<string, Row[]>> {\n return {\n posts: await run(`SELECT * FROM posts ORDER BY rowid DESC LIMIT ?`, [limit]),\n blocked: await run(`SELECT * FROM blocked ORDER BY rowid DESC LIMIT ?`, [limit]),\n feedback: await run(`SELECT * FROM feedback ORDER BY rowid DESC LIMIT ?`, [limit]),\n };\n}\n\nexport async function hasFeedback(uuid: string, kind: string): Promise<boolean> {\n const rows = await run(`SELECT 1 AS x FROM feedback WHERE uuid = ? AND kind = ? LIMIT 1`, [\n uuid,\n kind,\n ]);\n return rows.length > 0;\n}\n\nexport async function setFeedbackReason(uuid: string, kind: string, reason: string) {\n await run(`UPDATE feedback SET reason = ? WHERE uuid = ? AND kind = ?`, [reason, uuid, kind]);\n}\n\n",[],[164,295],"6e14e58e-00b1-41ec-864c-999505d71ef6","cl.ts","e81f36d4-ace8-11f1-81ba-1607ee4eb77e",8,["D",1789025175111],"/**\n * Craigslist access. Etiquette rules baked in, all from research/craigslist-mechanics.md:\n * - browser-like User-Agent, GET only (images.craigslist.org answers 405 to HEAD)\n * - never ?format=rss (403 + block page; the parameter itself is a bot signal)\n * - never touch /reply or /flag (robots.txt Disallow)\n * - if a body ever contains <title>blocked, log the blockID and STOP. Do not retry,\n * do not change egress. Evading a block is the fact that turns scraping into CFAA exposure.\n * Volume here is ~1 detail call + <=5 images per alerted post, three orders of magnitude\n * under craigslist's own written 1,000-pages/24h budget.\n */\n\nconst UA =\n \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36\";\n\nexport class BlockedError extends Error {\n constructor(public blockID: string, public url: string) {\n super(`craigslist block page (blockID=${blockID}) for ${url}`);\n this.name = \"BlockedError\";\n }\n}\n\nfunction assertNotBlocked(body: string, url: string) {\n if (/\\s*blocked\\s*<\\/title>/i.test(body)) {\n const m = body.match(/blockID[=:]\\s*(\\d+)/i);\n throw new BlockedError(m ? m[1] : \"unknown\", url);\n }\n}\n\nasync function get(url: string, timeoutMs: number, accept: string): Promise<Response> {\n return await fetch(url, {\n method: \"GET\",\n redirect: \"follow\",\n headers: { \"User-Agent\": UA, \"Accept\": accept, \"Accept-Language\": \"en-US,en;q=0.9\" },\n signal: AbortSignal.timeout(timeoutMs),\n });\n}\n\nexport interface DetailItem {\n postingId: number;\n postingUuid: string;\n repostOf?: number;\n recordCreated?: number;\n postedDate?: number;\n updatedDate?: number;\n images?: string[];\n title?: string;\n price?: number;\n priceString?: string;\n body?: string;\n category?: string;\n categoryId?: number;\n attributes?: { label?: string; value?: string }[];\n notices?: unknown[];\n location?: { lat?: number; lon?: number; neighborhood?: string; area?: string };\n url?: string;\n}\n\n/** GET sapi detail. Returns status 404 with item null for a post that is gone. */\nexport async function fetchDetail(\n uuid: string,\n timeoutMs = 10_000,\n): Promise<{ status: number; item: DetailItem | null }> {\n const url = `https://sapi.craigslist.org/web/v8/postings/${\n encodeURIComponent(uuid)\n }?cc=US&lang=en`;\n const res = await get(url, timeoutMs, \"application/json\");\n const text = await res.text();\n assertNotBlocked(text, url);\n if (res.status === 404) return { status: 404, item: null };\n if (!res.ok) return { status: res.status, item: null };\n const json = JSON.parse(text);\n const item = json?.data?.items?.[0] ?? null;\n return { status: res.status, item };\n}\n\n/**\n * Legacy alert rows carry only a numeric post id. The old numeric URL 301s to the\n * /view/d/<slug>/<uuid> form, so the redirect is the pid -> uuid oracle. A deleted or\n * expired post 404s here with no redirect.\n */\nexport async function resolveUuid(\n legacyUrl: string,\n timeoutMs = 10_000,\n): Promise<{ status: number; uuid: string | null }> {\n const res = await get(legacyUrl, timeoutMs, \"text/html\");\n const text = await res.text();\n assertNotBlocked(text, legacyUrl);\n if (!res.ok) return { status: res.status, uuid: null };\n const m = res.url.match(/\\/view\\/d\\/[^/]+\\/([A-Za-z0-9]{15,30})/) ??\n text.match(/rel=\"canonical\"[^>]*\\/view\\/d\\/[^/]+\\/([A-Za-z0-9]{15,30})/);\n return { status: res.status, uuid: m ? m[1] : null };\n}\n\n/** \"3:00U0U_ixFsruu1I5I_0ak07K\" -> the fetchable URL at the given size box. */\nexport function imageUrl(rawId: string, size = \"600x450\"): string {\n return `https://images.craigslist.org/${rawId.replace(/^\\d+:/, \"\")}_${size}.jpg`;\n}\n\nfunction toBase64(bytes: Uint8Array): string {\n let s = \"\";\n const chunk = 0x8000;\n for (let i = 0; i < bytes.length; i += chunk) {\n s += String.fromCharCode(...bytes.subarray(i, i + chunk));\n }\n return btoa(s);\n}\n\n/** Fetches one image and base64s it in memory. Never persisted — the storage quota is 10 MB. */\nexport async function fetchImageBase64(\n rawId: string,\n size = \"600x450\",\n timeoutMs = 10_000,\n): Promise<{ mediaType: string; data: string } | null> {\n const url = imageUrl(rawId, size);\n const res = await get(url, timeoutMs, \"image/*\");\n if (!res.ok) return null;\n const ct = res.headers.get(\"content-type\") ?? \"\";\n if (!ct.startsWith(\"image/\")) return null;\n const buf = new Uint8Array(await res.arrayBuffer());\n if (buf.length === 0) return null;\n return { mediaType: ct.split(\";\")[0].trim(), data: toBase64(buf) };\n}\n\n/** Strip the HTML craigslist puts in `body`, so the judge reads plain text. */\nexport function bodyText(html: string | undefined, max = 4000): string {\n if (!html) return \"\";\n return html\n .replace(/<showcontactinfo[^>]*>.*?<\\/showcontactinfo>/gis, \"[contact info]\")\n .replace(/<br\\s*\\/?>/gi, \"\\n\")\n .replace(/<[^>]+>/g, \"\")\n .replace(/ /g, \" \")\n .replace(/&/g, \"&\")\n .replace(/</g, \"<\")\n .replace(/>/g, \">\")\n .replace(/"/g, '\"')\n .replace(/'/g, \"'\")\n .replace(/\\n{3,}/g, \"\\n\\n\")\n .trim()\n .slice(0, max);\n}\n",[],[164,303],"c19b5d71-7c9f-4b31-8ff4-cbc4e764009a",["D",1789025174616],"directory",[177,178,179,180,181,182,183,184],[164],["D",1789025029995],["D",1789546246980],{"_11":12,"_132":70},{"_11":12,"_128":-5,"_69":70,"_30":68,"_129":130,"_131":24},{"_39":119,"_13":16,"_33":36,"_120":80,"_121":-5,"_122":-5,"_123":-5,"_124":-5,"_125":-5,"_126":14,"_127":12},"ctx",{"_52":-5,"_151":353,"_354":355,"_356":357,"_358":24,"_359":24,"_360":24,"_361":93,"_362":-5,"_363":-7,"_364":-7,"_365":26,"_366":24,"_367":24,"_368":-5,"_369":24,"_370":24,"_371":24,"_372":-5},"ENV",{"_324":325,"_326":327,"_328":16,"_329":330,"_331":332,"_333":334,"_335":336,"_337":338,"_339":340,"_341":342,"_343":344,"_345":346,"_347":348,"_349":350,"_351":352},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","9e167d8120a57de5d747867e0b686da51bcc19a4","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"] </article></body></html>

History · dedup.ts