\n \n AI Assistant\n

👋 Ask me anything

I'll search the knowledge base and answer based on what I find.

\n \n \n

\n\n\n\n`;\n\n return new Response(html, { headers: { \"Content-Type\": \"text/html\" } });\n}\n\n// ─── MAIN HANDLER ─────────────────────────────────────────────────────────────\nexport default async function(req: Request): Promise {\n const url = new URL(req.url);\n\n if (req.method === \"POST\" && url.pathname === \"/chat\") {\n return handleChat(req);\n }\n\n return renderUI();\n}\n```\n\nDeploy this val and visit its URL — you have a working chatbot. 🎉\n\n---\n\n## Step 3 — Customize It\n\n### Change the persona\n\nEdit `SYSTEM_PROMPT` to match your use case:\n\n```typescript\n// Support bot\nconst SYSTEM_PROMPT = `You are a helpful support assistant for Acme Inc.\nAnswer questions using ONLY the context provided. Be concise and friendly.\nIf the answer isn't in the context, say so and suggest contacting support.`;\n\n// Internal knowledge bot\nconst SYSTEM_PROMPT = `You are an internal assistant for the engineering team.\nAnswer questions based on the provided documentation.\nUse precise technical language. Quote the relevant section when helpful.`;\n\n// Sales assistant\nconst SYSTEM_PROMPT = `You are a product expert helping potential customers.\nAnswer questions based on the provided product information.\nBe enthusiastic but accurate. Never make up pricing or features.`;\n```\n\n### Tune retrieval quality\n\nAdjust the search parameters in `retrieveContext()`:\n\n```typescript\nbody: JSON.stringify({\n query,\n k: 4, // increase to 6-8 for broader context (longer prompts)\n maxDistance: 0.68, // decrease (e.g. 0.5) to be stricter — fewer but more relevant results\n // increase (e.g. 0.75) if the bot says \"no info\" too often\n hybrid: { enabled: true, alpha: 0.15 }, // increase alpha if your docs have specific terms/codes\n}),\n```\n\n**Not sure what `maxDistance` to set?** Run this in your browser:\n```\nhttps://YOUR_SLIMARMOR_ENDPOINT/calibrate?q=a+typical+user+question\n```\nIt tells you the `tight`, `balanced`, and `loose` thresholds for your actual data.\n\n### Use a more powerful model\n\nThe free proxy gives you `gpt-4o-mini` at no cost. For harder questions, switch models:\n\n```typescript\n// Free (unlimited on Val Town):\nmodel: \"gpt-4o-mini\" // best free option — fast, smart, good for Q&A\nmodel: \"gpt-4.1-nano\" // even faster, slightly less capable\nmodel: \"gpt-5-nano\" // latest nano — free tier\n\n// Pro users only (10 requests/24h before fallback):\nmodel: \"gpt-4o\" // most capable — use for complex reasoning\nmodel: \"gpt-4.1\" // latest high-quality model\n\n// Or bring your own key (no limits):\n// Add OPENAI_API_KEY env var, then:\nimport { OpenAI } from \"npm:openai\"; // uses your key automatically\n```\n\n### Add source citations in the UI\n\nThe `/chat` endpoint already returns `sources` (an array of SlimArmor record IDs). The UI displays them as \"📚 Sources: doc-1, doc-2\". To show friendlier source names, store a `title` field in your metadata:\n\n```typescript\n// When ingesting:\n{ id: \"faq-shipping\", text: \"...\", meta: { title: \"Shipping Policy\", url: \"/help/shipping\" } }\n\n// In retrieveContext(), return richer source info:\nconst sources = results.map((r: any) => ({\n id: r.id,\n title: r.meta?.title ?? r.id,\n url: r.meta?.url,\n}));\n\n// In the UI, render as a link:\nsrc.innerHTML = '📚 ' + sources.map(s =>\n s.url ? \\`\\${s.title}\\` : s.title\n).join(', ');\n```\n\n---\n\n## Step 4 — Keep Your Docs Fresh\n\nRe-run your ingest script whenever your content changes. SlimArmor's content-hash deduplication means **only changed docs get re-embedded** — the rest are skipped instantly at no cost.\n\n```typescript\n// Re-ingest is safe to run anytime.\n// If \"Refund Policy\" text didn't change → skipped (no API call)\n// If \"Shipping Policy\" changed → re-embedded automatically\n\nconsole.log(`✅ Done: ${result.embedded} embedded, ${result.skipped} skipped`);\n// → \"✅ Done: 1 embedded, 3 skipped\"\n```\n\n**For production**, hook this into your CMS or content pipeline:\n- Webhook from Notion/Contentful/Sanity on publish → trigger ingest val\n- Or run a scheduled interval val that re-ingests nightly\n\n---\n\n## How Context Injection Works (Under the Hood)\n\nWhen a user sends a message, here's exactly what the LLM receives:\n\n```\n[system]\nYou are a helpful support assistant for Acme Inc.\nAnswer questions using ONLY the context provided...\n\n[user - message 1]\nWhat's your return policy?\n\n[assistant - message 1]\nYou can return unused items within 30 days...\n\n[user - message 2] ← latest message, with context injected\nCan I return a digital product?\n\n---\nRelevant knowledge base context:\n[1] Our return policy allows returns within 30 days of purchase for unused items...\n Digital products and gift cards are non-refundable after download...\n[2] To start a return, email returns@example.com with your order number...\n---\n```\n\nThe LLM sees the full conversation history *plus* the freshly retrieved context for the latest question. It answers using that context, not its training data. That's RAG.\n\n---\n\n## Common Issues\n\n**Bot says \"I don't have that information\" even though you've ingested the docs**\n→ Your `maxDistance` is too strict. Try raising it from `0.68` to `0.72`, or run `/calibrate` to find the right value.\n\n**Bot makes things up / ignores the context**\n→ Make your system prompt stronger: `\"You MUST answer ONLY from the provided context. Do not use any prior knowledge.\"` Also lower `temperature` to `0.1`.\n\n**Bot repeats the same answer regardless of question**\n→ Your docs are too similar to each other. Make sure you've ingested varied content covering different topics.\n\n**Responses are slow**\n→ Normal — there's a round-trip to SlimArmor (~100ms) plus the LLM (~500-1500ms). For snappier UX, show the \"⏳ Searching...\" indicator (already in the UI above).\n\n**\"Embedding API error 401\" when ingesting**\n→ Your `NEBIUS_API_KEY` (or whichever provider) is missing or expired. The chatbot LLM side still works — only ingestion is broken.\n\n---\n\n## Full Architecture Recap\n\n```\nUser types message\n │\n ▼\nPOST /chat (my-chatbot.ts)\n │\n ├─▶ POST /search (SlimArmor)\n │ Uses Nebius embeddings to find relevant docs\n │ Returns top 4 chunks + their IDs\n │\n └─▶ openai.chat.completions.create() (Val Town free proxy)\n Sends: system prompt + history + user message + context\n Returns: answer text\n │\n ▼\nResponse: { reply: \"...\", sources: [\"doc-1\", \"doc-2\"] }\n │\n ▼\nUI renders answer + source list\n```\n\n**Two vals, one chatbot:**\n| Val | Purpose | Cost |\n|---|---|---|\n| `slimarmor` (your fork) | Stores + searches docs | Nebius free tier |\n| `my-chatbot` | Chat UI + RAG logic | Free (LLM via Val Town proxy) |\n\n---\n\n## Next Steps\n\n- **Add auth** — protect your chatbot with a password or login using Val Town's session tools\n- **Log conversations** — store chat history in SlimArmor itself (or a separate SQLite table) for analytics\n- **Multi-tenant** — use `meta.tenant_id` filters to serve different knowledge bases to different users from one instance\n- **Slack / Discord bot** — replace the UI handler with a webhook handler for your messaging platform\n- **Streaming responses** — use `openai.chat.completions.create({ stream: true })` for token-by-token output (see [Val Town streaming examples](https://www.val.town))\n",[],[203],{"_104":211,"_97":212,"_162":213,"_164":-5,"_165":147,"_166":146,"_168":169,"_106":214,"_101":215,"_172":173,"_174":216,"_176":22,"_177":-5,"_178":217,"_100":218},"6e9da8a2-15d7-11f1-a007-42dde27851f2","README.md","70cc2b8a-ffe3-11f0-a8f4-42dde27851f2",["D",1772415044284],16,"# 🛡️ SlimArmor — Lightweight Vector Database for Val Town\n\n> Semantic search powered by libSQL/Turso DiskANN — runs entirely on Val Town's built-in SQLite. No external database needed.\n\n**[→ Fork this val on Val Town](https://www.val.town/x/kamenxrider/slimarmor)** to deploy your own instance in ~2 minutes.\n\n---\n\n## What Is This?\n\nSlimArmor is a self-hosted vector database that lets you store text and search it **by meaning**, not just keywords.\n\n**Example:**\n- You store: `\"Dogs are loyal companions\"`\n- You search: `\"furry pets\"`\n- It finds the dog text — even though the words are different ✅\n\nThis is called **semantic search** and it's powered by AI embeddings. SlimArmor handles all the hard parts: embedding generation, storage, indexing, deduplication, and retrieval.\n\n---\n\n## Features\n\n| Feature | Detail |\n|---|---|\n| 🔍 **Semantic search** | Cosine similarity via libSQL DiskANN vector index |\n| ⚡ **Hybrid search** | Vector + BM25 keyword boosting |\n| 🧠 **Smart dedup** | Content-hashing skips re-embedding unchanged text |\n| 📦 **Batch upsert** | Send hundreds of records in one API call |\n| 🔖 **Metadata filters** | Filter results by any JSON metadata field |\n| 📄 **Chunked upsert** | Auto-split long documents into overlapping chunks |\n| 🔄 **Export / Import** | Migrate data between instances or providers |\n| 🖥️ **Browser CLI** | Built-in terminal UI at `/ui` — no client needed |\n| 🔐 **Optional auth** | Bearer token protection for write operations |\n| 🔌 **Multi-provider** | Nebius, OpenAI, OpenRouter, or any OpenAI-compatible API |\n\n---\n\n## Deploy Your Own (2 minutes)\n\n### Step 1 — Fork the val\n\nGo to **[val.town/x/kamenxrider/slimarmor](https://www.val.town/x/kamenxrider/slimarmor)** and click **Fork**.\n\n### Step 2 — Set environment variables\n\nIn your forked val, go to **Settings → Environment Variables** and add:\n\n**Required — pick one:**\n```\nNEBIUS_API_KEY=your-nebius-api-key # default provider, free tier available\n```\n```\nOPENAI_API_KEY=sk-your-key # + set EMBEDDING_PROVIDER=openai\n```\n```\nEMBEDDING_API_URL=https://... # any OpenAI-compatible API\nEMBEDDING_API_KEY=your-key\nEMBEDDING_MODEL=your-model-name\nEMBEDDING_DIM=1536 # must match your model's output dimensions\n```\n\n> **Any OpenAI-compatible embedding API works.** The only thing that matters is setting `EMBEDDING_DIM` to match your model's output dimensions — this determines how the database table is created. See [Provider Configuration](#provider-configuration) below for details and a model comparison table.\n\n**Recommended:**\n```\nADMIN_TOKEN=some-secret-token\n```\n\n> Without `ADMIN_TOKEN`, all operations are open (anyone can write). Set this to protect write endpoints.\n\n### Step 3 — Open your endpoint\n\nYour API is live at the URL shown in your val's HTTP endpoint. Visit `/ui` for the interactive browser CLI, or `/` for the API info page.\n\n### Step 4 — Add data and search\n\n```bash\n# Add a record\ncurl -X POST https://YOUR_ENDPOINT/upsert \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer YOUR_ADMIN_TOKEN\" \\\n -d '{\"id\": \"note-1\", \"text\": \"Dogs are loyal companions\", \"meta\": {\"category\": \"animals\"}}'\n\n# Search\ncurl -X POST https://YOUR_ENDPOINT/search \\\n -H \"Content-Type: application/json\" \\\n -d '{\"query\": \"furry pets\", \"k\": 5}'\n```\n\nThat's it. 🎉\n\n---\n\n## Provider Configuration\n\nSlimArmor works with **any OpenAI-compatible embedding API** — the provider doesn't matter, only two things do:\n\n1. **The API URL** accepts `POST /v1/embeddings` with `{\"model\": \"...\", \"input\": [...]}`\n2. **The embedding dimensions** your chosen model outputs — this is baked into the database schema at first run and cannot be changed without a full reset\n\n> ⚠️ **Dimensions are locked at first run.** When `setup()` is called for the first time, the table is created with a fixed `F32_BLOB(N)` column where N is your model's output dimension. Switching to a model with different dimensions requires exporting your data, clearing the DB, and re-importing. Use `EMBEDDING_DIM=auto` to detect dimensions automatically.\n\nSet env vars to configure your provider:\n\n### Nebius (Default — Best Value)\n```\nNEBIUS_API_KEY=your-key\n```\n| Model | Dimensions | Notes |\n|---|---|---|\n| `Qwen/Qwen3-Embedding-8B` | 4096 | Default — high quality, generous free tier |\n\n### OpenAI\n```\nEMBEDDING_PROVIDER=openai\nOPENAI_API_KEY=sk-your-key\n```\n| Model | Dimensions | Notes |\n|---|---|---|\n| `text-embedding-3-small` | 1536 | Fast & cheap — fits ~130k records/GB |\n| `text-embedding-3-large` | 3072 | Higher quality — fits ~65k records/GB |\n\n### OpenRouter\n```\nEMBEDDING_PROVIDER=openrouter\nOPENROUTER_API_KEY=your-key\nEMBEDDING_MODEL=openai/text-embedding-3-small\nEMBEDDING_DIM=1536\n```\n\n### Any Other OpenAI-Compatible API\nIf your provider exposes a `/v1/embeddings` endpoint, just point SlimArmor at it:\n```\nEMBEDDING_API_URL=https://your-provider.com/v1/embeddings\nEMBEDDING_API_KEY=your-key\nEMBEDDING_MODEL=your-model-name\nEMBEDDING_DIM=768\n```\nThis works with **Ollama**, **LM Studio**, **Together AI**, **Fireworks**, **Mistral**, **Cohere** (via their OpenAI-compat layer), and any other compatible service.\n\n### Choosing a Model — Dimensions vs Capacity\n\nDimensions determine both **search quality** and **storage capacity**:\n\n| Dimensions | Example Models | Records per 1 GB | Quality |\n|---|---|---|---|\n| 4096 | Nebius Qwen3-8B | ~47,500 | 🟢 Excellent |\n| 3072 | OpenAI text-embedding-3-large | ~65,000 | 🟢 Excellent |\n| 1536 | OpenAI text-embedding-3-small | ~130,000 | 🟡 Good |\n| 768 | nomic-embed-text, bge-small | ~260,000 | 🟡 Good |\n| 384 | all-MiniLM-L6-v2 | ~520,000 | 🟠 Okay for simple tasks |\n\nHigher dimensions = better semantic understanding but more storage. For most use cases, 1536 (OpenAI small) or 768 (nomic) is the sweet spot.\n\n> ⚠️ **Switching models requires a full reset.** Embeddings from different models are incompatible — a vector from model A means nothing when compared to a vector from model B. Export your data first (`GET /export`), then `POST /clear?confirm=yes`, update env vars, then `POST /import`.\n\n---\n\n## API Reference\n\nAll endpoints are relative to your val's HTTP endpoint URL.\n\n### Core Endpoints\n\n#### `POST /upsert` 🔐\nInsert or update one or many records.\n\n**Single record:**\n```json\n{ \"id\": \"doc-1\", \"text\": \"Your text here\", \"meta\": { \"category\": \"notes\" } }\n```\n\n**Batch (array):**\n```json\n[\n { \"id\": \"doc-1\", \"text\": \"First document\", \"meta\": { \"tag\": \"a\" } },\n { \"id\": \"doc-2\", \"text\": \"Second document\", \"meta\": { \"tag\": \"b\" } }\n]\n```\n\n**Response:**\n```json\n{ \"success\": true, \"embedded\": 2, \"skipped\": 0, \"results\": [...] }\n```\n- `embedded` — records that were re-embedded (text changed or new)\n- `skipped` — records whose text was unchanged (only meta updated, no API call)\n\n---\n\n#### `POST /search`\nSearch for semantically similar records.\n\n**Request:**\n```json\n{\n \"query\": \"search query text\",\n \"k\": 10,\n \"maxDistance\": 0.65,\n \"offset\": 0,\n \"filters\": { \"category\": \"tech\" },\n \"hybrid\": { \"enabled\": true, \"alpha\": 0.25 }\n}\n```\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `query` | string | required | Natural language search query |\n| `k` | number | 10 | Number of results to return (max 200) |\n| `maxDistance` | number | none | Filter out results with distance > this value |\n| `offset` | number | 0 | Skip first N results (for pagination) |\n| `filters` | object | none | Key/value metadata filters |\n| `hybrid.enabled` | boolean | false | Enable keyword boosting |\n| `hybrid.alpha` | number | 0.25 | Keyword weight 0–1 (0 = pure vector, 1 = pure keyword) |\n| `hybrid.keyword_k` | number | 2×k | Number of keyword candidates to consider |\n\n**Response:**\n```json\n{\n \"success\": true,\n \"count\": 3,\n \"results\": [\n {\n \"id\": \"doc-1\",\n \"text\": \"Dogs are loyal companions\",\n \"meta\": { \"category\": \"animals\" },\n \"distance\": 0.49,\n \"score\": 0.43,\n \"keyword_score\": 0.12\n }\n ]\n}\n```\n\n**Distance guide:**\n\n| Distance | Meaning |\n|---|---|\n| 0.0 – 0.4 | Very similar — always include |\n| 0.4 – 0.6 | Related — good default range |\n| 0.6 – 0.7 | Somewhat related |\n| 0.7+ | Likely unrelated — filter out |\n\nUse `GET /calibrate?q=your+query` to find the right threshold for your data.\n\n---\n\n#### `POST /delete` 🔐\nDelete a record by ID.\n\n```json\n{ \"id\": \"doc-1\" }\n```\n\n---\n\n#### `GET /get?id=`\nRetrieve a single record by ID.\n\n---\n\n#### `GET /list`\nList record IDs.\n\n| Param | Default | Description |\n|---|---|---|\n| `limit` | 100 | Max records to return (max 1000) |\n| `offset` | 0 | Pagination offset |\n| `prefix` | none | Filter IDs by prefix |\n\n---\n\n#### `POST /upsert_chunked` 🔐\nAutomatically split a long document into overlapping chunks and upsert each one.\n\n```json\n{\n \"id\": \"doc-long\",\n \"text\": \"...very long document...\",\n \"meta\": { \"source\": \"book\" },\n \"chunkSize\": 800,\n \"overlap\": 100\n}\n```\n\nChunks are stored as `doc-long::chunk1`, `doc-long::chunk2`, etc., each with `parent_id`, `chunk_index`, and `chunk_total` in their metadata.\n\n---\n\n#### `GET /export` 🔐\nExport records as JSON (without embeddings — text + meta only).\n\n| Param | Default | Description |\n|---|---|---|\n| `limit` | 200 | Max records |\n| `offset` | 0 | Pagination offset |\n\n---\n\n#### `POST /import` 🔐\nImport records from a previous export (re-embeds all text).\n\n```json\n{ \"records\": [{ \"id\": \"doc-1\", \"text\": \"...\", \"meta\": {} }] }\n```\nOr pass an array directly.\n\n---\n\n### Admin Endpoints\n\n#### `GET /` — API info + stats\n#### `GET /ping` — Health check\n#### `GET /stats` — Detailed storage stats\n#### `GET /auth` — Auth mode status\n\n#### `GET /test` 🔐\nRuns a smoke test: inserts 5 demo records, performs 3 searches, returns results and timing.\n\n#### `GET /seed?n=100` 🔐\nSeeds N synthetic records across 4 categories (tech, science, business, lifestyle). Useful for testing and threshold calibration. Max 1000 per call.\n\n#### `GET /calibrate?q=`\nAnalyzes the top 20 results for a query and suggests distance thresholds:\n- **tight** — top 3 results only\n- **balanced** — top 10 (recommended default)\n- **loose** — all retrieved\n\n#### `GET /validate`\nRuns self-checks: auth, stats, list, and search. Add `?write=yes` for full write tests (requires `ALLOW_WRITE_TESTS=1` env var).\n\n#### `POST /reindex` 🔐\nDrops and recreates the DiskANN vector index with current settings. Use after changing `INDEX_*` env vars.\n\n#### `POST /clear?confirm=yes` 🔐\nDeletes ALL records from the main table and FTS index. Cannot be undone.\n\n> 🔐 = requires `Authorization: Bearer ` header when `ADMIN_TOKEN` env var is set.\n\n---\n\n## Using as a TypeScript Module\n\nImport `vectordb.ts` directly into any other val:\n\n```typescript\nimport * as db from \"https://esm.town/v/kamenxrider/slimarmor/vectordb.ts\";\n\n// First-time setup (idempotent — safe to call on every cold start)\nawait db.setup();\n\n// Upsert a record\nawait db.upsert(\"note-1\", \"Buy milk and eggs\", { tag: \"shopping\" });\n\n// Batch upsert\nawait db.upsertMany([\n { id: \"note-2\", text: \"Call the dentist\", meta: { tag: \"health\" } },\n { id: \"note-3\", text: \"Finish the quarterly report\", meta: { tag: \"work\" } },\n]);\n\n// Search\nconst results = await db.search(\"grocery shopping\", 5, 0.65);\n// → [{ id, text, meta, distance }, ...]\n\n// Search with filters\nconst workItems = await db.search(\"meetings\", 10, undefined, {\n filters: { tag: \"work\" }\n});\n\n// Hybrid search (vector + keyword)\nconst hybrid = await db.search(\"dentist appointment\", 5, undefined, {\n hybrid: { enabled: true, alpha: 0.3 }\n});\n\n// Get a single record\nconst record = await db.get(\"note-1\");\n\n// Delete\nawait db.remove(\"note-1\");\n\n// Stats\nconst { count, estimated_storage_mb } = await db.stats();\n\n// Chunk and upsert a long document\nawait db.upsertChunked(\"essay-1\", longText, { source: \"blog\" }, {\n size: 800,\n overlap: 100,\n});\n```\n\n> **Note:** When importing as a module, SlimArmor uses the **same SQLite database as your val** — so it shares data with any HTTP API you've deployed. Set your embedding env vars in the importing val.\n\n---\n\n## Environment Variables Reference\n\n### Embedding Provider\n\n| Variable | Required | Default | Description |\n|---|---|---|---|\n| `NEBIUS_API_KEY` | If using Nebius | — | Nebius API key |\n| `OPENAI_API_KEY` | If using OpenAI | — | OpenAI API key |\n| `OPENROUTER_API_KEY` | If using OpenRouter | — | OpenRouter API key |\n| `EMBEDDING_PROVIDER` | No | `nebius` | Preset: `nebius`, `openai`, `openrouter` |\n| `EMBEDDING_API_URL` | No | (from preset) | Custom API URL |\n| `EMBEDDING_API_KEY` | No | — | Generic key fallback |\n| `EMBEDDING_MODEL` | No | (from preset) | Override model name |\n| `EMBEDDING_DIM` | No | (from preset) | Override dimensions, or `auto` to detect |\n\n### Auth\n\n| Variable | Required | Default | Description |\n|---|---|---|---|\n| `ADMIN_TOKEN` | No | — | Bearer token for write operations. If unset, all ops are open. |\n\n### Index Tuning (Advanced)\n\n| Variable | Default | Description |\n|---|---|---|\n| `INDEX_METRIC` | `cosine` | Distance metric: `cosine` or `l2` |\n| `INDEX_MAX_NEIGHBORS` | `64` | DiskANN graph degree (8–256). Lower = smaller index, less accurate |\n| `INDEX_COMPRESS_NEIGHBORS` | `float8` | Neighbor compression: `float8`, `float16`, `floatb16`, `float32`, `float1bit`, `none` |\n| `INDEX_ALPHA` | `1.2` | Graph density (≥1). Lower = faster/sparser |\n| `INDEX_SEARCH_L` | `200` | Query-time search effort. Higher = more accurate, slower |\n| `INDEX_INSERT_L` | `70` | Insert-time graph quality. Higher = better index, slower inserts |\n\n> After changing any `INDEX_*` variable, call `POST /reindex` to apply the new settings.\n\n### Validation\n\n| Variable | Default | Description |\n|---|---|---|\n| `ALLOW_WRITE_TESTS` | `0` | Set to `1` to enable `/validate?write=yes` |\n| `ALLOW_WRITE_TESTS_NOAUTH` | `0` | Set to `1` to allow write tests without auth header |\n\n---\n\n## Capacity & Performance\n\nBenchmarked with Nebius `Qwen/Qwen3-Embedding-8B` (4096 dims):\n\n| Metric | Value |\n|---|---|\n| Storage per record | ~22 KB |\n| Max records per 1 GB | ~47,500 |\n| Embedding latency | ~460ms per batch |\n| Search latency | <100ms |\n| Batch size | Up to 96 texts per API call |\n\nStorage scales directly with dimensions — lower-dim models fit far more records:\n\n| Model | Dims | Storage/record | Records/GB |\n|---|---|---|---|\n| Nebius Qwen3-8B | 4096 | ~22 KB | ~47,500 |\n| OpenAI text-embedding-3-large | 3072 | ~18 KB | ~65,000 |\n| OpenAI text-embedding-3-small | 1536 | ~10 KB | ~130,000 |\n| nomic-embed-text / bge-small | 768 | ~7 KB | ~260,000 |\n\n---\n\n## Browser CLI\n\nVisit `/ui` on your endpoint for a full terminal-style interface.\n\n**Commands:**\n\n| Command | Description |\n|---|---|\n| `help` | List all commands |\n| `search \"query\" [-k 10] [--max 0.65] [--mode balanced] [--filter key=value] [--hybrid] [--alpha 0.25]` | Semantic search |\n| `upsert \"text\"` | Add or update a record |\n| `get ` | Show a single record |\n| `del ` | Delete a record |\n| `list [--limit 20] [--prefix id-prefix]` | List record IDs |\n| `stats` | Storage and index info |\n| `calibrate \"query\"` | Suggest distance thresholds |\n| `seed ` | Seed N synthetic records |\n| `clear --yes` | Delete all records |\n| `reindex` | Rebuild vector index |\n| `auth ` | Set admin token for session |\n| `logout` | Clear auth token |\n| `ping` | Health check |\n| `cls` | Clear screen |\n\n**Search modes (shorthand for `--max`):**\n- `--mode tight` → maxDistance 0.5\n- `--mode balanced` → maxDistance 0.64\n- `--mode loose` → maxDistance 0.7\n\n---\n\n## Architecture\n\n```\n┌──────────────────────────────────────────────────────┐\n│ Browser CLI (ui.ts) │\n│ Terminal UI at /ui — no JS framework │\n└─────────────────────────┬────────────────────────────┘\n │\n ▼\n┌──────────────────────────────────────────────────────┐\n│ HTTP API (api.ts) │\n│ /upsert /search /delete /stats /seed ... │\n│ Bearer token auth · Input validation · Error handling│\n└──────────────┬───────────────────────────────────────┘\n │\n ▼\n┌──────────────────────────────────────────────────────┐\n│ Vector DB Core (vectordb.ts) │\n│ setup · upsert · upsertMany · search · remove │\n│ Content-hash dedup · Batch embed · FTS sync │\n└───────────┬──────────────────────┬───────────────────┘\n │ │\n ▼ ▼\n┌───────────────────┐ ┌───────────────────────────────┐\n│ Embedding API │ │ Val Town SQLite (libSQL) │\n│ (OpenAI-compat) │ │ F32_BLOB · DiskANN index │\n│ Nebius / OpenAI │ │ FTS5 · vector_top_k │\n│ OpenRouter / etc │ │ vector_distance_cos │\n└───────────────────┘ └───────────────────────────────┘\n```\n\n### Database Schema\n\n```sql\n-- Main records table\nCREATE TABLE vectordb (\n id TEXT PRIMARY KEY,\n text TEXT NOT NULL,\n text_hash TEXT NOT NULL, -- SHA-256 for change detection\n embedding F32_BLOB(4096), -- Vector column (dim varies by provider)\n meta_json TEXT, -- Optional JSON metadata\n updated_at INTEGER NOT NULL -- Unix ms timestamp\n);\n\n-- DiskANN vector index\nCREATE INDEX vectordb_embedding_idx\nON vectordb (libsql_vector_idx(embedding,\n 'metric=cosine',\n 'max_neighbors=64',\n 'compress_neighbors=float8'\n));\n\n-- Full-text search index for hybrid search\nCREATE VIRTUAL TABLE vectordb_fts USING fts5(id, text);\n\n-- Internal metadata (stores resolved embedding dim, etc.)\nCREATE TABLE vectordb_meta (key TEXT PRIMARY KEY, value TEXT);\n```\n\n---\n\n## Troubleshooting\n\n**`Embedding API error 401`**\n→ Your API key is missing or expired. Check the relevant env var (`NEBIUS_API_KEY`, `OPENAI_API_KEY`, etc.).\n\n**`Embedding dim mismatch`**\n→ Your `EMBEDDING_DIM` env var doesn't match what the model returns. Either remove `EMBEDDING_DIM` (use preset default) or set `EMBEDDING_DIM=auto`.\n\n**`vector index(insert): failed to insert shadow row`**\n→ The DiskANN index is out of sync (usually after a raw SQL `DELETE`). Call `POST /reindex` to rebuild it.\n\n**Search returns irrelevant results**\n→ Lower `maxDistance` (try `0.5` instead of `0.7`). Use `/calibrate?q=...` to find the right value for your data.\n\n**Search returns no results**\n→ Check `GET /stats` to confirm data exists. Try raising `maxDistance`, or simplify your query.\n\n**Slow inserts**\n→ Each batch of up to 96 records makes one embedding API call (~460ms). For large imports, use `POST /import` with large batches rather than individual `/upsert` calls.\n\n**Switched providers, search broken**\n→ Embeddings are model-specific. Export data, `POST /clear?confirm=yes`, update env vars, then `POST /import`.\n\n---\n\n## Files\n\n| File | Description |\n|---|---|\n| `api.ts` | HTTP API — the main entry point (deploy this as your val's HTTP handler) |\n| `vectordb.ts` | Core library — import this directly in other vals |\n| `ui.ts` | Browser CLI — terminal interface served at `/ui` |\n| `README.md` | This document |\n| `GUIDE.md` | Beginner step-by-step guide with 6 real-world use cases |\n| `RAG_CHATBOT.md` | 📖 **Full tutorial: build a RAG chatbot** using SlimArmor + Val Town's free OpenAI proxy |\n| `CHANGES.md` | Changelog |\n| `HANDOVER.md` | Technical deep-dive for developers |\n\n---\n\n## License\n\nMIT — fork freely.\n",[],[212],{"_104":220,"_97":221,"_162":117,"_164":-5,"_165":147,"_166":222,"_168":169,"_106":223,"_101":224,"_172":225,"_174":226,"_176":22,"_177":-5,"_178":227,"_100":228},"4f8c074a-15d4-11f1-8b72-42dde27851f2","api.ts",81,["D",1772413703692],14,"http","/**\n * Vector DB HTTP API (v4 - With Browser CLI)\n * \n * UI:\n * - GET /ui → Browser CLI interface\n * \n * Core API:\n * - POST /upsert → Upsert record {id, text, meta?} [AUTH]\n * - POST /search → Search {query, k?, maxDistance?, offset?, filters?, hybrid?}\n * - POST /delete → Delete {id} [AUTH]\n * - GET /get?id=... → Get single record\n * - GET /list?limit=... → List record IDs (supports offset, prefix)\n * - POST /upsert_chunked → Chunk + upsert {id, text, meta?, chunkSize?, overlap?} [AUTH]\n * - GET /export → Export records [AUTH]\n * - POST /import → Import records [AUTH]\n * \n * Admin/Testing:\n * - GET / → Info + endpoints\n * - GET /ping → Health check\n * - GET /stats → Detailed stats (count, avg text len, storage)\n * - GET /test → Smoke test with demo data [AUTH]\n * - GET /seed?n=100 → Seed N synthetic records for scale testing [AUTH]\n * - GET /calibrate?q=... → Suggest distance thresholds for a query\n * - GET /validate → Self-checks (optional write tests)\n * - POST /reindex → Recreate index with optimized settings [AUTH]\n * - POST /clear → Delete ALL records (dangerous!) [AUTH]\n * - GET /auth → Auth requirement status\n * \n * Authentication:\n * - Set ADMIN_TOKEN env var to enable auth\n * - Pass token via Authorization: Bearer header\n * - Write operations require auth when ADMIN_TOKEN is set\n */\n\nimport * as db from \"./vectordb.ts\";\nimport sqlite from \"https://esm.town/v/std/sqlite/main.ts\";\nimport { serveUI, serveUIJS } from \"./ui.ts\";\n\n// ============================================================\n// AUTH HELPER\n// ============================================================\n\nconst ADMIN_TOKEN = Deno.env.get(\"ADMIN_TOKEN\");\n\nfunction isAuthorized(req: Request): boolean {\n // If no ADMIN_TOKEN set, allow all (open mode)\n if (!ADMIN_TOKEN) return true;\n \n const authHeader = req.headers.get(\"Authorization\");\n if (!authHeader) return false;\n \n const [type, token] = authHeader.split(\" \");\n return type === \"Bearer\" && token === ADMIN_TOKEN;\n}\n\nfunction unauthorized(): Response {\n return Response.json(\n { error: \"Unauthorized. Provide valid Authorization: Bearer header.\" },\n { status: 401 }\n );\n}\n\nfunction isWriteTestRequested(url: URL): boolean {\n const value = (url.searchParams.get(\"write\") || \"\").toLowerCase();\n return value === \"1\" || value === \"true\" || value === \"yes\";\n}\n\nfunction isWriteTestsEnabled(): boolean {\n return (Deno.env.get(\"ALLOW_WRITE_TESTS\") || \"\").toLowerCase() === \"1\";\n}\n\nfunction isWriteTestsNoAuthEnabled(): boolean {\n return (Deno.env.get(\"ALLOW_WRITE_TESTS_NOAUTH\") || \"\").toLowerCase() === \"1\";\n}\n\nfunction normalizeFilters(input: unknown): Record | undefined {\n if (!input || typeof input !== \"object\" || Array.isArray(input)) return undefined;\n const out: Record = {};\n\n for (const [key, value] of Object.entries(input as Record)) {\n if (value === null || typeof value === \"string\" || typeof value === \"number\" || typeof value === \"boolean\") {\n out[key] = value;\n } else if (Array.isArray(value)) {\n const cleaned = value.filter(\n (v) => typeof v === \"string\" || typeof v === \"number\" || typeof v === \"boolean\"\n ) as Array;\n if (cleaned.length > 0) out[key] = cleaned;\n }\n }\n\n return Object.keys(out).length > 0 ? out : undefined;\n}\n\n// ============================================================\n// SYNTHETIC DATA FOR SEEDING\n// ============================================================\n\nconst SEED_TEMPLATES = {\n tech: [\n \"Learn how to build REST APIs with Node.js and Express framework.\",\n \"Docker containers simplify deployment and improve consistency across environments.\",\n \"TypeScript adds static typing to JavaScript for better developer experience.\",\n \"GraphQL provides a flexible query language for APIs.\",\n \"Kubernetes orchestrates containerized applications at scale.\",\n \"React hooks enable state management in functional components.\",\n \"PostgreSQL is a powerful open-source relational database.\",\n \"Redis provides fast in-memory caching for web applications.\",\n \"WebSockets enable real-time bidirectional communication.\",\n \"Git version control tracks changes and enables collaboration.\",\n ],\n science: [\n \"Photosynthesis converts sunlight into chemical energy in plants.\",\n \"Black holes are regions where gravity is so strong light cannot escape.\",\n \"DNA contains the genetic instructions for all living organisms.\",\n \"Quantum entanglement links particles across vast distances.\",\n \"Climate change affects global weather patterns and ecosystems.\",\n \"Neurons transmit electrical signals throughout the nervous system.\",\n \"Plate tectonics explains the movement of Earth's continental plates.\",\n \"Enzymes catalyze biochemical reactions in living cells.\",\n \"The periodic table organizes elements by atomic number and properties.\",\n \"Evolution by natural selection drives species adaptation over time.\",\n ],\n business: [\n \"Market segmentation helps target specific customer groups effectively.\",\n \"Cash flow management is critical for business sustainability.\",\n \"Brand positioning differentiates products in competitive markets.\",\n \"Supply chain optimization reduces costs and improves efficiency.\",\n \"Customer retention strategies increase lifetime value.\",\n \"Agile methodology enables iterative product development.\",\n \"Risk assessment helps identify potential business threats.\",\n \"Mergers and acquisitions can accelerate company growth.\",\n \"Employee engagement correlates with productivity and retention.\",\n \"Data-driven decision making improves business outcomes.\",\n ],\n lifestyle: [\n \"Regular exercise improves cardiovascular health and mental wellbeing.\",\n \"Mediterranean diet emphasizes olive oil, fish, and fresh vegetables.\",\n \"Mindfulness meditation reduces stress and improves focus.\",\n \"Quality sleep is essential for cognitive function and recovery.\",\n \"Sustainable fashion reduces environmental impact of clothing.\",\n \"Urban gardening brings fresh produce to city dwellers.\",\n \"Remote work offers flexibility but requires discipline.\",\n \"Financial literacy helps people make better money decisions.\",\n \"Travel broadens perspectives and creates lasting memories.\",\n \"Minimalism focuses on owning fewer but more meaningful things.\",\n ],\n};\n\nfunction generateSeedData(n: number): Array<{ id: string; text: string; meta: { category: string } }> {\n const categories = Object.keys(SEED_TEMPLATES) as Array;\n const data: Array<{ id: string; text: string; meta: { category: string } }> = [];\n \n for (let i = 0; i < n; i++) {\n const category = categories[i % categories.length];\n const templates = SEED_TEMPLATES[category];\n const baseText = templates[i % templates.length];\n \n // Add variation to make each record unique\n const variation = ` (Record ${i + 1}, variation ${Math.floor(i / templates.length) + 1})`;\n \n data.push({\n id: `seed-${i + 1}`,\n text: baseText + variation,\n meta: { category, seed_index: i + 1 },\n });\n }\n \n return data;\n}\n\n// ============================================================\n// HANDLER\n// ============================================================\n\nexport default async function handler(req: Request): Promise {\n const url = new URL(req.url);\n const path = url.pathname;\n\n try {\n // ==================== GET /ui ====================\n if (req.method === \"GET\" && path === \"/ui\") {\n return serveUI();\n }\n\n // ==================== GET /ui.js ====================\n if (req.method === \"GET\" && path === \"/ui.js\") {\n return serveUIJS();\n }\n\n // ==================== /cli (no-JS fallback UI) ====================\n if (path === \"/cli\") {\n if (req.method === \"GET\") {\n return serveCLIFallback({\n baseUrl: url.origin,\n method: url.searchParams.get(\"method\"),\n path: url.searchParams.get(\"path\"),\n body: url.searchParams.get(\"body\"),\n token: url.searchParams.get(\"token\"),\n });\n }\n if (req.method === \"POST\") {\n return handleCLIFallback(req);\n }\n }\n\n // Ensure DB is set up for all other routes\n await db.setup();\n\n // ==================== GET /ping ====================\n if (req.method === \"GET\" && path === \"/ping\") {\n return Response.json({ \n status: \"ok\", \n timestamp: new Date().toISOString(),\n });\n }\n\n // ==================== GET /auth ====================\n if (req.method === \"GET\" && path === \"/auth\") {\n return Response.json({\n auth_required: !!ADMIN_TOKEN,\n mode: ADMIN_TOKEN ? \"protected\" : \"open\",\n });\n }\n\n // ==================== GET / ====================\n if (req.method === \"GET\" && path === \"/\") {\n const stats = await db.stats();\n const provider = db.getProviderInfo();\n const indexInfo = db.getIndexInfo();\n return Response.json({\n name: \"SlimArmor - Mini Vector DB\",\n version: \"v4\",\n ui: \"/ui\",\n auth_required: !!ADMIN_TOKEN,\n embedding_provider: provider,\n index_settings: indexInfo,\n stats,\n endpoints: {\n ui: {\n \"GET /ui\": \"Browser CLI interface\",\n },\n core: {\n \"POST /upsert\": \"Upsert {id, text, meta?} [AUTH]\",\n \"POST /search\": \"Search {query, k?, maxDistance?, offset?, filters?, hybrid?}\",\n \"POST /delete\": \"Delete {id} [AUTH]\",\n \"GET /get?id=...\": \"Get single record\",\n \"GET /list?limit=...&offset=...&prefix=...\": \"List record IDs\",\n \"POST /upsert_chunked\": \"Chunk + upsert {id, text, meta?, chunkSize?, overlap?} [AUTH]\",\n \"GET /export?limit=...&offset=...\": \"Export records [AUTH]\",\n \"POST /import\": \"Import records (batch upsert) [AUTH]\",\n },\n admin: {\n \"GET /ping\": \"Health check\",\n \"GET /auth\": \"Auth requirement status\",\n \"GET /stats\": \"Detailed storage stats\",\n \"GET /test\": \"Smoke test with 5 demo records [AUTH]\",\n \"GET /seed?n=100\": \"Seed N synthetic records [AUTH]\",\n \"GET /calibrate?q=...\": \"Suggest distance thresholds\",\n \"POST /reindex\": \"Recreate optimized index [AUTH]\",\n \"POST /clear\": \"Delete ALL records [AUTH]\",\n },\n },\n });\n }\n\n // ==================== GET /stats ====================\n if (req.method === \"GET\" && path === \"/stats\") {\n const basicStats = await db.stats();\n const provider = db.getProviderInfo();\n const indexInfo = db.getIndexInfo();\n \n // Get additional stats from DB\n const textStats = await sqlite.execute(`\n SELECT \n COUNT(*) as count,\n AVG(LENGTH(text)) as avg_text_length,\n SUM(LENGTH(text)) as total_text_bytes\n FROM vectordb\n `);\n \n const row = textStats.rows[0] as { \n count: number; \n avg_text_length: number | null; \n total_text_bytes: number | null;\n };\n \n const count = Number(row.count);\n const avgTextLength = row.avg_text_length ? Math.round(Number(row.avg_text_length)) : 0;\n const totalTextBytes = row.total_text_bytes ? Number(row.total_text_bytes) : 0;\n const dims = provider.dimensions;\n const embeddingBytes = count * dims * 4; // dims * 4 bytes per float32\n const bytesPerRecord = (dims * 4) + 6000; // keep in sync with vectordb.ts estimate\n\n return Response.json({\n count,\n avg_text_length: avgTextLength,\n storage: {\n text_bytes: totalTextBytes,\n text_mb: Math.round(totalTextBytes / 1024 / 1024 * 100) / 100,\n embedding_bytes: embeddingBytes,\n embedding_mb: Math.round(embeddingBytes / 1024 / 1024 * 100) / 100,\n estimated_total_mb: basicStats.estimated_storage_mb,\n },\n index_settings: indexInfo,\n capacity: {\n current_pct_of_1gb: Math.round((basicStats.estimated_storage_mb / 1024) * 100 * 100) / 100,\n estimated_max_records: Math.floor((1024 * 1024 * 1024) / bytesPerRecord),\n },\n });\n }\n\n // ==================== GET /test ====================\n if (req.method === \"GET\" && path === \"/test\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const results: Record = {};\n const startTime = Date.now();\n\n // Insert demo records\n const demoData = [\n { id: \"demo-1\", text: \"Dogs are loyal companions that love to play fetch.\", meta: { category: \"animals\" } },\n { id: \"demo-2\", text: \"Cats are independent pets that enjoy sunny windowsills.\", meta: { category: \"animals\" } },\n { id: \"demo-3\", text: \"JavaScript is a programming language for web development.\", meta: { category: \"tech\" } },\n { id: \"demo-4\", text: \"Python is widely used for data science and machine learning.\", meta: { category: \"tech\" } },\n { id: \"demo-5\", text: \"Coffee is a popular morning beverage made from roasted beans.\", meta: { category: \"food\" } },\n ];\n\n // Fix #4: use upsertMany for batch embedding efficiency\n results[\"1_upsert\"] = { status: \"inserting 5 demo records...\" };\n await db.upsertMany(demoData);\n results[\"1_upsert\"] = { status: \"done\", count: demoData.length };\n\n // Search test 1: animals (with distance scores)\n const search1 = await db.search(\"furry pets that make sounds\", 3);\n results[\"2_search_pets\"] = { \n query: \"furry pets that make sounds\", \n results: search1.map(r => ({ id: r.id, text: r.text, distance: r.distance.toFixed(4) }))\n };\n\n // Search test 2: programming (with distance scores)\n const search2 = await db.search(\"coding and software development\", 3);\n results[\"3_search_coding\"] = { \n query: \"coding and software development\", \n results: search2.map(r => ({ id: r.id, text: r.text, distance: r.distance.toFixed(4) }))\n };\n\n // Search test 3: with maxDistance filter (only very relevant)\n const search3 = await db.search(\"furry pets\", 5, 0.5);\n results[\"4_search_filtered\"] = {\n query: \"furry pets (maxDistance=0.5)\",\n results: search3.map(r => ({ id: r.id, text: r.text, distance: r.distance.toFixed(4) }))\n };\n\n // Stats\n const stats = await db.stats();\n results[\"5_stats\"] = stats;\n\n // Timing\n const elapsed = Date.now() - startTime;\n results[\"6_timing\"] = { elapsed_ms: elapsed };\n\n return Response.json({\n success: true,\n message: \"Smoke test complete!\",\n results,\n });\n }\n\n // ==================== GET /seed?n=100 ====================\n if (req.method === \"GET\" && path === \"/seed\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const n = Math.min(parseInt(url.searchParams.get(\"n\") ?? \"100\", 10), 1000); // Max 1000 per call\n const startTime = Date.now();\n \n const seedData = generateSeedData(n);\n\n // Fix #4: use upsertMany so all records are embedded in batches, not one-by-one\n const batchResult = await db.upsertMany(seedData);\n const embedded = batchResult.embedded;\n const skipped = batchResult.skipped;\n\n const elapsed = Date.now() - startTime;\n const stats = await db.stats();\n \n return Response.json({\n success: true,\n seeded: n,\n embedded,\n skipped,\n elapsed_ms: elapsed,\n avg_ms_per_record: Math.round(elapsed / n),\n stats,\n });\n }\n\n // ==================== GET /calibrate?q=... ====================\n if (req.method === \"GET\" && path === \"/calibrate\") {\n const query = url.searchParams.get(\"q\");\n \n if (!query) {\n return Response.json({ error: \"Missing q query param\" }, { status: 400 });\n }\n \n // Get top 20 results\n const results = await db.search(query, 20);\n \n if (results.length === 0) {\n return Response.json({ \n error: \"No results found. Add some data first.\",\n suggestion: \"Try GET /seed?n=100 to add test data\",\n }, { status: 404 });\n }\n \n const distances = results.map(r => r.distance);\n \n // Calculate thresholds\n const tight = distances[Math.min(2, distances.length - 1)]; // 3rd result\n const balanced = distances[Math.min(9, distances.length - 1)]; // 10th result\n const loose = distances[distances.length - 1]; // last result\n \n return Response.json({\n query,\n total_results: results.length,\n distance_distribution: {\n min: distances[0].toFixed(4),\n max: distances[distances.length - 1].toFixed(4),\n median: distances[Math.floor(distances.length / 2)].toFixed(4),\n },\n suggested_thresholds: {\n tight: { value: Number(tight.toFixed(4)), description: \"Top 3 only, very strict\" },\n balanced: { value: Number(balanced.toFixed(4)), description: \"Top 10, good default\" },\n loose: { value: Number(loose.toFixed(4)), description: \"Include all retrieved\" },\n },\n top_5_results: results.slice(0, 5).map(r => ({\n id: r.id,\n text: r.text.slice(0, 80) + (r.text.length > 80 ? \"...\" : \"\"),\n distance: r.distance.toFixed(4),\n })),\n });\n }\n\n // ==================== GET /validate ====================\n if (req.method === \"GET\" && path === \"/validate\") {\n const start = Date.now();\n const tests: Array<{ name: string; ok: boolean; skipped?: boolean; details?: unknown }> = [];\n const add = (name: string, ok: boolean, details?: unknown, skipped = false) => {\n tests.push({ name, ok, skipped, details });\n };\n\n const provider = db.getProviderInfo();\n const authInfo = db.getEmbeddingAuthInfo();\n\n add(\"auth\", true, { auth_required: !!ADMIN_TOKEN });\n\n try {\n const stats = await db.stats();\n add(\"stats\", true, stats);\n } catch (e) {\n add(\"stats\", false, { error: e instanceof Error ? e.message : String(e) });\n }\n\n try {\n const ids = await db.listIds(1);\n add(\"list\", true, { count: ids.length });\n } catch (e) {\n add(\"list\", false, { error: e instanceof Error ? e.message : String(e) });\n }\n\n if (!authInfo.has_key) {\n add(\n \"search\",\n false,\n { skipped_reason: `Missing embedding key (${authInfo.required_env})` },\n true\n );\n } else {\n try {\n const results = await db.search(\"validation test\", 3);\n add(\"search\", true, { count: results.length });\n } catch (e) {\n add(\"search\", false, { error: e instanceof Error ? e.message : String(e) });\n }\n }\n\n const writeRequested = isWriteTestRequested(url);\n const writeEnabled = writeRequested && isWriteTestsEnabled();\n const writeNoAuth = writeEnabled && isWriteTestsNoAuthEnabled();\n const writeAuthorized = !ADMIN_TOKEN || isAuthorized(req) || writeNoAuth;\n\n if (!writeRequested) {\n add(\"write_tests\", false, { skipped_reason: \"write tests not requested (add ?write=yes)\" }, true);\n } else if (!writeEnabled) {\n add(\"write_tests\", false, { skipped_reason: \"ALLOW_WRITE_TESTS is not enabled\" }, true);\n } else if (!writeAuthorized) {\n add(\"write_tests\", false, { skipped_reason: \"not authorized for write tests\" }, true);\n } else if (!authInfo.has_key) {\n add(\"write_tests\", false, { skipped_reason: `Missing embedding key (${authInfo.required_env})` }, true);\n } else {\n const testId = `__slimarmor_validate__${Date.now()}`;\n const testText = `SlimArmor validation record ${new Date().toISOString()}`;\n const testMeta = { category: \"validation\", created_at: new Date().toISOString() };\n\n try {\n const upsertRes = await db.upsert(testId, testText, testMeta);\n add(\"write_upsert\", true, { embedded: upsertRes.embedded, id: testId });\n\n const filterRes = await db.search(\n testText,\n 5,\n undefined,\n { filters: { category: \"validation\" } }\n );\n const filterFound = filterRes.some(r => r.id === testId);\n add(\"filter_search\", filterFound, { found: filterFound });\n\n const hybridRes = await db.search(\n testText,\n 5,\n undefined,\n { hybrid: { enabled: true, alpha: 0.25 } }\n );\n const hybridFound = hybridRes.some(r => r.id === testId);\n add(\"hybrid_search\", hybridFound, { found: hybridFound });\n\n const delRes = await db.remove(testId);\n add(\"write_delete\", delRes.deleted, { deleted: delRes.deleted });\n } catch (e) {\n add(\"write_tests\", false, { error: e instanceof Error ? e.message : String(e) });\n }\n }\n\n const failed = tests.filter(t => !t.ok && !t.skipped).length;\n return Response.json({\n ok: failed === 0,\n duration_ms: Date.now() - start,\n provider,\n embedding_key_present: authInfo.has_key,\n write_tests_requested: writeRequested,\n write_tests_enabled: writeEnabled,\n tests,\n });\n }\n\n // ==================== POST /upsert ====================\n if (req.method === \"POST\" && path === \"/upsert\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const body = await req.json();\n\n const normalizeRecord = (input: unknown): db.UpsertInput | null => {\n if (!input || typeof input !== \"object\") return null;\n const rec = input as { id?: unknown; text?: unknown; meta?: unknown };\n if (!rec.id || !rec.text) return null;\n const meta = rec.meta && typeof rec.meta === \"object\" && !Array.isArray(rec.meta)\n ? (rec.meta as Record)\n : undefined;\n return { id: String(rec.id), text: String(rec.text), meta };\n };\n\n let records: db.UpsertInput[] | null = null;\n\n if (Array.isArray(body)) {\n records = body.map(normalizeRecord).filter(Boolean) as db.UpsertInput[];\n } else if (body && typeof body === \"object\" && Array.isArray((body as { records?: unknown }).records)) {\n records = (body as { records: unknown[] }).records.map(normalizeRecord).filter(Boolean) as db.UpsertInput[];\n }\n\n if (records) {\n if (records.length === 0) {\n return Response.json({ error: \"No valid records provided\" }, { status: 400 });\n }\n const result = await db.upsertMany(records);\n return Response.json({ success: true, ...result });\n }\n\n const single = normalizeRecord(body);\n if (!single) {\n return Response.json({ error: \"Missing id or text\" }, { status: 400 });\n }\n\n const result = await db.upsert(single.id, single.text, single.meta);\n return Response.json({ success: true, ...result });\n }\n\n // ==================== POST /upsert_chunked ====================\n if (req.method === \"POST\" && path === \"/upsert_chunked\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const body = await req.json() as { \n id?: string; \n text?: string; \n meta?: Record; \n chunkSize?: number; \n overlap?: number; \n };\n \n if (!body.id || !body.text) {\n return Response.json({ error: \"Missing id or text\" }, { status: 400 });\n }\n\n const options = {\n size: typeof body.chunkSize === \"number\" ? body.chunkSize : undefined,\n overlap: typeof body.overlap === \"number\" ? body.overlap : undefined,\n };\n\n const result = await db.upsertChunked(body.id, body.text, body.meta, options);\n return Response.json({ success: true, ...result });\n }\n\n // ==================== POST /search ====================\n if (req.method === \"POST\" && path === \"/search\") {\n const body = await req.json() as { \n query?: string; \n k?: number; \n maxDistance?: number; \n offset?: number;\n filters?: Record;\n hybrid?: { enabled?: boolean; alpha?: number; keyword_k?: number };\n };\n \n if (!body.query) {\n return Response.json({ error: \"Missing query\" }, { status: 400 });\n }\n\n const filters = normalizeFilters(body.filters);\n const hybrid = body.hybrid && typeof body.hybrid === \"object\"\n ? {\n enabled: !!body.hybrid.enabled,\n alpha: typeof body.hybrid.alpha === \"number\" ? body.hybrid.alpha : undefined,\n keyword_k: typeof body.hybrid.keyword_k === \"number\" ? body.hybrid.keyword_k : undefined,\n }\n : undefined;\n\n const kRaw = typeof body.k === \"number\" ? body.k : 10;\n const k = Math.min(Math.max(1, kRaw), 200);\n\n const results = await db.search(\n body.query, \n k, \n body.maxDistance, \n { offset: body.offset ?? 0, filters, hybrid }\n );\n return Response.json({ \n success: true, \n query: body.query, \n k,\n offset: body.offset ?? 0,\n maxDistance: body.maxDistance ?? \"none\",\n count: results.length, \n results \n });\n }\n\n // ==================== POST /delete ====================\n if (req.method === \"POST\" && path === \"/delete\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const body = await req.json() as { id?: string };\n \n if (!body.id) {\n return Response.json({ error: \"Missing id\" }, { status: 400 });\n }\n\n const result = await db.remove(body.id);\n return Response.json({ success: true, ...result });\n }\n\n // ==================== GET /get?id=... ====================\n if (req.method === \"GET\" && path === \"/get\") {\n const id = url.searchParams.get(\"id\");\n \n if (!id) {\n return Response.json({ error: \"Missing id query param\" }, { status: 400 });\n }\n\n const record = await db.get(id);\n if (!record) {\n return Response.json({ error: \"Not found\" }, { status: 404 });\n }\n\n return Response.json({ success: true, record });\n }\n\n // ==================== GET /list?limit=... ====================\n if (req.method === \"GET\" && path === \"/list\") {\n const limitRaw = parseInt(url.searchParams.get(\"limit\") ?? \"100\", 10);\n const limit = Math.min(Math.max(1, limitRaw), 1000);\n const offset = Math.max(0, parseInt(url.searchParams.get(\"offset\") ?? \"0\", 10));\n const prefix = url.searchParams.get(\"prefix\") ?? undefined;\n const ids = await db.listIds(limit, offset, prefix);\n return Response.json({ success: true, count: ids.length, offset, ids });\n }\n\n // ==================== GET /export ====================\n if (req.method === \"GET\" && path === \"/export\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const limitRaw = parseInt(url.searchParams.get(\"limit\") ?? \"200\", 10);\n const limit = Math.min(Math.max(1, limitRaw), 1000);\n const offset = Math.max(0, parseInt(url.searchParams.get(\"offset\") ?? \"0\", 10));\n\n const rows = await sqlite.execute({\n sql: `SELECT id, text, meta_json, updated_at FROM vectordb ORDER BY updated_at DESC LIMIT ? OFFSET ?`,\n args: [limit, offset],\n });\n\n const records = (rows.rows as Array<{ id: string; text: string; meta_json: string | null; updated_at: number }>).map(r => ({\n id: r.id,\n text: r.text,\n meta: r.meta_json ? JSON.parse(r.meta_json) : undefined,\n updated_at: r.updated_at,\n }));\n\n return Response.json({ success: true, count: records.length, offset, records });\n }\n\n // ==================== POST /import ====================\n if (req.method === \"POST\" && path === \"/import\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const body = await req.json();\n const recordsRaw = Array.isArray(body)\n ? body\n : (body && typeof body === \"object\" && Array.isArray((body as { records?: unknown }).records))\n ? (body as { records: unknown[] }).records\n : null;\n\n if (!recordsRaw) {\n return Response.json({ error: \"Missing records array\" }, { status: 400 });\n }\n\n const normalizeRecord = (input: unknown): db.UpsertInput | null => {\n if (!input || typeof input !== \"object\") return null;\n const rec = input as { id?: unknown; text?: unknown; meta?: unknown };\n if (!rec.id || !rec.text) return null;\n const meta = rec.meta && typeof rec.meta === \"object\" && !Array.isArray(rec.meta)\n ? (rec.meta as Record)\n : undefined;\n return { id: String(rec.id), text: String(rec.text), meta };\n };\n\n const records = recordsRaw.map(normalizeRecord).filter(Boolean) as db.UpsertInput[];\n if (records.length === 0) {\n return Response.json({ error: \"No valid records provided\" }, { status: 400 });\n }\n\n const result = await db.upsertMany(records);\n return Response.json({ success: true, ...result });\n }\n\n // ==================== POST /reindex ====================\n if (req.method === \"POST\" && path === \"/reindex\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const result = await db.reindex();\n return Response.json(result);\n }\n\n // ==================== POST /clear ====================\n if (req.method === \"POST\" && path === \"/clear\") {\n // Requires auth\n if (!isAuthorized(req)) return unauthorized();\n\n const confirm = url.searchParams.get(\"confirm\");\n \n if (confirm !== \"yes\") {\n return Response.json({ \n error: \"This will delete ALL records. Add ?confirm=yes to proceed.\",\n warning: \"This action cannot be undone!\",\n }, { status: 400 });\n }\n \n // Fix #3: clear FTS table alongside main table to prevent ghost keyword results\n await sqlite.batch([\n { sql: \"DELETE FROM vectordb\" },\n { sql: \"DELETE FROM vectordb_fts\" },\n ]);\n return Response.json({ success: true, message: \"All records deleted\" });\n }\n\n // ==================== 404 ====================\n return Response.json({ error: \"Not found\" }, { status: 404 });\n\n } catch (e) {\n const msg = e instanceof Error ? e.message : String(e);\n return Response.json({ error: msg }, { status: 500 });\n }\n}\n\n// ============================================================\n// NO-JS FALLBACK CLI\n// ============================================================\n\nfunction escapeHtml(value: string): string {\n return value\n .replace(/&/g, \"&\")\n .replace(//g, \">\");\n}\n\nfunction renderCLIFallback(params: {\n baseUrl: string;\n method?: string;\n path?: string;\n body?: string;\n token?: string;\n response?: string;\n error?: string;\n}): string {\n const method = params.method ?? \"GET\";\n const path = params.path ?? \"/stats\";\n const body = params.body ?? \"\";\n const token = params.token ?? \"\";\n const response = params.response ? escapeHtml(params.response) : \"\";\n const error = params.error ? escapeHtml(params.error) : \"\";\n\n return `\n\n\n \n \n SlimArmor CLI (No JS)\n \n\n\n

🛡️ SlimArmor CLI (No JS)

Fallback interface when the interactive CLI is blocked. Submits requests server-side. Write tests require auth unless ALLOW_WRITE_TESTS_NOAUTH=1 is temporarily set.

\n\n

\n \n \n \n \n \n

\n\n \n\n ${error ? `

${error}

` : \"\"}\n ${response ? `

${response}

` : \"\"}\n\n`;\n}\n\nfunction serveCLIFallback(params: {\n baseUrl: string;\n method?: string | null;\n path?: string | null;\n body?: string | null;\n token?: string | null;\n response?: string;\n error?: string;\n}): Response {\n const html = renderCLIFallback({\n baseUrl: params.baseUrl,\n method: params.method ?? undefined,\n path: params.path ?? undefined,\n body: params.body ?? undefined,\n token: params.token ?? undefined,\n response: params.response,\n error: params.error,\n });\n return new Response(html, {\n headers: { \"Content-Type\": \"text/html; charset=utf-8\" },\n });\n}\n\nasync function handleCLIFallback(req: Request): Promise {\n const url = new URL(req.url);\n const form = await req.formData();\n const method = String(form.get(\"method\") || \"GET\").toUpperCase();\n const path = String(form.get(\"path\") || \"/stats\");\n const token = String(form.get(\"token\") || \"\");\n const body = String(form.get(\"body\") || \"\");\n\n const targetUrl = new URL(path, url.origin).toString();\n const headers: Record = { Accept: \"application/json\" };\n if (token) headers.Authorization = `Bearer ${token}`;\n\n try {\n const init: RequestInit = { method, headers };\n if (method === \"POST\") {\n headers[\"Content-Type\"] = \"application/json\";\n init.body = body || \"{}\";\n }\n const res = await fetch(targetUrl, init);\n const text = await res.text();\n\n return serveCLIFallback({\n baseUrl: url.origin,\n method,\n path,\n body,\n token,\n response: text,\n });\n } catch (err) {\n return serveCLIFallback({\n baseUrl: url.origin,\n method,\n path,\n body,\n token,\n error: err instanceof Error ? err.message : String(err),\n });\n }\n}\n",[],[221],{"_104":230,"_97":231,"_162":232,"_164":-5,"_165":147,"_166":233,"_168":169,"_106":234,"_101":235,"_172":236,"_174":237,"_176":22,"_177":-5,"_178":238,"_100":239},"3559cc28-0098-11f1-9fea-42dde27851f2","ui.ts","c8ff15e8-ffe7-11f0-9667-42dde27851f2",63,["D",1770078915492],3,"script","/**\n * SlimArmor Browser CLI\n * A terminal-style interface for the vector database\n */\n\nconst CLI_HTML = `\n\n\n \n \n SlimArmor CLI\n \n\n\n \n \n

\n \n

\n ❯\n \n

\n\n \n\n`;\n\nconst CLI_JS = `\n const API_BASE = window.location.origin;\n let adminToken = sessionStorage.getItem('adminToken') || '';\n let serverAuthRequired = null;\n let commandHistory = JSON.parse(sessionStorage.getItem('commandHistory') || '[]');\n let historyIndex = -1;\n \n const output = document.getElementById('output');\n const input = document.getElementById('input');\n const authStatus = document.getElementById('auth-status');\n\n if (input) {\n input.focus();\n document.addEventListener('click', () => input.focus());\n }\n \n // Threshold presets\n const THRESHOLDS = {\n tight: 0.5,\n balanced: 0.64,\n loose: 0.7\n };\n \n // Update auth status display\n function updateAuthStatus() {\n if (adminToken) {\n authStatus.textContent = '🔓 authenticated';\n authStatus.className = 'status authed';\n return;\n }\n \n if (serverAuthRequired === false) {\n authStatus.textContent = '🟢 open mode';\n authStatus.className = 'status authed';\n } else {\n authStatus.textContent = '🔒 read-only';\n authStatus.className = 'status';\n }\n }\n\n async function refreshAuthStatus() {\n try {\n const data = await api('GET', '/auth');\n if (typeof data.auth_required === 'boolean') {\n serverAuthRequired = data.auth_required;\n }\n } catch {\n // ignore\n }\n updateAuthStatus();\n }\n \n // Print line to output\n function print(text, className = '') {\n const line = document.createElement('div');\n line.className = 'line ' + className;\n line.innerHTML = text;\n output.appendChild(line);\n output.scrollTop = output.scrollHeight;\n }\n \n // Print raw HTML\n function printHTML(html) {\n const container = document.createElement('div');\n container.innerHTML = html;\n output.appendChild(container);\n output.scrollTop = output.scrollHeight;\n }\n \n // API call helper\n async function api(method, path, body = null) {\n const headers = { 'Content-Type': 'application/json' };\n if (adminToken) {\n headers['Authorization'] = 'Bearer ' + adminToken;\n }\n \n const opts = { method, headers };\n if (body) opts.body = JSON.stringify(body);\n \n const res = await fetch(API_BASE + path, opts);\n return res.json();\n }\n \n // Parse command into tokens (respects quotes)\n function parseCommand(input) {\n const tokens = [];\n let current = '';\n let inQuotes = false;\n let quoteChar = '';\n \n for (let i = 0; i < input.length; i++) {\n const c = input[i];\n \n if ((c === '\"' || c === \"'\") && !inQuotes) {\n inQuotes = true;\n quoteChar = c;\n } else if (c === quoteChar && inQuotes) {\n inQuotes = false;\n quoteChar = '';\n } else if (c === ' ' && !inQuotes) {\n if (current) tokens.push(current);\n current = '';\n } else {\n current += c;\n }\n }\n if (current) tokens.push(current);\n \n return tokens;\n }\n \n // Parse flags from tokens\n function parseFlags(tokens) {\n const args = [];\n const flags = {};\n \n for (let i = 0; i < tokens.length; i++) {\n const t = tokens[i];\n if (t.startsWith('--')) {\n const key = t.slice(2);\n const next = tokens[i + 1];\n if (next && !next.startsWith('-')) {\n flags[key] = next;\n i++;\n } else {\n flags[key] = true;\n }\n } else if (t.startsWith('-') && t.length === 2) {\n const key = t.slice(1);\n const next = tokens[i + 1];\n if (next && !next.startsWith('-')) {\n flags[key] = next;\n i++;\n } else {\n flags[key] = true;\n }\n } else {\n args.push(t);\n }\n }\n \n return { args, flags };\n }\n\n function coerceValue(value) {\n if (value === 'true') return true;\n if (value === 'false') return false;\n const num = Number(value);\n if (!Number.isNaN(num) && value.trim() !== '') return num;\n return value;\n }\n \n // Make ID clickable\n function clickableId(id) {\n return '' + escapeHtml(id) + '';\n }\n \n // Escape HTML\n function escapeHtml(str) {\n return String(str).replace(/&/g, '&').replace(//g, '>');\n }\n \n // Commands\n const commands = {\n async help() {\n printHTML(\\`\n

📖 Core Commands

search \"query\"Semantic search

-k 10Max results (default: 10)

--mode balancedtight|balanced|loose

--max 0.64Max distance threshold

--offset 0Pagination offset

--filter key=valueMeta filters (comma-separated)

--hybridKeyword + vector boost

--alpha 0.25Hybrid weight (0..1)

get <id>Show record details

listList all record IDs

--limit 20Max records to show

--prefix seed-Filter by ID prefix

--offset 0Pagination offset

statsStorage & index stats

calibrate \"query\"Suggest thresholds

✏️ Write Commands (requires auth)

upsert <id> \"text\"Create/update record

--meta '{\"tag\":\"notes\"}'Optional metadata

del <id>Delete record

seed 100Seed synthetic data

clear --yesDelete ALL records

reindexRecreate vector index

🔐 Auth

auth <token>Set admin token

logoutClear auth token

💡 Tips

↑/↓Navigate command history

Click IDsAuto-fill get command

\n \\`);\n },\n \n async stats() {\n print('Fetching stats...', 'info');\n const data = await api('GET', '/stats');\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n printHTML(\\`\n

Records\\${data.count.toLocaleString()}

Avg Text Length\\${data.avg_text_length} chars

Text Storage\\${data.storage.text_mb} MB

Embedding Storage\\${data.storage.embedding_mb} MB

Total Estimated\\${data.storage.estimated_total_mb} MB

Capacity Used\\${data.capacity.current_pct_of_1gb}% of 1GB

Est. Max Records~\\${data.capacity.estimated_max_records.toLocaleString()}

Index Metric\\${data.index_settings.metric}

Index Compression\\${data.index_settings.compress_neighbors}

\n \\`);\n },\n \n async search(args, flags) {\n if (args.length < 1) {\n print('Usage: search \"query\" [-k 10] [--mode balanced] [--max 0.64] [--offset 0] [--filter key=value] [--hybrid --alpha 0.25]', 'error');\n return;\n }\n \n const query = args[0];\n const k = parseInt(flags.k || '10');\n const offset = parseInt(flags.offset || '0');\n \n // Determine maxDistance from mode or explicit --max\n let maxDistance;\n if (flags.max) {\n maxDistance = parseFloat(flags.max);\n } else if (flags.mode && THRESHOLDS[flags.mode]) {\n maxDistance = THRESHOLDS[flags.mode];\n }\n\n // Parse filters\n let filters;\n if (flags.filter) {\n filters = {};\n const pairs = String(flags.filter).split(',');\n pairs.forEach(p => {\n const [rawKey, rawValue] = p.split('=');\n if (!rawKey || rawValue === undefined) return;\n const key = rawKey.trim();\n let value = rawValue.trim();\n\n // Array values via pipe: key=a|b|c\n if (value.includes('|')) {\n const vals = value.split('|').map(v => v.trim()).filter(Boolean).map(coerceValue);\n if (vals.length > 0) filters[key] = vals;\n return;\n }\n\n filters[key] = coerceValue(value);\n });\n }\n\n const hybrid = !!flags.hybrid || typeof flags.alpha !== 'undefined';\n const alpha = typeof flags.alpha !== 'undefined' ? parseFloat(flags.alpha) : undefined;\n \n print(\\`Searching for \"\\${query}\"...\\`, 'info');\n \n const data = await api('POST', '/search', { \n query, \n k, \n maxDistance, \n offset,\n filters,\n hybrid: hybrid ? { enabled: true, alpha } : undefined,\n });\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n print(\\`Found \\${data.count} results (k=\\${k}, offset=\\${offset}, maxDistance=\\${maxDistance || 'none'})\\`, 'success');\n \n if (data.results.length === 0) {\n print('No matching results.', 'info');\n return;\n }\n \n let html = '';\n data.results.forEach((r, i) => {\n const text = r.text.length > 60 ? r.text.slice(0, 60) + '...' : r.text;\n html += \\`\n \n \n \n \n \\`;\n });\n html += '

#	ID	Distance	Text
\\${i + 1}	\\${clickableId(r.id)}	\\${r.distance.toFixed(4)}	\\${escapeHtml(text)}

';\n printHTML(html);\n },\n \n async get(args) {\n if (args.length < 1) {\n print('Usage: get ', 'error');\n return;\n }\n \n const id = args[0];\n const data = await api('GET', '/get?id=' + encodeURIComponent(id));\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n const r = data.record;\n printHTML(\\`\n

ID: \\${escapeHtml(r.id)}

Text: \\${escapeHtml(r.text)}

\n \\${r.meta ? '

Meta: ' + escapeHtml(JSON.stringify(r.meta)) + '

' : ''}\n

Updated: \\${new Date(r.updated_at).toLocaleString()}

\n \\`);\n print(\\`💡 del \\${id} | upsert \\${id} \"new text\"\\`, 'info');\n },\n \n async list(args, flags) {\n const limit = parseInt(flags.limit || '20');\n const offset = parseInt(flags.offset || '0');\n const prefix = flags.prefix ? String(flags.prefix) : '';\n const query = new URLSearchParams({ limit, offset });\n if (prefix) query.set('prefix', prefix);\n const data = await api('GET', '/list?' + query.toString());\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n let ids = data.ids;\n \n print(\\`Found \\${ids.length} record(s)\\${flags.prefix ? ' matching prefix \"' + flags.prefix + '\"' : ''}\\`, 'success');\n \n if (ids.length === 0) return;\n \n let html = '';\n ids.forEach((id, i) => {\n html += \\`\n \n \n \n \\`;\n });\n html += '

#	ID	Actions
\\${i + 1}	\\${clickableId(id)}	del

';\n printHTML(html);\n },\n \n async calibrate(args) {\n if (args.length < 1) {\n print('Usage: calibrate \"query\"', 'error');\n return;\n }\n \n const query = args[0];\n print(\\`Calibrating for \"\\${query}\"...\\`, 'info');\n \n const data = await api('GET', '/calibrate?q=' + encodeURIComponent(query));\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n printHTML(\\`\n

Query\"\\${escapeHtml(data.query)}\"

Results Found\\${data.total_results}

Distance Min\\${data.distance_distribution.min}

Distance Median\\${data.distance_distribution.median}

Distance Max\\${data.distance_distribution.max}

Suggested Thresholds

🎯 Tight (top 3)\\${data.suggested_thresholds.tight.value}

⚖️ Balanced (top 10)\\${data.suggested_thresholds.balanced.value}

📦 Loose (all)\\${data.suggested_thresholds.loose.value}

\n \\`);\n \n print(\\`💡 Try: search \"\\${query}\" --mode tight\\`, 'info');\n },\n \n async upsert(args, flags) {\n if (!adminToken) {\n print('❌ Authentication required. Use: auth ', 'error');\n return;\n }\n \n if (args.length < 2) {\n print('Usage: upsert \"text...\" [--meta \\'{\"key\":\"value\"}\\']', 'error');\n return;\n }\n \n const id = args[0];\n const text = args.slice(1).join(' ');\n let meta;\n \n if (flags.meta) {\n try {\n meta = JSON.parse(flags.meta);\n } catch {\n print('Invalid --meta JSON. Example: --meta \\'{\"tag\":\"notes\"}\\'', 'error');\n return;\n }\n }\n \n print(\\`Upserting \"\\${id}\"...\\`, 'info');\n \n const data = await api('POST', '/upsert', { id, text, meta });\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n print(\\`✅ \\${data.embedded ? 'Embedded and saved' : 'Updated (text unchanged)'}: \\${id}\\`, 'success');\n },\n \n async del(args) {\n if (!adminToken) {\n print('❌ Authentication required. Use: auth ', 'error');\n return;\n }\n \n if (args.length < 1) {\n print('Usage: del ', 'error');\n return;\n }\n \n const id = args[0];\n const data = await api('POST', '/delete', { id });\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n print(\\`✅ Deleted: \\${id} (deleted: \\${data.deleted})\\`, 'success');\n },\n \n async seed(args) {\n if (!adminToken) {\n print('❌ Authentication required. Use: auth ', 'error');\n return;\n }\n \n const n = parseInt(args[0] || '100');\n print(\\`Seeding \\${n} records (this may take a while)...\\`, 'info');\n \n const data = await api('GET', '/seed?n=' + n);\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n printHTML(\\`\n

Seeded\\${data.seeded}

Embedded\\${data.embedded}

Skipped\\${data.skipped}

Time\\${(data.elapsed_ms / 1000).toFixed(1)}s

Avg per record\\${data.avg_ms_per_record}ms

\n \\`);\n },\n \n async clear(args, flags) {\n if (!adminToken) {\n print('❌ Authentication required. Use: auth ', 'error');\n return;\n }\n \n if (!flags.yes) {\n print('⚠️ This will DELETE ALL records! Use: clear --yes', 'error');\n return;\n }\n \n print('Clearing database...', 'info');\n const data = await api('POST', '/clear?confirm=yes');\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n print('✅ All records deleted', 'success');\n },\n \n async reindex() {\n if (!adminToken) {\n print('❌ Authentication required. Use: auth ', 'error');\n return;\n }\n \n print('Recreating index...', 'info');\n const data = await api('POST', '/reindex');\n \n if (data.error) {\n print('Error: ' + data.error, 'error');\n return;\n }\n \n print('✅ Index recreated', 'success');\n },\n \n auth(args) {\n if (args.length < 1) {\n print('Usage: auth ', 'error');\n return;\n }\n \n adminToken = args[0];\n sessionStorage.setItem('adminToken', adminToken);\n updateAuthStatus();\n print('✅ Token saved (stored in session)', 'success');\n },\n \n logout() {\n adminToken = '';\n sessionStorage.removeItem('adminToken');\n updateAuthStatus();\n print('✅ Logged out', 'success');\n },\n \n async ping() {\n const data = await api('GET', '/ping');\n print('🏓 Pong! Server time: ' + data.timestamp, 'success');\n },\n \n cls() {\n output.innerHTML = '';\n },\n \n clear_output() {\n output.innerHTML = '';\n }\n };\n \n // Aliases\n commands.delete = commands.del;\n commands['?'] = commands.help;\n commands.ls = commands.list;\n \n // Run a command\n async function runCommand(cmdStr) {\n // Add to history\n if (cmdStr && cmdStr !== commandHistory[0]) {\n commandHistory.unshift(cmdStr);\n if (commandHistory.length > 50) commandHistory.pop();\n sessionStorage.setItem('commandHistory', JSON.stringify(commandHistory));\n }\n historyIndex = -1;\n \n print(cmdStr, 'command');\n \n const tokens = parseCommand(cmdStr);\n if (tokens.length === 0) return;\n \n const cmd = tokens[0].toLowerCase();\n const { args, flags } = parseFlags(tokens.slice(1));\n \n if (commands[cmd]) {\n try {\n await commands[cmd](args, flags);\n } catch (e) {\n print('Error: ' + e.message, 'error');\n }\n } else {\n print(\\`Unknown command: \\${cmd}. Type 'help' for available commands.\\`, 'error');\n }\n }\n \n // Make runCommand available globally for click handlers\n window.runCommand = runCommand;\n \n // Input handling\n input.addEventListener('keydown', async (e) => {\n if (e.key === 'Enter') {\n const cmd = input.value.trim();\n input.value = '';\n if (cmd) await runCommand(cmd);\n } else if (e.key === 'ArrowUp') {\n e.preventDefault();\n if (historyIndex < commandHistory.length - 1) {\n historyIndex++;\n input.value = commandHistory[historyIndex];\n }\n } else if (e.key === 'ArrowDown') {\n e.preventDefault();\n if (historyIndex > 0) {\n historyIndex--;\n input.value = commandHistory[historyIndex];\n } else {\n historyIndex = -1;\n input.value = '';\n }\n }\n });\n \n // Initial state\n updateAuthStatus();\n refreshAuthStatus();\n \n // Welcome message\n print('🛡️ SlimArmor Vector DB CLI', 'highlight');\n print(\"Type 'help' for available commands\", 'info');\n print('', '');\n `;\n\nexport function serveUI(): Response {\n return new Response(CLI_HTML, {\n headers: { \"Content-Type\": \"text/html; charset=utf-8\" },\n });\n}\n\nexport function serveUIJS(): Response {\n return new Response(CLI_JS, {\n headers: {\n \"Content-Type\": \"application/javascript; charset=utf-8\",\n \"Cache-Control\": \"no-store\",\n },\n });\n}\n",[],[231],{"_104":241,"_97":242,"_162":243,"_164":-5,"_165":147,"_166":244,"_168":169,"_106":245,"_101":171,"_172":236,"_174":246,"_176":22,"_177":-5,"_178":247,"_100":248},"440ce362-15d4-11f1-8b72-42dde27851f2","vectordb.ts","85b83e64-ffe1-11f0-a01c-42dde27851f2",78,["D",1772413684396],"/**\n * Mini Vector DB for Val Town (v3 - Multi-Provider)\n * \n * Supports any OpenAI-compatible embedding provider:\n * - Nebius (default)\n * - OpenAI\n * - OpenRouter\n * - Azure OpenAI\n * - Local (Ollama, LM Studio)\n * - Any OpenAI-compatible API\n * \n * Configure via environment variables:\n * - EMBEDDING_API_URL (default: Nebius)\n * - EMBEDDING_API_KEY (required)\n * - EMBEDDING_MODEL (default: Qwen/Qwen3-Embedding-8B)\n * - EMBEDDING_DIM (default: 4096)\n */\n\nimport sqlite from \"https://esm.town/v/std/sqlite/main.ts\";\n\n// ============================================================\n// CONFIGURATION (via environment variables)\n// ============================================================\n\n// Provider presets\nconst PROVIDERS = {\n nebius: {\n url: \"https://api.tokenfactory.nebius.com/v1/embeddings\",\n keyEnv: \"NEBIUS_API_KEY\",\n defaultModel: \"Qwen/Qwen3-Embedding-8B\",\n defaultDim: 4096,\n },\n openai: {\n url: \"https://api.openai.com/v1/embeddings\",\n keyEnv: \"OPENAI_API_KEY\",\n defaultModel: \"text-embedding-3-small\",\n defaultDim: 1536,\n },\n openrouter: {\n url: \"https://openrouter.ai/api/v1/embeddings\",\n keyEnv: \"OPENROUTER_API_KEY\",\n defaultModel: \"openai/text-embedding-3-small\",\n defaultDim: 1536,\n },\n} as const;\n\ntype ProviderName = keyof typeof PROVIDERS;\n\n// Table/index names\nconst TABLE_NAME = \"vectordb\";\nconst INDEX_NAME = \"vectordb_embedding_idx\";\nconst FTS_TABLE = \"vectordb_fts\";\nconst META_TABLE = \"vectordb_meta\";\n\nlet RESOLVED_DIM: number | null = null;\nlet FTS_AVAILABLE: boolean | null = null;\n\n// Fix #2: Module-level setup guard — skips redundant SQLite round-trips on warm requests\nlet _setupDone = false;\n\n// Get configuration from environment\nfunction getConfig() {\n // Check for explicit provider setting\n const providerName = (Deno.env.get(\"EMBEDDING_PROVIDER\") || \"nebius\") as ProviderName;\n const provider = PROVIDERS[providerName] || PROVIDERS.nebius;\n \n // Allow overrides via env vars\n const url = Deno.env.get(\"EMBEDDING_API_URL\") || provider.url;\n const model = Deno.env.get(\"EMBEDDING_MODEL\") || provider.defaultModel;\n const rawDim = Deno.env.get(\"EMBEDDING_DIM\");\n const dimAuto = (rawDim || \"\").toLowerCase() === \"auto\";\n const parsedDim = parseInt(rawDim || String(provider.defaultDim), 10);\n const dim = RESOLVED_DIM ?? (Number.isFinite(parsedDim) && parsedDim > 0 ? parsedDim : provider.defaultDim);\n \n // Get API key (check provider-specific first, then generic)\n const apiKey = Deno.env.get(provider.keyEnv) || Deno.env.get(\"EMBEDDING_API_KEY\");\n \n return { url, model, dim, dimAuto, apiKey, providerName };\n}\n\nfunction getIndexConfig() {\n const metricRaw = (Deno.env.get(\"INDEX_METRIC\") || \"\").toLowerCase();\n const metric = metricRaw === \"l2\" ? \"l2\" : \"cosine\";\n const maxNeighborsRaw = parseInt(Deno.env.get(\"INDEX_MAX_NEIGHBORS\") || \"64\", 10);\n const maxNeighbors = Number.isFinite(maxNeighborsRaw) && maxNeighborsRaw >= 8 && maxNeighborsRaw <= 256\n ? maxNeighborsRaw\n : 64;\n const compressRaw = (Deno.env.get(\"INDEX_COMPRESS_NEIGHBORS\") || \"\").toLowerCase();\n const compressNeighbors = (\n compressRaw === \"none\" ||\n compressRaw === \"float1bit\" ||\n compressRaw === \"float8\" ||\n compressRaw === \"float16\" ||\n compressRaw === \"floatb16\" ||\n compressRaw === \"float32\"\n )\n ? compressRaw\n : \"float8\";\n\n const alphaRaw = Deno.env.get(\"INDEX_ALPHA\");\n const alphaParsed = alphaRaw ? parseFloat(alphaRaw) : 1.2;\n const alpha = Number.isFinite(alphaParsed) && alphaParsed >= 1 ? alphaParsed : 1.2;\n\n const searchLRaw = Deno.env.get(\"INDEX_SEARCH_L\");\n const searchLParsed = searchLRaw ? parseInt(searchLRaw, 10) : 200;\n const searchL = Number.isFinite(searchLParsed) && searchLParsed > 0 ? searchLParsed : 200;\n\n const insertLRaw = Deno.env.get(\"INDEX_INSERT_L\");\n const insertLParsed = insertLRaw ? parseInt(insertLRaw, 10) : 70;\n const insertL = Number.isFinite(insertLParsed) && insertLParsed > 0 ? insertLParsed : 70;\n\n return {\n metric,\n maxNeighbors,\n compressNeighbors,\n alpha,\n searchL,\n insertL,\n hasAlpha: !!alphaRaw,\n hasSearchL: !!searchLRaw,\n hasInsertL: !!insertLRaw,\n };\n}\n\n// ============================================================\n// TYPES\n// ============================================================\n\nexport interface VectorRecord {\n id: string;\n text: string;\n meta?: Record;\n updated_at: number;\n}\n\nexport interface SearchResult {\n id: string;\n text: string;\n meta?: Record;\n distance: number;\n score?: number;\n keyword_score?: number;\n}\n\nexport interface Stats {\n count: number;\n estimated_storage_mb: number;\n}\n\nexport interface ProviderInfo {\n provider: string;\n url: string;\n model: string;\n dimensions: number;\n dim_auto: boolean;\n}\n\nexport interface IndexInfo {\n metric: string;\n max_neighbors: number;\n compress_neighbors: string;\n alpha: number;\n search_l: number;\n insert_l: number;\n}\n\nexport interface EmbeddingAuthInfo {\n required_env: string;\n has_key: boolean;\n}\n\nexport type FilterValue = string | number | boolean | null | Array;\n\nexport interface SearchOptions {\n filters?: Record;\n offset?: number;\n hybrid?: {\n enabled?: boolean;\n alpha?: number; // 0..1, keyword weight\n keyword_k?: number;\n };\n}\n\nexport interface UpsertInput {\n id: string;\n text: string;\n meta?: Record;\n}\n\nexport interface ChunkOptions {\n size?: number; // approx chars per chunk\n overlap?: number; // approx chars overlap\n}\n\n// ============================================================\n// GUARDRAILS\n// ============================================================\n\nfunction assertDim(vec: number[], expected: number): void {\n if (vec.length !== expected) {\n throw new Error(`Embedding dim mismatch: got ${vec.length}, expected ${expected}`);\n }\n}\n\nasync function hashText(text: string): Promise {\n const encoder = new TextEncoder();\n const data = encoder.encode(text);\n const hashBuffer = await crypto.subtle.digest(\"SHA-256\", data);\n const hashArray = Array.from(new Uint8Array(hashBuffer));\n return hashArray.map(b => b.toString(16).padStart(2, \"0\")).join(\"\");\n}\n\n// ============================================================\n// EMBEDDING CLIENT (OpenAI-Compatible)\n// ============================================================\n\ninterface EmbeddingResponse {\n data: Array<{ embedding: number[]; index: number }>;\n model: string;\n usage?: { prompt_tokens: number; total_tokens: number };\n}\n\nfunction getExpectedDim(config: ReturnType): number | null {\n if (RESOLVED_DIM) return RESOLVED_DIM;\n if (config.dimAuto) return null;\n return config.dim;\n}\n\nasync function embedManyRaw(inputs: string[], config = getConfig()): Promise {\n if (!config.apiKey) {\n throw new Error(`Missing API key. Set ${PROVIDERS[config.providerName]?.keyEnv || \"EMBEDDING_API_KEY\"} env var`);\n }\n\n const controller = new AbortController();\n const timeoutId = setTimeout(() => controller.abort(), 30000);\n\n try {\n const res = await fetch(config.url, {\n method: \"POST\",\n headers: {\n \"Authorization\": `Bearer ${config.apiKey}`,\n \"Content-Type\": \"application/json\",\n },\n body: JSON.stringify({ \n model: config.model, \n input: inputs,\n }),\n signal: controller.signal,\n });\n\n if (!res.ok) {\n const errText = await res.text();\n throw new Error(`Embedding API error ${res.status}: ${errText}`);\n }\n\n const json = (await res.json()) as EmbeddingResponse;\n const embeddings: Array = new Array(inputs.length);\n for (const item of json.data) {\n embeddings[item.index] = item.embedding;\n }\n\n if (embeddings.some((e) => !e)) {\n throw new Error(\"Embedding API returned incomplete data\");\n }\n\n return embeddings as number[][];\n } finally {\n clearTimeout(timeoutId);\n }\n}\n\nasync function embedMany(inputs: string[]): Promise {\n const config = getConfig();\n const expectedDim = getExpectedDim(config);\n const batchSize = 96;\n const result: number[][] = [];\n\n for (let i = 0; i < inputs.length; i += batchSize) {\n const batch = inputs.slice(i, i + batchSize);\n const embeddings = await embedManyRaw(batch, config);\n if (expectedDim !== null) {\n for (const emb of embeddings) assertDim(emb, expectedDim);\n }\n result.push(...embeddings);\n }\n\n return result;\n}\n\nasync function embed(input: string): Promise {\n const [embedding] = await embedMany([input]);\n return embedding;\n}\n\n// ============================================================\n// VECTOR DB API\n// ============================================================\n\nasync function ensureFtsTable(): Promise {\n if (FTS_AVAILABLE !== null) return;\n try {\n await sqlite.execute(`CREATE VIRTUAL TABLE IF NOT EXISTS ${FTS_TABLE} USING fts5(id, text)`);\n FTS_AVAILABLE = true;\n } catch {\n FTS_AVAILABLE = false;\n }\n}\n\nasync function resolveDimForSetup(): Promise {\n const config = getConfig();\n if (!config.dimAuto) return config.dim;\n if (RESOLVED_DIM) return RESOLVED_DIM;\n\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${META_TABLE} (\n key TEXT PRIMARY KEY,\n value TEXT\n )\n `);\n\n const existing = await sqlite.execute({\n sql: `SELECT value FROM ${META_TABLE} WHERE key = ?`,\n args: [\"dim\"],\n });\n\n const stored = (existing.rows?.[0] as { value?: string } | undefined)?.value;\n const parsed = stored ? parseInt(stored, 10) : NaN;\n if (Number.isFinite(parsed) && parsed > 0) {\n RESOLVED_DIM = parsed;\n return parsed;\n }\n\n // Try to read dimension from existing table schema (if present)\n const tableInfo = await sqlite.execute(`PRAGMA table_info(${TABLE_NAME})`);\n const embeddingCol = (tableInfo.rows as Array<{ name: string; type: string }>).find(r => r.name === \"embedding\");\n if (embeddingCol && typeof embeddingCol.type === \"string\") {\n const match = embeddingCol.type.match(/F32_BLOB\$(\\d+)\$/i);\n if (match) {\n const detected = parseInt(match[1], 10);\n if (Number.isFinite(detected) && detected > 0) {\n RESOLVED_DIM = detected;\n await sqlite.execute({\n sql: `INSERT OR REPLACE INTO ${META_TABLE} (key, value) VALUES (?, ?)`,\n args: [\"dim\", String(RESOLVED_DIM)],\n });\n return RESOLVED_DIM;\n }\n }\n }\n\n const [embedding] = await embedManyRaw([\"dimension probe\"], config);\n RESOLVED_DIM = embedding.length;\n\n await sqlite.execute({\n sql: `INSERT OR REPLACE INTO ${META_TABLE} (key, value) VALUES (?, ?)`,\n args: [\"dim\", String(RESOLVED_DIM)],\n });\n\n return RESOLVED_DIM;\n}\n\n// Fix #5: syncFtsUpsert uses a single batch instead of 2 sequential queries\nasync function syncFtsUpsert(id: string, text: string): Promise {\n if (FTS_AVAILABLE === null) await ensureFtsTable();\n if (!FTS_AVAILABLE) return;\n try {\n await sqlite.batch([\n { sql: `DELETE FROM ${FTS_TABLE} WHERE id = ?`, args: [id] },\n { sql: `INSERT INTO ${FTS_TABLE} (id, text) VALUES (?, ?)`, args: [id, text] },\n ]);\n } catch {\n // ignore FTS failures to keep core functionality working\n }\n}\n\nasync function syncFtsDelete(id: string): Promise {\n if (FTS_AVAILABLE === null) await ensureFtsTable();\n if (!FTS_AVAILABLE) return;\n try {\n await sqlite.execute({\n sql: `DELETE FROM ${FTS_TABLE} WHERE id = ?`,\n args: [id],\n });\n } catch {\n // ignore FTS failures to keep core functionality working\n }\n}\n\n/**\n * Get current provider configuration\n */\nexport function getProviderInfo(): ProviderInfo {\n const config = getConfig();\n return {\n provider: config.providerName,\n url: config.url,\n model: config.model,\n dimensions: config.dim,\n dim_auto: config.dimAuto,\n };\n}\n\nexport function getIndexInfo(): IndexInfo {\n const cfg = getIndexConfig();\n return {\n metric: cfg.metric,\n max_neighbors: cfg.maxNeighbors,\n compress_neighbors: cfg.compressNeighbors,\n alpha: cfg.alpha,\n search_l: cfg.searchL,\n insert_l: cfg.insertL,\n };\n}\n\nexport function getEmbeddingAuthInfo(): EmbeddingAuthInfo {\n const config = getConfig();\n const requiredEnv = PROVIDERS[config.providerName]?.keyEnv || \"EMBEDDING_API_KEY\";\n return {\n required_env: requiredEnv,\n has_key: !!config.apiKey,\n };\n}\n\n/**\n * Setup: Create table and optimized index\n * Automatically uses the configured embedding dimension.\n * Fix #2: Skips all SQLite work on subsequent warm calls via _setupDone guard.\n */\nexport async function setup(): Promise {\n if (_setupDone) return;\n\n const indexConfig = getIndexConfig();\n const dim = await resolveDimForSetup();\n\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${META_TABLE} (\n key TEXT PRIMARY KEY,\n value TEXT\n )\n `);\n \n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLE_NAME} (\n id TEXT PRIMARY KEY,\n text TEXT NOT NULL,\n text_hash TEXT NOT NULL,\n embedding F32_BLOB(${dim}),\n meta_json TEXT,\n updated_at INTEGER NOT NULL\n )\n `);\n\n await ensureFtsTable();\n\n try {\n const idxArgs = [\n `metric=${indexConfig.metric}`,\n `max_neighbors=${indexConfig.maxNeighbors}`,\n ];\n if (indexConfig.compressNeighbors !== \"none\") {\n idxArgs.push(`compress_neighbors=${indexConfig.compressNeighbors}`);\n }\n if (indexConfig.hasAlpha) idxArgs.push(`alpha=${indexConfig.alpha}`);\n if (indexConfig.hasSearchL) idxArgs.push(`search_l=${indexConfig.searchL}`);\n if (indexConfig.hasInsertL) idxArgs.push(`insert_l=${indexConfig.insertL}`);\n const idxArgsSql = idxArgs.map((s) => `'${s}'`).join(\", \");\n\n await sqlite.execute(`\n CREATE INDEX ${INDEX_NAME} \n ON ${TABLE_NAME} (libsql_vector_idx(embedding, ${idxArgsSql}))\n `);\n } catch (e) {\n const msg = e instanceof Error ? e.message : String(e);\n if (!msg.includes(\"already exists\")) {\n throw e;\n }\n }\n\n _setupDone = true;\n}\n\n/**\n * Recreate index with optimized settings\n */\nexport async function reindex(): Promise<{ success: boolean; message: string }> {\n try {\n const indexConfig = getIndexConfig();\n await sqlite.execute(`DROP INDEX IF EXISTS ${INDEX_NAME}`);\n\n const idxArgs = [\n `metric=${indexConfig.metric}`,\n `max_neighbors=${indexConfig.maxNeighbors}`,\n ];\n if (indexConfig.compressNeighbors !== \"none\") {\n idxArgs.push(`compress_neighbors=${indexConfig.compressNeighbors}`);\n }\n if (indexConfig.hasAlpha) idxArgs.push(`alpha=${indexConfig.alpha}`);\n if (indexConfig.hasSearchL) idxArgs.push(`search_l=${indexConfig.searchL}`);\n if (indexConfig.hasInsertL) idxArgs.push(`insert_l=${indexConfig.insertL}`);\n const idxArgsSql = idxArgs.map((s) => `'${s}'`).join(\", \");\n\n await sqlite.execute(`\n CREATE INDEX ${INDEX_NAME} \n ON ${TABLE_NAME} (libsql_vector_idx(embedding, ${idxArgsSql}))\n `);\n return { success: true, message: \"Index recreated with optimized settings\" };\n } catch (e) {\n const msg = e instanceof Error ? e.message : String(e);\n return { success: false, message: msg };\n }\n}\n\n/**\n * Upsert: Insert or update a record\n */\nexport async function upsert(\n id: string, \n text: string, \n meta?: Record\n): Promise<{ embedded: boolean }> {\n const newHash = await hashText(text);\n const metaJson = meta ? JSON.stringify(meta) : null;\n const now = Date.now();\n\n const existing = await sqlite.execute({\n sql: `SELECT text_hash FROM ${TABLE_NAME} WHERE id = ?`,\n args: [id],\n });\n\n const existingHash = (existing.rows?.[0] as { text_hash: string } | undefined)?.text_hash;\n\n if (existingHash === newHash) {\n await sqlite.execute({\n sql: `UPDATE ${TABLE_NAME} SET meta_json = ?, updated_at = ? WHERE id = ?`,\n args: [metaJson, now, id],\n });\n return { embedded: false };\n }\n\n const embedding = await embed(text);\n\n await sqlite.execute({\n sql: `\n INSERT INTO ${TABLE_NAME} (id, text, text_hash, embedding, meta_json, updated_at)\n VALUES (?, ?, ?, vector32(?), ?, ?)\n ON CONFLICT(id) DO UPDATE SET\n text = excluded.text,\n text_hash = excluded.text_hash,\n embedding = excluded.embedding,\n meta_json = excluded.meta_json,\n updated_at = excluded.updated_at\n `,\n args: [id, text, newHash, JSON.stringify(embedding), metaJson, now],\n });\n\n await syncFtsUpsert(id, text);\n\n return { embedded: true };\n}\n\n/**\n * Batch upsert: Insert or update many records\n * Fix #1: Uses sqlite.batch() instead of per-record loops for both embed and skip paths.\n * Fix #5: FTS syncs are batched alongside main upserts in one round-trip.\n */\nexport async function upsertMany(\n records: UpsertInput[]\n): Promise<{ embedded: number; skipped: number; results: Array<{ id: string; embedded: boolean }> }> {\n if (records.length === 0) return { embedded: 0, skipped: 0, results: [] };\n\n const ids = records.map((r) => r.id);\n const placeholders = ids.map(() => \"?\").join(\",\");\n const existing = await sqlite.execute({\n sql: `SELECT id, text_hash FROM ${TABLE_NAME} WHERE id IN (${placeholders})`,\n args: ids,\n });\n\n const existingMap = new Map();\n for (const row of existing.rows as Array<{ id: string; text_hash: string }>) {\n existingMap.set(row.id, row.text_hash);\n }\n\n const hashes = await Promise.all(records.map((r) => hashText(r.text)));\n const toEmbed: Array<{ record: UpsertInput; hash: string }> = [];\n const toSkip: Array<{ record: UpsertInput; hash: string }> = [];\n\n records.forEach((record, idx) => {\n const newHash = hashes[idx];\n const existingHash = existingMap.get(record.id);\n if (existingHash === newHash) {\n toSkip.push({ record, hash: newHash });\n } else {\n toEmbed.push({ record, hash: newHash });\n }\n });\n\n const results: Array<{ id: string; embedded: boolean }> = [];\n const now = Date.now();\n\n // Fix #1a: Batch all meta-only updates in one round-trip\n if (toSkip.length > 0) {\n const skipStatements = toSkip.map(({ record }) => ({\n sql: `UPDATE ${TABLE_NAME} SET meta_json = ?, updated_at = ? WHERE id = ?`,\n args: [record.meta ? JSON.stringify(record.meta) : null, now, record.id],\n }));\n await sqlite.batch(skipStatements);\n for (const { record } of toSkip) {\n results.push({ id: record.id, embedded: false });\n }\n }\n\n // Fix #1b: Batch all embed upserts + FTS syncs in one round-trip\n if (toEmbed.length > 0) {\n const texts = toEmbed.map((item) => item.record.text);\n const embeddings = await embedMany(texts);\n\n const upsertStatements = toEmbed.map(({ record, hash }, i) => ({\n sql: `\n INSERT INTO ${TABLE_NAME} (id, text, text_hash, embedding, meta_json, updated_at)\n VALUES (?, ?, ?, vector32(?), ?, ?)\n ON CONFLICT(id) DO UPDATE SET\n text = excluded.text,\n text_hash = excluded.text_hash,\n embedding = excluded.embedding,\n meta_json = excluded.meta_json,\n updated_at = excluded.updated_at\n `,\n args: [\n record.id,\n record.text,\n hash,\n JSON.stringify(embeddings[i]),\n record.meta ? JSON.stringify(record.meta) : null,\n now,\n ],\n }));\n\n // FTS sync statements batched alongside upserts (Fix #5)\n if (FTS_AVAILABLE === null) await ensureFtsTable();\n const ftsStatements = FTS_AVAILABLE\n ? toEmbed.flatMap(({ record }) => [\n { sql: `DELETE FROM ${FTS_TABLE} WHERE id = ?`, args: [record.id] },\n { sql: `INSERT INTO ${FTS_TABLE} (id, text) VALUES (?, ?)`, args: [record.id, record.text] },\n ])\n : [];\n\n await sqlite.batch([...upsertStatements, ...ftsStatements]);\n\n for (const { record } of toEmbed) {\n results.push({ id: record.id, embedded: true });\n }\n }\n\n return {\n embedded: toEmbed.length,\n skipped: toSkip.length,\n results,\n };\n}\n\n/**\n * Simple chunking helper (approx chars)\n */\nexport function chunkText(text: string, options: ChunkOptions = {}): string[] {\n const size = Math.max(200, options.size ?? 800);\n const overlap = Math.max(0, options.overlap ?? 100);\n if (text.length <= size) return [text];\n\n const chunks: string[] = [];\n let start = 0;\n while (start < text.length) {\n const end = Math.min(text.length, start + size);\n let chunk = text.slice(start, end);\n\n // Try to break on whitespace for nicer chunks\n const lastSpace = chunk.lastIndexOf(\" \");\n if (lastSpace > size * 0.6) {\n chunk = chunk.slice(0, lastSpace);\n }\n\n chunks.push(chunk.trim());\n start += Math.max(chunk.length - overlap, 1);\n }\n\n return chunks.filter((c) => c.length > 0);\n}\n\n/**\n * Chunked upsert: splits text and upserts as id::chunkN\n */\nexport async function upsertChunked(\n id: string,\n text: string,\n meta?: Record,\n options: ChunkOptions = {}\n): Promise<{ chunks: number; embedded: number; skipped: number }> {\n const chunks = chunkText(text, options);\n const records: UpsertInput[] = chunks.map((chunk, i) => ({\n id: `${id}::chunk${i + 1}`,\n text: chunk,\n meta: {\n ...(meta || {}),\n parent_id: id,\n chunk_index: i + 1,\n chunk_total: chunks.length,\n },\n }));\n\n const result = await upsertMany(records);\n return { chunks: chunks.length, embedded: result.embedded, skipped: result.skipped };\n}\n\n/**\n * Search: Semantic search using vector similarity\n */\nfunction buildFilterClause(filters?: Record): { clause: string; args: Array } {\n if (!filters || Object.keys(filters).length === 0) return { clause: \"\", args: [] };\n\n const clauses: string[] = [];\n const args: Array = [];\n\n for (const [key, value] of Object.entries(filters)) {\n const safeKey = key.replace(/[^a-zA-Z0-9_]/g, \"\");\n if (!safeKey || safeKey !== key) {\n throw new Error(`Invalid filter key: ${key}`);\n }\n\n const path = `$.${safeKey}`;\n\n if (Array.isArray(value)) {\n if (value.length === 0) continue;\n const placeholders = value.map(() => \"?\").join(\",\");\n clauses.push(`json_extract(t.meta_json, ?) IN (${placeholders})`);\n args.push(path, ...value);\n } else if (value === null) {\n clauses.push(`json_extract(t.meta_json, ?) IS NULL`);\n args.push(path);\n } else {\n clauses.push(`json_extract(t.meta_json, ?) = ?`);\n args.push(path, value);\n }\n }\n\n if (clauses.length === 0) return { clause: \"\", args: [] };\n return { clause: `WHERE ${clauses.join(\" AND \")}`, args };\n}\n\nasync function getKeywordScores(query: string, limit: number): Promise> {\n if (FTS_AVAILABLE === null) await ensureFtsTable();\n if (!FTS_AVAILABLE) return new Map();\n\n try {\n const res = await sqlite.execute({\n sql: `SELECT id, bm25(${FTS_TABLE}) as score FROM ${FTS_TABLE} WHERE ${FTS_TABLE} MATCH ? ORDER BY score LIMIT ?`,\n args: [query, limit],\n });\n\n const map = new Map();\n for (const row of res.rows as Array<{ id: string; score: number }>) {\n const raw = typeof row.score === \"number\" ? row.score : Number(row.score);\n const clamped = Math.max(0, raw);\n const score = 1 / (1 + clamped);\n map.set(row.id, score);\n }\n return map;\n } catch {\n return new Map();\n }\n}\n\nexport async function search(\n query: string, \n k: number = 10,\n maxDistance?: number,\n options: SearchOptions = {}\n): Promise {\n const offset = Math.max(0, options.offset ?? 0);\n const filters = options.filters;\n const { clause, args: filterArgs } = buildFilterClause(filters);\n\n const queryEmbedding = await embed(query);\n const queryVecJson = JSON.stringify(queryEmbedding);\n const internalK = Math.min((k + offset) * (filters ? 3 : 1), 1000);\n\n const result = await sqlite.execute({\n sql: `\n SELECT \n t.id, \n t.text, \n t.meta_json,\n vector_distance_cos(t.embedding, vector32(?)) as distance\n FROM vector_top_k('${INDEX_NAME}', vector32(?), ?) v\n JOIN ${TABLE_NAME} t ON t.rowid = v.id\n ${clause}\n ORDER BY distance ASC\n `,\n args: [queryVecJson, queryVecJson, internalK, ...filterArgs],\n });\n\n let results = (result.rows as Array<{ \n id: string; \n text: string; \n meta_json: string | null;\n distance: number;\n }>).map(row => ({\n id: row.id,\n text: row.text,\n meta: row.meta_json ? JSON.parse(row.meta_json) : undefined,\n distance: row.distance,\n }));\n\n if (maxDistance !== undefined) {\n results = results.filter(r => r.distance <= maxDistance);\n }\n\n const hybridEnabled = !!options.hybrid?.enabled || typeof options.hybrid?.alpha === \"number\";\n if (hybridEnabled) {\n const alpha = Math.min(1, Math.max(0, options.hybrid?.alpha ?? 0.25));\n const keywordK = Math.max(20, options.hybrid?.keyword_k ?? (k + offset) * 2);\n const keywordScores = await getKeywordScores(query, keywordK);\n\n results = results.map((r) => {\n const vecScore = Math.max(0, 1 - r.distance);\n const kwScore = keywordScores.get(r.id) ?? 0;\n const combined = (1 - alpha) * vecScore + alpha * kwScore;\n return { ...r, score: combined, keyword_score: kwScore };\n });\n\n results.sort((a, b) => (b.score ?? 0) - (a.score ?? 0));\n }\n\n return results.slice(offset, offset + k);\n}\n\n/**\n * Delete: Remove a record by ID\n */\nexport async function remove(id: string): Promise<{ deleted: boolean }> {\n const result = await sqlite.execute({\n sql: `DELETE FROM ${TABLE_NAME} WHERE id = ?`,\n args: [id],\n });\n if ((result.rowsAffected ?? 0) > 0) {\n await syncFtsDelete(id);\n }\n return { deleted: (result.rowsAffected ?? 0) > 0 };\n}\n\n/**\n * Stats: Get record count and storage estimate\n * Fix #6: Uses RESOLVED_DIM when available so auto-dim mode reports correctly.\n */\nexport async function stats(): Promise {\n const config = getConfig();\n const result = await sqlite.execute(`SELECT COUNT(*) as count FROM ${TABLE_NAME}`);\n const count = Number((result.rows[0] as { count: number }).count);\n \n // Fix #6: prefer RESOLVED_DIM over config.dim for accurate estimate in auto-dim mode\n const dim = RESOLVED_DIM ?? config.dim;\n const bytesPerRecord = (dim * 4) + 6000;\n const estimated_storage_mb = Math.round((count * bytesPerRecord) / 1024 / 1024 * 100) / 100;\n\n return { count, estimated_storage_mb };\n}\n\n/**\n * Get a single record by ID\n */\nexport async function get(id: string): Promise {\n const result = await sqlite.execute({\n sql: `SELECT id, text, meta_json, updated_at FROM ${TABLE_NAME} WHERE id = ?`,\n args: [id],\n });\n\n if (!result.rows || result.rows.length === 0) return null;\n\n const row = result.rows[0] as { id: string; text: string; meta_json: string | null; updated_at: number };\n return {\n id: row.id,\n text: row.text,\n meta: row.meta_json ? JSON.parse(row.meta_json) : undefined,\n updated_at: row.updated_at,\n };\n}\n\n/**\n * List all record IDs\n */\nexport async function listIds(limit: number = 100, offset: number = 0, prefix?: string): Promise {\n const args: Array = [];\n let whereClause = \"\";\n if (prefix) {\n whereClause = \"WHERE id LIKE ?\";\n args.push(`${prefix}%`);\n }\n\n args.push(limit, offset);\n\n const result = await sqlite.execute({\n sql: `SELECT id FROM ${TABLE_NAME} ${whereClause} ORDER BY updated_at DESC LIMIT ? OFFSET ?`,\n args,\n });\n return (result.rows as Array<{ id: string }>).map(r => r.id);\n}\n",[],[242],"files",[159,181,191,201,210,219,229,240],"counts",{"_253":33,"_254":255,"_256":33,"_257":258,"_262":263},"pulls","branches",1,"forks","envVars",{"_259":260,"_261":33},"count",4,"emptyCount","connections",{"_259":33,"_264":33},"unfilledCount","contents",[267,270,273,276,279,282,285,288],{"_104":160,"_97":161,"_162":163,"_164":-5,"_165":147,"_166":167,"_168":169,"_106":268,"_101":171,"_172":173,"_174":175,"_176":22,"_177":-5,"_100":269},["D",1772414331632],[161],{"_104":182,"_97":183,"_162":184,"_164":-5,"_165":147,"_166":185,"_168":169,"_106":271,"_101":187,"_172":173,"_174":188,"_176":22,"_177":-5,"_100":272},["D",1772414846689],[183],{"_104":192,"_97":193,"_162":194,"_164":-5,"_165":147,"_166":195,"_168":169,"_106":274,"_101":197,"_172":173,"_174":198,"_176":22,"_177":-5,"_100":275},["D",1772414659182],[193],{"_104":202,"_97":203,"_162":204,"_164":-5,"_165":147,"_166":205,"_168":169,"_106":277,"_101":33,"_172":173,"_174":207,"_176":22,"_177":-5,"_100":278},["D",1772415022810],[203],{"_104":211,"_97":212,"_162":213,"_164":-5,"_165":147,"_166":146,"_168":169,"_106":280,"_101":215,"_172":173,"_174":216,"_176":22,"_177":-5,"_100":281},["D",1772415044284],[212],{"_104":220,"_97":221,"_162":117,"_164":-5,"_165":147,"_166":222,"_168":169,"_106":283,"_101":224,"_172":225,"_174":226,"_176":22,"_177":-5,"_100":284},["D",1772413703692],[221],{"_104":230,"_97":231,"_162":232,"_164":-5,"_165":147,"_166":233,"_168":169,"_106":286,"_101":235,"_172":236,"_174":237,"_176":22,"_177":-5,"_100":287},["D",1770078915492],[231],{"_104":241,"_97":242,"_162":243,"_164":-5,"_165":147,"_166":244,"_168":169,"_106":289,"_101":171,"_172":236,"_174":246,"_176":22,"_177":-5,"_100":290},["D",1772413684396],[242],"upstreamProject","upstreamBranch","emptyEnvVars","isNew","flags",[],"routes/_app.x.$handle.$name.environment-variables",{"_95":96,"_97":98,"_102":299,"_257":304,"_323":22},{"_104":105,"_97":98,"_106":300,"_108":-5,"_109":22,"_110":111,"_112":113,"_114":-5,"_14":115,"_116":117,"_118":22,"_119":-5,"_120":-5,"_121":33,"_16":122,"_123":124,"_125":126,"_127":128,"_129":128,"_130":131,"_132":131,"_133":131,"_134":128,"_135":128,"_23":301,"_139":302,"_144":303,"_156":22},["D",1769996247298],{"_95":96,"_137":138},{"_95":96,"_141":138,"_14":115,"_142":143},{"_101":146,"_97":45,"_104":147,"_148":105,"_149":-5,"_150":-5,"_151":-5,"_152":-5,"_153":-5,"_154":98,"_155":96},[305,312,315,319],{"_306":307,"_106":308,"_309":310,"_112":74,"_311":22},"key","ALLOW_WRITE_TESTS_NOAUTH",["D",1770081668471],"updatedAt",["D",1770081677270],"isEmpty",{"_306":313,"_106":314,"_309":-5,"_112":74,"_311":22},"ALLOW_WRITE_TESTS",["D",1770081633724],{"_306":316,"_106":317,"_309":318,"_112":74,"_311":22},"NEBIUS_API_KEY",["D",1769997118992],["D",1772415276516],{"_306":320,"_106":321,"_309":322,"_112":74,"_311":22},"OPENROUTER_API_KEY",["D",1769996567600],["D",1772415267781],"isTeamOrg","actionData","errors"]

Key	Description
ALLOW_WRITE_TESTS_NOAUTH
ALLOW_WRITE_TESTS
NEBIUS_API_KEY
OPENROUTER_API_KEY

kamenxrider

slimarmor

Environment variables