slimarmor

Below is a noob-friendly, end-to-end manual to build semantic search on Val Town using val-scoped SQLite (backed by Turso) and native vector search (ASSUMED available), with embeddings from OpenRouter.

You’ll end up with:

A table of documents

Automatic chunking + embedding

A vector index

A search endpoint (HTTP val) that returns top matches

This guide is written assuming these SQL features exist in your Val Town SQLite:

F32_BLOB(D) vector column type

vector('[1,2,3]') constructor

libsql_vector_idx(...) index function

vector_top_k(indexName, queryVector, k) ANN search function

(If any of those error in practice, you’ll swap the storage/query parts later—but follow this guide as-is for the assumed path.)

0. What you are building Concept

Semantic search works like this:

Split each document into chunks

Turn each chunk into an embedding (array of floats)

Store those vectors in SQLite

Search:

embed the query

ask SQLite “give me the top-K closest vectors”

return those chunks (and their documents)

Why chunking matters (important)

Embedding “one big doc” often sucks. Chunking:

improves relevance

makes results explainable (you return the exact chunk that matched)

avoids “topic dilution”

1. Prerequisites (do this first) 1.1 Create a Val Town account + a new project/val

Create a new HTTP val (this will be your search API + UI).

Create a new Cron val (this will index docs periodically—optional but nice).

You can name them:

semanticSearch_http

semanticSearch_indexer

1.2 Add environment variables (secrets)

You need an OpenRouter key.

Create an environment variable in Val Town:

OPENROUTER_API_KEY = your key

Optional but recommended:

SITE_URL (for HTTP-Referer)

SITE_NAME (for X-Title)

Never hardcode secrets in source.

2. Database schema (SQLite tables)

Create a “setup” function that you can run once (or run automatically at startup).

We’ll store:

docs: original documents

chunks: chunk text + vector embedding + metadata

optional: chunk_fts for lexical search (hybrid search later)

2.1 Schema SQL import sqlite from "https://esm.town/v/std/sqlite/main.ts";

export async function setupDb() { // Documents table await sqlite.execute(CREATE TABLE IF NOT EXISTS docs ( id TEXT PRIMARY KEY, title TEXT NOT NULL, content TEXT NOT NULL, updated_at INTEGER NOT NULL ););

// Chunks table: each doc becomes multiple chunks // ASSUMPTION: F32_BLOB(D) exists. // Pick a dimension D that matches your embedding model output. // For this guide: we’ll use D=1024 (adjust if your model returns different length). await sqlite.execute(CREATE TABLE IF NOT EXISTS chunks ( id TEXT PRIMARY KEY, doc_id TEXT NOT NULL, chunk_index INTEGER NOT NULL, chunk_text TEXT NOT NULL, embedding F32_BLOB(1024), updated_at INTEGER NOT NULL, FOREIGN KEY (doc_id) REFERENCES docs(id) ););

// Vector index for ANN search // ASSUMPTION: libsql_vector_idx exists. await sqlite.execute(CREATE INDEX IF NOT EXISTS chunks_embedding_idx ON chunks (libsql_vector_idx(embedding, 'metric=cosine')););

// Helpful index for grouping chunks by doc quickly await sqlite.execute(CREATE INDEX IF NOT EXISTS chunks_doc_idx ON chunks (doc_id, chunk_index);); }

2.2 About dimensions (super important)

Your vector column dimension must match the embedding length returned by your model.

You are using:

model: qwen/qwen3-embedding-8b

encoding_format: "float"

So the API returns data[0].embedding as a float array.

You must check how long it is once (log it), then set F32_BLOB().

You’ll do that in the embedding test step below.

3. Embeddings client (OpenRouter) 3.1 Minimal embedding function

This function:

calls OpenRouter embeddings endpoint

returns a float array

supports batch inputs too

type OpenRouterEmbeddingResponse = { data: Array<{ embedding: number[] }>; };

const OPENROUTER_URL = "https://openrouter.ai/api/v1/embeddings";

export async function embedText(input: string | string[]) { const apiKey = Deno.env.get("OPENROUTER_API_KEY"); if (!apiKey) throw new Error("Missing OPENROUTER_API_KEY env var");

const referer = Deno.env.get("SITE_URL") ?? "https://val.town"; const title = Deno.env.get("SITE_NAME") ?? "Val Town Semantic Search";

const res = await fetch(OPENROUTER_URL, { method: "POST", headers: { Authorization: Bearer ${apiKey}, "Content-Type": "application/json", "HTTP-Referer": referer, "X-Title": title, }, body: JSON.stringify({ model: "qwen/qwen3-embedding-8b", input, encoding_format: "float", }), });

if (!res.ok) { const errText = await res.text(); throw new Error(OpenRouter embeddings error ${res.status}: ${errText}); }

const json = (await res.json()) as OpenRouterEmbeddingResponse;

// If input is string -> one embedding // If input is string[] -> batch embeddings, one per item return json.data.map((d) => d.embedding); }

3.2 One-time test: confirm embedding dimension

Make a quick test function and log length:

export async function testEmbeddingDim() { const [emb] = await embedText("hello world"); console.log("embedding length:", emb.length); }

Run it once. Then update your schema F32_BLOB(1024) to the actual length.

4. Chunking (split docs into smaller pieces)

You want chunks that are:

not too large (cost)

not too small (loses context)

Good noob defaults:

chunk size: ~800–1200 characters

overlap: ~150–250 characters

Here’s a simple chunker:

export function chunkText(text: string, chunkSize = 1000, overlap = 200) { const chunks: string[] = []; let i = 0;

while (i < text.length) { const end = Math.min(i + chunkSize, text.length); const chunk = text.slice(i, end).trim(); if (chunk) chunks.push(chunk); if (end === text.length) break; i = end - overlap; if (i < 0) i = 0; }

return chunks; }

5. Storing vectors in SQLite

We need to convert the float array to something SQLite’s vector functions accept.

ASSUMPTION: vector('[...]') accepts a JSON-like string list.

So we’ll create helper toVectorSqlLiteral():

export function toVectorSqlLiteral(embedding: number[]) { // We produce a string like: vector('[0.1,0.2,0.3]') // Keep a reasonable precision (6–8 decimals is fine) const arr = embedding.map((n) => Number.isFinite(n) ? +n.toFixed(8) : 0); return vector('${JSON.stringify(arr)}'); }

If your engine expects a different format later, this is the only function you’ll change.

6. Indexing pipeline (insert docs → create chunks → embed → store) 6.1 Insert or update a document import sqlite from "https://esm.town/v/std/sqlite/main.ts";

export async function upsertDoc(id: string, title: string, content: string) { const updatedAt = Date.now();

await sqlite.execute( INSERT INTO docs (id, title, content, updated_at) VALUES (?, ?, ?, ?) ON CONFLICT(id) DO UPDATE SET title=excluded.title, content=excluded.content, updated_at=excluded.updated_at, [id, title, content, updatedAt] ); }

6.2 Rebuild chunks for one doc

This deletes old chunks and re-creates them.

import { chunkText } from "./chunker.ts"; import { embedText } from "./embeddings.ts"; import { toVectorSqlLiteral } from "./vectorSql.ts"; import sqlite from "https://esm.town/v/std/sqlite/main.ts";

export async function reindexDoc(docId: string) { const docRes = await sqlite.execute( SELECT id, title, content, updated_at FROM docs WHERE id = ?, [docId] );

const doc = docRes.rows?.[0]; if (!doc) throw new Error(Doc not found: ${docId});

const content = String(doc.content); const updatedAt = Number(doc.updated_at);

// Delete old chunks await sqlite.execute(DELETE FROM chunks WHERE doc_id = ?, [docId]);

// Chunk it const chunks = chunkText(content, 1000, 200);

// Embed in batches (OpenRouter supports batch) // Batch size: keep it reasonable (e.g. 32) to avoid huge payloads. const BATCH = 32;

let chunkIndex = 0;

for (let i = 0; i < chunks.length; i += BATCH) { const batch = chunks.slice(i, i + BATCH); const embeddings = await embedText(batch); // returns number[][]

// Insert each chunk
for (let j = 0; j < batch.length; j++) {
  const chunkText = batch[j];
  const emb = embeddings[j];

  const chunkId = `${docId}::${chunkIndex}`;
  const vec = toVectorSqlLiteral(emb);

  // We can’t parameterize SQL functions like vector(...) easily,
  // so we embed the vector literal directly.
  // Text fields still use parameters (safe).
  await sqlite.execute(
    `INSERT INTO chunks (id, doc_id, chunk_index, chunk_text, embedding, updated_at)
     VALUES (?, ?, ?, ?, ${vec}, ?)`,
    [chunkId, docId, chunkIndex, chunkText, updatedAt]
  );

  chunkIndex++;
}

} }

Safety note (important)

We are injecting the vector('...') literal into SQL. That’s okay only because:

it’s generated from numbers you got from OpenRouter, not user input

you control the formatting

chunk text and ids remain parameterized

If you later store user-supplied vectors (rare), you must sanitize.

7. Searching (vector_top_k)

We:

embed the query

ask SQLite for top K nearest chunks

join to docs so we can show doc title

return results

import sqlite from "https://esm.town/v/std/sqlite/main.ts"; import { embedText } from "./embeddings.ts"; import { toVectorSqlLiteral } from "./vectorSql.ts";

export async function semanticSearch(query: string, k = 10) { const [qEmb] = await embedText(query); const qVec = toVectorSqlLiteral(qEmb);

// ASSUMPTION: // vector_top_k('chunks_embedding_idx', , k) returns rows with (id, distance?) and id=ROWID // Many examples show it returns (id, distance) where id is rowid of table rows. // // We will JOIN on chunks.rowid = v.id (or v.id = chunks.rowid) const res = await sqlite.execute( SELECT d.id AS doc_id, d.title AS doc_title, c.id AS chunk_id, c.chunk_index, c.chunk_text FROM vector_top_k('chunks_embedding_idx', ${qVec}, ?) v JOIN chunks c ON c.rowid = v.id JOIN docs d ON d.id = c.doc_id ORDER BY c.doc_id, c.chunk_index, [k] );

return res.rows ?? []; }

If your engine returns a different shape (like id named rowid or includes distance), you’ll adjust this query—but this is the common pattern.

8. Build a simple HTTP API (and optional UI)

Create an HTTP val semanticSearch_http:

import { setupDb } from "./dbSetup.ts"; import { semanticSearch } from "./search.ts";

export default async function handler(req: Request): Promise { await setupDb();

const url = new URL(req.url);

// Quick UI: open in browser if (req.method === "GET" && url.pathname === "/") { return new Response( <html> <head><meta charset="utf-8" /><title>Semantic Search</title></head> <body style="font-family: system-ui; max-width: 900px; margin: 40px auto;"> <h1>Semantic Search</h1> <form method="GET" action="/search"> <input name="q" style="width: 70%; padding: 10px; font-size: 16px;" placeholder="Search..." /> <button style="padding: 10px 14px; font-size: 16px;">Search</button> </form> <p style="opacity: .7;">Uses OpenRouter embeddings + SQLite vector search (assumed).</p> </body> </html>, { headers: { "Content-Type": "text/html; charset=utf-8" } } ); }

// API endpoint: /search?q=... if (req.method === "GET" && url.pathname === "/search") { const q = url.searchParams.get("q") ?? ""; const k = Number(url.searchParams.get("k") ?? "10");

if (!q.trim()) {
  return new Response(JSON.stringify({ error: "Missing q" }), {
    status: 400,
    headers: { "Content-Type": "application/json" },
  });
}

const results = await semanticSearch(q, Number.isFinite(k) ? k : 10);

// If you want HTML results when opened in browser:
const accept = req.headers.get("accept") ?? "";
if (accept.includes("text/html")) {
  const items = results.map((r: any) => `
    <div style="padding: 14px; border: 1px solid #ddd; border-radius: 10px; margin: 12px 0;">
      <div style="font-weight: 700;">${escapeHtml(String(r.doc_title))}</div>
      <div style="opacity: .7; font-size: 12px;">doc: ${escapeHtml(String(r.doc_id))} · chunk ${r.chunk_index}</div>
      <pre style="white-space: pre-wrap; margin-top: 10px;">${escapeHtml(String(r.chunk_text))}</pre>
    </div>
  `).join("");

  return new Response(
    `
    <html>
      <head><meta charset="utf-8" /><title>Results</title></head>
      <body style="font-family: system-ui; max-width: 900px; margin: 40px auto;">
        <a href="/">← back</a>
        <h2>Results for: ${escapeHtml(q)}</h2>
        ${items || "<p>No results</p>"}
      </body>
    </html>
    `,
    { headers: { "Content-Type": "text/html; charset=utf-8" } }
  );
}

// JSON by default
return new Response(JSON.stringify({ q, results }), {
  headers: { "Content-Type": "application/json" },
});

}

return new Response("Not found", { status: 404 }); }

function escapeHtml(s: string) { return s .replaceAll("&", "&") .replaceAll("<", "<") .replaceAll(">", ">") .replaceAll('"', """) .replaceAll("'", "'"); }

9. Add documents (two easy ways) Option A: Hardcode a “seed docs” function (quickest for noobs)

Create a val seedDocs:

import { setupDb } from "./dbSetup.ts"; import { upsertDoc } from "./docs.ts"; import { reindexDoc } from "./indexer.ts";

export async function seedDocs() { await setupDb();

const docs = [ { id: "doc-hello", title: "Hello Doc", content: "Hello world. This is a test document about dogs, cats, and APIs.", }, { id: "doc-webhooks", title: "Webhook Notes", content: "To verify a webhook signature, compute HMAC of the payload and compare.", }, ];

for (const d of docs) { await upsertDoc(d.id, d.title, d.content); await reindexDoc(d.id); }

return { ok: true, count: docs.length }; }

Run seedDocs(). Then open your HTTP val and search.

Option B: Create an HTTP “ingest” endpoint (more practical)

Add to your HTTP handler:

POST /ingest with {id,title,content}

upsert + reindex

(Ask and I’ll paste the exact code, but the above already gives you the primitives.)

10. Add a Cron val (optional, recommended)

If your documents change elsewhere, you can reindex on a schedule.

Cron val example:

Every hour:

select docs updated recently

reindex them

import sqlite from "https://esm.town/v/std/sqlite/main.ts"; import { setupDb } from "./dbSetup.ts"; import { reindexDoc } from "./indexer.ts";

export default async function cron() { await setupDb();

// Reindex docs updated in the last 2 hours (example) const since = Date.now() - 2 * 60 * 60 * 1000;

const res = await sqlite.execute( SELECT id FROM docs WHERE updated_at >= ?, [since] );

for (const row of res.rows ?? []) { await reindexDoc(String(row.id)); }

return { ok: true, reindexed: res.rows?.length ?? 0 }; }

11. “Hybrid search” upgrade (semantic + exact keywords)

Once you have semantic working, add lexical search so exact identifiers/words win when appropriate.

11.1 Create an FTS table await sqlite.execute(CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5(chunk_id, chunk_text););

11.2 Keep it in sync when indexing

After inserting a chunk, also upsert into FTS:

await sqlite.execute( INSERT INTO chunk_fts (chunk_id, chunk_text) VALUES (?, ?), [chunkId, chunkText] );

If chunk ids can be reinserted, you may prefer:

delete old FTS rows for doc first, then insert fresh

11.3 Hybrid query

Get top 50 semantic chunks

Get top 50 lexical chunks

Merge + dedupe in TypeScript

Sort with a simple scoring heuristic (semantic rank + lexical rank)

This is extremely effective and beginner-friendly.

12. Common pitfalls + fixes Pitfall A: “Vector column dimension mismatch”

Symptoms:

insert fails

or vector_top_k fails

Fix:

log embedding length using testEmbeddingDim()

change F32_BLOB(1024) to the real number

rebuild DB table (or create a new table)

Pitfall B: “SQL functions not found” (vector_top_k, libsql_vector_idx)

This means the assumed native vector layer isn’t enabled.

For this guide, we assume it is. If it isn’t, the fallback paths are:

store embeddings as JSON blobs, do brute-force cosine in JS (slow but works)

use Postgres + pgvector externally (like the 2024 blog post)

use a hosted vector DB

But don’t mix those into this manual yet—you asked to assume it works.

Pitfall C: Timeouts / rate limits from OpenRouter

Fixes:

reduce batch size (e.g., 16)

add a small delay between batches

cache embeddings using content_hash so you don’t recompute unchanged chunks

Pitfall D: High cost

Even if embeddings are cheap, big docs + too many chunks adds up.

Fixes:

chunk bigger (1200–1500 chars)

only embed changed docs

add a “max docs indexed” limit in early prototypes

13. Minimal checklist (copy/paste workflow)

Add env var: OPENROUTER_API_KEY

Run testEmbeddingDim() → note length D

Set schema F32_BLOB(D)

Run setupDb()

Add docs (seed or ingest)

Run reindexDoc(docId) for each doc

Hit /search?q=...