A lightweight, optimized vector database built on Val Town's SQLite (powered by Turso/libSQL) with Nebius Qwen3-Embedding-8B embeddings.

• 4096-dimensional embeddings via Nebius Qwen3-Embedding-8B
• Semantic search with cosine similarity
• Distance scores in search results (0 = identical, 1 = orthogonal, 2 = opposite)
• Smart re-embedding - only re-embeds when text changes (content hash check)
• Optimized index for storage efficiency
• Scale testing tools - seed data, calibrate thresholds, detailed stats

Optimization	Benefit
compress_neighbors=float8	75% less index storage
max_neighbors=64	66% fewer neighbors vs default
Distance filtering	Filter out low-relevance results

~22KB per record → ~45,000 records per 1GB

Method	Endpoint	Description
POST	/upsert	Insert/update record {id, text, meta?}
POST	/search	Search {query, k?, maxDistance?}
POST	/delete	Delete record {id}
GET	/get?id=...	Get single record
GET	/list?limit=...	List record IDs

Method	Endpoint	Description
GET	/	Info + all endpoints
GET	/ping	Health check
GET	/stats	Detailed storage stats
GET	/test	Smoke test with 5 demo records
GET	/seed?n=100	Seed N synthetic records (max 1000)
GET	/calibrate?q=...	Suggest distance thresholds for a query
POST	/reindex	Recreate index with optimized settings
POST	/clear?confirm=yes	Delete ALL records (dangerous!)

curl -X POST https://YOUR_ENDPOINT/upsert \
  -H "Content-Type: application/json" \
  -d '{"id": "doc-1", "text": "Dogs are loyal pets", "meta": {"category": "animals"}}'

curl -X POST https://YOUR_ENDPOINT/search \
  -H "Content-Type: application/json" \
  -d '{"query": "furry pets", "k": 5}'

Only return results with distance ≤ 0.5 (more relevant):

curl -X POST https://YOUR_ENDPOINT/search \
  -H "Content-Type: application/json" \
  -d '{"query": "furry pets", "k": 10, "maxDistance": 0.5}'

curl https://YOUR_ENDPOINT/seed?n=100

curl "https://YOUR_ENDPOINT/calibrate?q=machine+learning"

Returns suggested thresholds:

• tight (top 3 only)
• balanced (top 10, recommended default)
• loose (all retrieved)

The distance field uses cosine distance:

Distance	Meaning
0.0 - 0.3	Very similar
0.3 - 0.5	Related
0.5 - 0.7	Somewhat related
0.7+	Likely unrelated

Tip: Use /calibrate?q=... to find the right threshold for your data.

To test how the system performs at scale:

1. Seed data: GET /seed?n=1000
2. Check stats: GET /stats
3. Run queries: POST /search
4. Calibrate: GET /calibrate?q=your+query

Recommended test sizes: 100 → 1,000 → 5,000 records

You can import vectordb.ts directly in other Val Town projects:

import * as db from "https://esm.town/v/kamenxrider/slimarmor/vectordb.ts";

// Setup (creates table + index if not exists)
await db.setup();

// Upsert
await db.upsert("my-doc", "Some text to embed", { category: "notes" });

// Search
const results = await db.search("search query", 10, 0.5);
// Returns: [{ id, text, meta, distance }, ...]

// Delete
await db.remove("my-doc");

// Stats
const stats = await db.stats();
// Returns: { count: 5, estimated_storage_mb: 0.11 }

Variable	Description
NEBIUS_API_KEY	Your Nebius API key for embeddings

File	Description
vectordb.ts	Core vector DB library (importable)
api.ts	HTTP API endpoints
README.md	This file

• Runtime: Val Town (Deno)
• Database: Val Town SQLite (Turso/libSQL)
• Embeddings: Nebius Qwen3-Embedding-8B (4096 dims)
• Vector Index: DiskANN with cosine similarity

MIT