Last Updated: 2026-02-02
Status: Production-ready v4 (with Browser CLI)

---

SlimArmor is a mini vector database for Val Town that provides semantic search capabilities using SQLite with libSQL/Turso vector extensions.

• Stores text with AI-generated embeddings (4096 dimensions by default)
• Enables semantic search (search by meaning, not keywords)
• Returns distance scores for ranking results
• Supports any OpenAI-compatible embedding API
• NEW: Browser CLI for interactive management
• NEW: Metadata filtering + hybrid keyword boost
• NEW: Batch upsert, chunked upsert, export/import

---

┌─────────────────────────────────────────────────────────────┐
│                    Browser CLI (ui.ts)                       │
│  Terminal-style web UI at /ui                                │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                      HTTP API (api.ts)                       │
│  /ui, /upsert, /search, /delete, /stats, /calibrate, etc.   │
│  + Admin token authentication for write operations           │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                   Vector DB Core (vectordb.ts)               │
│  - setup(), upsert(), search(), remove(), stats()           │
│  - Content hash check (skip re-embedding unchanged text)    │
│  - Dimension assertion (fail fast on mismatch)              │
└─────────────────────────────────────────────────────────────┘
                    │                       │
                    ▼                       ▼
┌──────────────────────────┐  ┌────────────────────────────────┐
│   Embedding Provider     │  │   Val Town SQLite (Turso)      │
│   (OpenAI-compatible)    │  │   - F32_BLOB vector columns    │
│   - Nebius (default)     │  │   - libsql_vector_idx (DiskANN)│
│   - OpenAI               │  │   - vector_top_k queries       │
│   - OpenRouter           │  │   - vector_distance_cos        │
│   - Custom               │  └────────────────────────────────┘
└──────────────────────────┘

---

A terminal-style web interface for interacting with the vector database.

Features:

• Monospace terminal aesthetic (GitHub dark theme)
• Command history (↑/↓ arrows)
• Clickable IDs in results (auto-fills get <id>)
• Color-coded distance scores (green <0.5, orange 0.5-0.65, red >0.65)
• Session-based auth token storage
• Mobile responsive
• Pretty tables and stats boxes (not raw JSON)

Access: GET /ui

CLI Commands:

Command	Description
help	Show all available commands
stats	Storage & index statistics
search "query" [-k 10] [--mode balanced] [--max 0.64] [--filter key=value] [--hybrid] [--alpha 0.25] [--offset 0]	Semantic search
get <id>	Show record details
list [--limit 20] [--prefix seed-]	List record IDs
calibrate "query"	Suggest distance thresholds
upsert <id> "text..."	Create/update record [AUTH]
del <id>	Delete record [AUTH]
seed 100	Seed synthetic data [AUTH]
clear --yes	Delete ALL records [AUTH]
reindex	Recreate vector index [AUTH]
auth <token>	Set admin token
logout	Clear auth token
ping	Health check
cls	Clear output

Command aliases: ls → list, delete → del, ? → help

Search modes:

• --mode tight → maxDistance 0.5 (top 3 only)
• --mode balanced → maxDistance 0.64 (top 10)
• --mode loose → maxDistance 0.7 (include all)

The main library that can be imported into other vals.

Key exports:

• setup() - Creates table and index
• upsert(id, text, meta?) - Insert/update with smart re-embedding
• search(query, k?, maxDistance?) - Semantic search
• remove(id) - Delete record
• stats() - Count and storage estimate
• get(id) - Get single record
• listIds(limit?) - List all IDs
• reindex() - Recreate index
• getProviderInfo() - Current embedding config

Configuration via env vars:

• EMBEDDING_PROVIDER - Preset: nebius, openai, openrouter
• EMBEDDING_API_URL - Custom API URL
• EMBEDDING_API_KEY - API key
• EMBEDDING_MODEL - Model name
• EMBEDDING_DIM - Vector dimensions

RESTful API layer with Browser CLI and admin token authentication.

UI:

• GET /ui - Browser CLI interface

Core endpoints:

• POST /upsert - Insert/update record [AUTH]
• POST /search - Semantic search
• POST /delete - Delete record [AUTH]
• GET /get?id=... - Get record
• GET /list - List IDs (supports offset/prefix)
• POST /upsert_chunked - Chunk + upsert [AUTH]
• GET /export - Export records [AUTH]
• POST /import - Import records (batch upsert) [AUTH]

Admin endpoints:

• GET / - API info + provider config
• GET /ping - Health check
• GET /stats - Detailed storage stats
• GET /test - Smoke test [AUTH]
• GET /seed?n=100 - Seed synthetic data [AUTH]
• GET /calibrate?q=... - Threshold suggestions
• GET /validate - Self-checks (optional write tests)
• POST /reindex - Recreate index [AUTH]
• POST /clear?confirm=yes - Delete all [AUTH]
• GET /auth - Auth status

Authentication (NEW in v4):

• Set ADMIN_TOKEN env var to enable authentication
• Pass token via Authorization: Bearer <token> header
• Write operations ([AUTH]) require auth when ADMIN_TOKEN is set
• If ADMIN_TOKEN not set, all operations are open (current default)

---

CREATE TABLE vectordb (
  id TEXT PRIMARY KEY,
  text TEXT NOT NULL,
  text_hash TEXT NOT NULL,      -- SHA-256 for change detection
  embedding F32_BLOB(4096),     -- Vector column (dimension varies by provider)
  meta_json TEXT,               -- Optional JSON metadata
  updated_at INTEGER NOT NULL   -- Unix timestamp ms
);

CREATE INDEX vectordb_embedding_idx 
ON vectordb (libsql_vector_idx(embedding, 'metric=cosine', 'max_neighbors=64', 'compress_neighbors=float8'));

CREATE VIRTUAL TABLE vectordb_fts USING fts5(id, text);

CREATE TABLE vectordb_meta (
  key TEXT PRIMARY KEY,
  value TEXT
);

---

We tested and applied these Turso-documented optimizations:

Setting	Value	Why
metric=cosine	Cosine distance	Standard for text embeddings
max_neighbors=64	64 neighbors	Down from default ~192, saves storage
compress_neighbors=float8	1 byte/dim	75% less index storage

Trade-off: Slightly lower recall accuracy, significantly lower storage.

---

Metric	Value
Storage per record	~22 KB
Estimated max records/GB	~47,500
Embedding latency (Nebius)	~460ms
Search latency	<100ms

From calibration with "machine learning" query:

• Min: 0.46 (highly relevant)
• Median: 0.64
• Max: 0.67 (least relevant in top 20)

Recommended thresholds:

• Tight: 0.5 (top 3 only)
• Balanced: 0.64 (top 10)
• Loose: 0.7 (include all)

---

1. Single embedding dimension - Table created with fixed dimension. Changing providers requires clearing data.

2. Chunking is naive - Built-in chunker is character-based and may not respect sentences.

3. Hybrid search is boost-only - Keyword scores re-rank vector results; it does not union keyword-only matches.

4. FTS optional - If FTS5 is unavailable, hybrid search silently falls back to vector-only.

5. Sync embedding calls - Each upsert calls embedding API synchronously. Batch support not implemented.

6. No pagination - Search returns up to k results, no cursor-based pagination.

---

If continuing development, consider:

1. Hybrid union retrieval - Include keyword-only matches beyond vector candidates

2. Background indexing - Queue-based async embedding

3. Advanced filters - Range queries, nested JSON paths, boolean logic

4. Cursor pagination - Stable pagination for large datasets

5. Multi-index - Support different embedding models in same DB

---

GET /ui

Interactive terminal interface - type help to see commands.

GET /test

Inserts 5 demo records, runs searches, shows results. (Requires auth if ADMIN_TOKEN set)

GET /seed?n=1000

Seeds 1000 synthetic records (takes ~8 minutes). (Requires auth if ADMIN_TOKEN set)

GET /calibrate?q=your+query

Analyzes distance distribution, suggests thresholds.

---

Variable	Required	Default	Description
ADMIN_TOKEN	No	-	Enable auth for write operations
EMBEDDING_PROVIDER	No	nebius	Preset: nebius, openai, openrouter
NEBIUS_API_KEY	If nebius	-	Nebius API key
OPENAI_API_KEY	If openai	-	OpenAI API key
OPENROUTER_API_KEY	If openrouter	-	OpenRouter API key
EMBEDDING_API_URL	No	(from preset)	Custom API URL
EMBEDDING_API_KEY	No	-	Generic API key fallback
EMBEDDING_MODEL	No	(from preset)	Override model name
EMBEDDING_DIM	No	(from preset)	Override dimensions (auto to detect)
INDEX_METRIC	No	cosine	Vector distance metric (cosine or l2)
INDEX_MAX_NEIGHBORS	No	64	DiskANN neighbors (8-256)
INDEX_COMPRESS_NEIGHBORS	No	float8	float8 or none
ALLOW_WRITE_TESTS	No	0	Set to 1 to allow write tests via /validate?write=yes

---

Provider returned different dimension than expected. Check EMBEDDING_DIM env var matches your model.

Set the appropriate env var for your provider.

Lower maxDistance (try 0.5 instead of 0.7).

Normal - each insert requires an API call (~460ms). Batch support not implemented.

Clear data with POST /clear?confirm=yes and re-insert.

Set ADMIN_TOKEN env var and use auth <token> in CLI, or pass Authorization: Bearer <token> header in API calls.

---

• TypeScript throughout
• Proper error handling with typed errors
• Parameterized SQL (no injection risk)
• Content hash prevents unnecessary re-embedding
• Dimension assertion fails fast on mismatch
• 30s timeout on embedding API calls
• AbortController for cancellation
• Admin token auth for write protection

---

1. Verified Val Town SQLite supports vectors - F32_BLOB, libsql_vector_idx, vector_top_k all work
2. Tested Nebius embedding API - Qwen3-Embedding-8B returns 4096 dims
3. Built core vectordb.ts - upsert, search, delete, stats
4. Added optimizations - compress_neighbors=float8, max_neighbors=64
5. Added distance scores - Returns cosine distance in results
6. Added maxDistance filter - Filter out low-relevance results
7. Added admin tools - /seed, /calibrate, /stats, /clear
8. Made multi-provider - Nebius, OpenAI, OpenRouter, custom
9. Documented everything - README, GUIDE, HANDOVER
10. Added Browser CLI (v4) - Terminal-style web UI at /ui with auth support

---

• Val: https://www.val.town/x/kamenxrider/slimarmor
• Browser CLI: https://kamenxrider--95fbe492ffe111f0bee942dde27851f2.web.val.run/ui
• API Endpoint: https://kamenxrider--95fbe492ffe111f0bee942dde27851f2.web.val.run
• Module: https://esm.town/v/kamenxrider/slimarmor/vectordb.ts

---

This document is for the next developer/AI continuing work on SlimArmor.