Semantic search powered by libSQL/Turso DiskANN — runs entirely on Val Town's built-in SQLite. No external database needed.

→ Fork this val on Val Town to deploy your own instance in ~2 minutes.

---

SlimArmor is a self-hosted vector database that lets you store text and search it by meaning, not just keywords.

Example:

• You store: "Dogs are loyal companions"
• You search: "furry pets"
• It finds the dog text — even though the words are different ✅

This is called semantic search and it's powered by AI embeddings. SlimArmor handles all the hard parts: embedding generation, storage, indexing, deduplication, and retrieval.

---

Feature	Detail
🔍 Semantic search	Cosine similarity via libSQL DiskANN vector index
⚡ Hybrid search	Vector + BM25 keyword boosting
🧠 Smart dedup	Content-hashing skips re-embedding unchanged text
📦 Batch upsert	Send hundreds of records in one API call
🔖 Metadata filters	Filter results by any JSON metadata field
📄 Chunked upsert	Auto-split long documents into overlapping chunks
🔄 Export / Import	Migrate data between instances or providers
🖥️ Browser CLI	Built-in terminal UI at /ui — no client needed
🔐 Optional auth	Bearer token protection for write operations
🔌 Multi-provider	Nebius, OpenAI, OpenRouter, or any OpenAI-compatible API

---

Go to val.town/x/kamenxrider/slimarmor and click Fork.

In your forked val, go to Settings → Environment Variables and add:

Required:

NEBIUS_API_KEY=your-nebius-api-key

Get a free Nebius API key at nebius.com. Alternatively, use OpenAI or any OpenAI-compatible provider — see Provider Configuration below.

Recommended:

ADMIN_TOKEN=some-secret-token

Without ADMIN_TOKEN, all operations are open (anyone can write). Set this to protect write endpoints.

Your API is live at the URL shown in your val's HTTP endpoint. Visit /ui for the interactive browser CLI, or / for the API info page.

# Add a record
curl -X POST https://YOUR_ENDPOINT/upsert \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_ADMIN_TOKEN" \
  -d '{"id": "note-1", "text": "Dogs are loyal companions", "meta": {"category": "animals"}}'

# Search
curl -X POST https://YOUR_ENDPOINT/search \
  -H "Content-Type: application/json" \
  -d '{"query": "furry pets", "k": 5}'

That's it. 🎉

---

SlimArmor works with any OpenAI-compatible embedding API — the provider doesn't matter, only two things do:

1. The API URL accepts POST /v1/embeddings with {"model": "...", "input": [...]}
2. The embedding dimensions your chosen model outputs — this is baked into the database schema at first run and cannot be changed without a full reset

⚠️ Dimensions are locked at first run. When setup() is called for the first time, the table is created with a fixed F32_BLOB(N) column where N is your model's output dimension. Switching to a model with different dimensions requires exporting your data, clearing the DB, and re-importing. Use EMBEDDING_DIM=auto to detect dimensions automatically.

Set env vars to configure your provider:

NEBIUS_API_KEY=your-key

Model	Dimensions	Notes
Qwen/Qwen3-Embedding-8B	4096	Default — high quality, generous free tier

EMBEDDING_PROVIDER=openai
OPENAI_API_KEY=sk-your-key

Model	Dimensions	Notes
text-embedding-3-small	1536	Fast & cheap — fits ~130k records/GB
text-embedding-3-large	3072	Higher quality — fits ~65k records/GB

EMBEDDING_PROVIDER=openrouter
OPENROUTER_API_KEY=your-key
EMBEDDING_MODEL=openai/text-embedding-3-small
EMBEDDING_DIM=1536

If your provider exposes a /v1/embeddings endpoint, just point SlimArmor at it:

EMBEDDING_API_URL=https://your-provider.com/v1/embeddings
EMBEDDING_API_KEY=your-key
EMBEDDING_MODEL=your-model-name
EMBEDDING_DIM=768

This works with Ollama, LM Studio, Together AI, Fireworks, Mistral, Cohere (via their OpenAI-compat layer), and any other compatible service.

Dimensions determine both search quality and storage capacity:

Dimensions	Example Models	Records per 1 GB	Quality
4096	Nebius Qwen3-8B	~47,500	🟢 Excellent
3072	OpenAI text-embedding-3-large	~65,000	🟢 Excellent
1536	OpenAI text-embedding-3-small	~130,000	🟡 Good
768	nomic-embed-text, bge-small	~260,000	🟡 Good
384	all-MiniLM-L6-v2	~520,000	🟠 Okay for simple tasks

Higher dimensions = better semantic understanding but more storage. For most use cases, 1536 (OpenAI small) or 768 (nomic) is the sweet spot.

⚠️ Switching models requires a full reset. Embeddings from different models are incompatible — a vector from model A means nothing when compared to a vector from model B. Export your data first (GET /export), then POST /clear?confirm=yes, update env vars, then POST /import.

---

All endpoints are relative to your val's HTTP endpoint URL.

Insert or update one or many records.

Single record:

{ "id": "doc-1", "text": "Your text here", "meta": { "category": "notes" } }

Batch (array):

[
  { "id": "doc-1", "text": "First document", "meta": { "tag": "a" } },
  { "id": "doc-2", "text": "Second document", "meta": { "tag": "b" } }
]

Response:

{ "success": true, "embedded": 2, "skipped": 0, "results": [...] }

• embedded — records that were re-embedded (text changed or new)
• skipped — records whose text was unchanged (only meta updated, no API call)

---

Search for semantically similar records.

Request:

{
  "query": "search query text",
  "k": 10,
  "maxDistance": 0.65,
  "offset": 0,
  "filters": { "category": "tech" },
  "hybrid": { "enabled": true, "alpha": 0.25 }
}

Field	Type	Default	Description
query	string	required	Natural language search query
k	number	10	Number of results to return (max 200)
maxDistance	number	none	Filter out results with distance > this value
offset	number	0	Skip first N results (for pagination)
filters	object	none	Key/value metadata filters
hybrid.enabled	boolean	false	Enable keyword boosting
hybrid.alpha	number	0.25	Keyword weight 0–1 (0 = pure vector, 1 = pure keyword)
hybrid.keyword_k	number	2×k	Number of keyword candidates to consider

Response:

{
  "success": true,
  "count": 3,
  "results": [
    {
      "id": "doc-1",
      "text": "Dogs are loyal companions",
      "meta": { "category": "animals" },
      "distance": 0.49,
      "score": 0.43,
      "keyword_score": 0.12
    }
  ]
}

Distance guide:

Distance	Meaning
0.0 – 0.4	Very similar — always include
0.4 – 0.6	Related — good default range
0.6 – 0.7	Somewhat related
0.7+	Likely unrelated — filter out

Use GET /calibrate?q=your+query to find the right threshold for your data.

---

Delete a record by ID.

{ "id": "doc-1" }

---

Retrieve a single record by ID.

---

List record IDs.

Param	Default	Description
limit	100	Max records to return (max 1000)
offset	0	Pagination offset
prefix	none	Filter IDs by prefix

---

Automatically split a long document into overlapping chunks and upsert each one.

{
  "id": "doc-long",
  "text": "...very long document...",
  "meta": { "source": "book" },
  "chunkSize": 800,
  "overlap": 100
}

Chunks are stored as doc-long::chunk1, doc-long::chunk2, etc., each with parent_id, chunk_index, and chunk_total in their metadata.

---

Export records as JSON (without embeddings — text + meta only).

Param	Default	Description
limit	200	Max records
offset	0	Pagination offset

---

Import records from a previous export (re-embeds all text).

{ "records": [{ "id": "doc-1", "text": "...", "meta": {} }] }

Or pass an array directly.

---

Runs a smoke test: inserts 5 demo records, performs 3 searches, returns results and timing.

Seeds N synthetic records across 4 categories (tech, science, business, lifestyle). Useful for testing and threshold calibration. Max 1000 per call.

Analyzes the top 20 results for a query and suggests distance thresholds:

• tight — top 3 results only
• balanced — top 10 (recommended default)
• loose — all retrieved

Runs self-checks: auth, stats, list, and search. Add ?write=yes for full write tests (requires ALLOW_WRITE_TESTS=1 env var).

Drops and recreates the DiskANN vector index with current settings. Use after changing INDEX_* env vars.

Deletes ALL records from the main table and FTS index. Cannot be undone.

🔐 = requires Authorization: Bearer <ADMIN_TOKEN> header when ADMIN_TOKEN env var is set.

---

Import vectordb.ts directly into any other val:

import * as db from "https://esm.town/v/kamenxrider/slimarmor/vectordb.ts";

// First-time setup (idempotent — safe to call on every cold start)
await db.setup();

// Upsert a record
await db.upsert("note-1", "Buy milk and eggs", { tag: "shopping" });

// Batch upsert
await db.upsertMany([
  { id: "note-2", text: "Call the dentist", meta: { tag: "health" } },
  { id: "note-3", text: "Finish the quarterly report", meta: { tag: "work" } },
]);

// Search
const results = await db.search("grocery shopping", 5, 0.65);
// → [{ id, text, meta, distance }, ...]

// Search with filters
const workItems = await db.search("meetings", 10, undefined, {
  filters: { tag: "work" }
});

// Hybrid search (vector + keyword)
const hybrid = await db.search("dentist appointment", 5, undefined, {
  hybrid: { enabled: true, alpha: 0.3 }
});

// Get a single record
const record = await db.get("note-1");

// Delete
await db.remove("note-1");

// Stats
const { count, estimated_storage_mb } = await db.stats();

// Chunk and upsert a long document
await db.upsertChunked("essay-1", longText, { source: "blog" }, {
  size: 800,
  overlap: 100,
});

Note: When importing as a module, SlimArmor uses the same SQLite database as your val — so it shares data with any HTTP API you've deployed. Set your embedding env vars in the importing val.

---

Variable	Required	Default	Description
NEBIUS_API_KEY	If using Nebius	—	Nebius API key
OPENAI_API_KEY	If using OpenAI	—	OpenAI API key
OPENROUTER_API_KEY	If using OpenRouter	—	OpenRouter API key
EMBEDDING_PROVIDER	No	nebius	Preset: nebius, openai, openrouter
EMBEDDING_API_URL	No	(from preset)	Custom API URL
EMBEDDING_API_KEY	No	—	Generic key fallback
EMBEDDING_MODEL	No	(from preset)	Override model name
EMBEDDING_DIM	No	(from preset)	Override dimensions, or auto to detect

Variable	Required	Default	Description
ADMIN_TOKEN	No	—	Bearer token for write operations. If unset, all ops are open.

Variable	Default	Description
INDEX_METRIC	cosine	Distance metric: cosine or l2
INDEX_MAX_NEIGHBORS	64	DiskANN graph degree (8–256). Lower = smaller index, less accurate
INDEX_COMPRESS_NEIGHBORS	float8	Neighbor compression: float8, float16, floatb16, float32, float1bit, none
INDEX_ALPHA	1.2	Graph density (≥1). Lower = faster/sparser
INDEX_SEARCH_L	200	Query-time search effort. Higher = more accurate, slower
INDEX_INSERT_L	70	Insert-time graph quality. Higher = better index, slower inserts

After changing any INDEX_* variable, call POST /reindex to apply the new settings.

Variable	Default	Description
ALLOW_WRITE_TESTS	0	Set to 1 to enable /validate?write=yes
ALLOW_WRITE_TESTS_NOAUTH	0	Set to 1 to allow write tests without auth header

---

Benchmarked with Nebius Qwen/Qwen3-Embedding-8B (4096 dims):

Metric	Value
Storage per record	~22 KB
Max records per 1 GB	~47,500
Embedding latency	~460ms per batch
Search latency	<100ms
Batch size	Up to 96 texts per API call

Storage scales directly with dimensions — lower-dim models fit far more records:

Model	Dims	Storage/record	Records/GB
Nebius Qwen3-8B	4096	~22 KB	~47,500
OpenAI text-embedding-3-large	3072	~18 KB	~65,000
OpenAI text-embedding-3-small	1536	~10 KB	~130,000
nomic-embed-text / bge-small	768	~7 KB	~260,000

---

Visit /ui on your endpoint for a full terminal-style interface.

Commands:

Command	Description
help	List all commands
search "query" [-k 10] [--max 0.65] [--mode balanced] [--filter key=value] [--hybrid] [--alpha 0.25]	Semantic search
upsert <id> "text"	Add or update a record
get <id>	Show a single record
del <id>	Delete a record
list [--limit 20] [--prefix id-prefix]	List record IDs
stats	Storage and index info
calibrate "query"	Suggest distance thresholds
seed <n>	Seed N synthetic records
clear --yes	Delete all records
reindex	Rebuild vector index
auth <token>	Set admin token for session
logout	Clear auth token
ping	Health check
cls	Clear screen

Search modes (shorthand for --max):

• --mode tight → maxDistance 0.5
• --mode balanced → maxDistance 0.64
• --mode loose → maxDistance 0.7

---

┌──────────────────────────────────────────────────────┐
│              Browser CLI  (ui.ts)                    │
│         Terminal UI at /ui — no JS framework         │
└─────────────────────────┬────────────────────────────┘
                          │
                          ▼
┌──────────────────────────────────────────────────────┐
│               HTTP API  (api.ts)                     │
│  /upsert  /search  /delete  /stats  /seed  ...       │
│  Bearer token auth · Input validation · Error handling│
└──────────────┬───────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────┐
│           Vector DB Core  (vectordb.ts)              │
│  setup · upsert · upsertMany · search · remove       │
│  Content-hash dedup · Batch embed · FTS sync         │
└───────────┬──────────────────────┬───────────────────┘
            │                      │
            ▼                      ▼
┌───────────────────┐  ┌───────────────────────────────┐
│  Embedding API    │  │   Val Town SQLite (libSQL)    │
│  (OpenAI-compat)  │  │   F32_BLOB · DiskANN index   │
│  Nebius / OpenAI  │  │   FTS5 · vector_top_k        │
│  OpenRouter / etc │  │   vector_distance_cos        │
└───────────────────┘  └───────────────────────────────┘

-- Main records table
CREATE TABLE vectordb (
  id          TEXT PRIMARY KEY,
  text        TEXT NOT NULL,
  text_hash   TEXT NOT NULL,          -- SHA-256 for change detection
  embedding   F32_BLOB(4096),         -- Vector column (dim varies by provider)
  meta_json   TEXT,                   -- Optional JSON metadata
  updated_at  INTEGER NOT NULL        -- Unix ms timestamp
);

-- DiskANN vector index
CREATE INDEX vectordb_embedding_idx
ON vectordb (libsql_vector_idx(embedding,
  'metric=cosine',
  'max_neighbors=64',
  'compress_neighbors=float8'
));

-- Full-text search index for hybrid search
CREATE VIRTUAL TABLE vectordb_fts USING fts5(id, text);

-- Internal metadata (stores resolved embedding dim, etc.)
CREATE TABLE vectordb_meta (key TEXT PRIMARY KEY, value TEXT);

---

Embedding API error 401 → Your API key is missing or expired. Check the relevant env var (NEBIUS_API_KEY, OPENAI_API_KEY, etc.).

Embedding dim mismatch → Your EMBEDDING_DIM env var doesn't match what the model returns. Either remove EMBEDDING_DIM (use preset default) or set EMBEDDING_DIM=auto.

vector index(insert): failed to insert shadow row → The DiskANN index is out of sync (usually after a raw SQL DELETE). Call POST /reindex to rebuild it.

Search returns irrelevant results → Lower maxDistance (try 0.5 instead of 0.7). Use /calibrate?q=... to find the right value for your data.

Search returns no results → Check GET /stats to confirm data exists. Try raising maxDistance, or simplify your query.

Slow inserts → Each batch of up to 96 records makes one embedding API call (~460ms). For large imports, use POST /import with large batches rather than individual /upsert calls.

Switched providers, search broken → Embeddings are model-specific. Export data, POST /clear?confirm=yes, update env vars, then POST /import.

---

File	Description
api.ts	HTTP API — the main entry point (deploy this as your val's HTTP handler)
vectordb.ts	Core library — import this directly in other vals
ui.ts	Browser CLI — terminal interface served at /ui
README.md	This document
GUIDE.md	Beginner step-by-step guide
CHANGES.md	Changelog

---

MIT — fork freely.