Last Updated: 2026-02-02
Status: Production-ready v3

SlimArmor is a mini vector database for Val Town that provides semantic search capabilities using SQLite with libSQL/Turso vector extensions.

• Stores text with AI-generated embeddings (4096 dimensions by default)
• Enables semantic search (search by meaning, not keywords)
• Returns distance scores for ranking results
• Supports any OpenAI-compatible embedding API

┌─────────────────────────────────────────────────────────────┐
│                      HTTP API (api.ts)                       │
│  /upsert, /search, /delete, /stats, /calibrate, etc.        │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                   Vector DB Core (vectordb.ts)               │
│  - setup(), upsert(), search(), remove(), stats()           │
│  - Content hash check (skip re-embedding unchanged text)    │
│  - Dimension assertion (fail fast on mismatch)              │
└─────────────────────────────────────────────────────────────┘
                    │                       │
                    ▼                       ▼
┌──────────────────────────┐  ┌────────────────────────────────┐
│   Embedding Provider     │  │   Val Town SQLite (Turso)      │
│   (OpenAI-compatible)    │  │   - F32_BLOB vector columns    │
│   - Nebius (default)     │  │   - libsql_vector_idx (DiskANN)│
│   - OpenAI               │  │   - vector_top_k queries       │
│   - OpenRouter           │  │   - vector_distance_cos        │
│   - Custom               │  └────────────────────────────────┘
└──────────────────────────┘

The main library that can be imported into other vals.

Key exports:

• setup() - Creates table and index
• upsert(id, text, meta?) - Insert/update with smart re-embedding
• search(query, k?, maxDistance?) - Semantic search
• remove(id) - Delete record
• stats() - Count and storage estimate
• get(id) - Get single record
• listIds(limit?) - List all IDs
• reindex() - Recreate index
• getProviderInfo() - Current embedding config

Configuration via env vars:

• EMBEDDING_PROVIDER - Preset: nebius, openai, openrouter
• EMBEDDING_API_URL - Custom API URL
• EMBEDDING_API_KEY - API key
• EMBEDDING_MODEL - Model name
• EMBEDDING_DIM - Vector dimensions

RESTful API layer with admin/testing tools.

Core endpoints:

• POST /upsert - Insert/update record
• POST /search - Semantic search
• POST /delete - Delete record
• GET /get?id=... - Get record
• GET /list - List IDs

Admin endpoints:

• GET / - API info + provider config
• GET /ping - Health check
• GET /stats - Detailed storage stats
• GET /seed?n=100 - Seed synthetic data
• GET /calibrate?q=... - Threshold suggestions
• POST /reindex - Recreate index
• POST /clear?confirm=yes - Delete all

CREATE TABLE vectordb (
  id TEXT PRIMARY KEY,
  text TEXT NOT NULL,
  text_hash TEXT NOT NULL,      -- SHA-256 for change detection
  embedding F32_BLOB(4096),     -- Vector column (dimension varies by provider)
  meta_json TEXT,               -- Optional JSON metadata
  updated_at INTEGER NOT NULL   -- Unix timestamp ms
);

CREATE INDEX vectordb_embedding_idx 
ON vectordb (libsql_vector_idx(embedding, 'metric=cosine', 'max_neighbors=64', 'compress_neighbors=float8'));

We tested and applied these Turso-documented optimizations:

Trade-off: Slightly lower recall accuracy, significantly lower storage.

From calibration with "machine learning" query:

• Min: 0.46 (highly relevant)
• Median: 0.64
• Max: 0.67 (least relevant in top 20)

Recommended thresholds:

• Tight: 0.5 (top 3 only)
• Balanced: 0.64 (top 10)
• Loose: 0.7 (include all)

1. Single embedding dimension - Table created with fixed dimension. Changing providers requires clearing data.

2. No chunking - Each record = one embedding. Long documents should be pre-chunked by the user.

3. No hybrid search - Pure vector search, no FTS fallback. Could be added later.

4. Sync embedding calls - Each upsert calls embedding API synchronously. Batch support not implemented.

5. No pagination - Search returns up to k results, no cursor-based pagination.

If continuing development, consider:

1. Chunking support - Auto-split long documents, store as docId::chunkN

2. Hybrid search - Add FTS5 table, merge vector + keyword results

3. Batch embeddings - Batch multiple texts in one API call

4. Background indexing - Queue-based async embedding

5. Metadata filtering - SQL WHERE clauses on meta_json fields

6. Multi-index - Support different embedding models in same DB

GET /test

Inserts 5 demo records, runs searches, shows results.

GET /seed?n=1000

Seeds 1000 synthetic records (takes ~8 minutes).

GET /calibrate?q=your+query

Analyzes distance distribution, suggests thresholds.

Provider returned different dimension than expected. Check EMBEDDING_DIM env var matches your model.

Set the appropriate env var for your provider.

Lower maxDistance (try 0.5 instead of 0.7).

Normal - each insert requires an API call (~460ms). Batch support not implemented.

Clear data with POST /clear?confirm=yes and re-insert.

• TypeScript throughout
• Proper error handling with typed errors
• Parameterized SQL (no injection risk)
• Content hash prevents unnecessary re-embedding
• Dimension assertion fails fast on mismatch
• 30s timeout on embedding API calls
• AbortController for cancellation

1. Verified Val Town SQLite supports vectors - F32_BLOB, libsql_vector_idx, vector_top_k all work
2. Tested Nebius embedding API - Qwen3-Embedding-8B returns 4096 dims
3. Built core vectordb.ts - upsert, search, delete, stats
4. Added optimizations - compress_neighbors=float8, max_neighbors=64
5. Added distance scores - Returns cosine distance in results
6. Added maxDistance filter - Filter out low-relevance results
7. Added admin tools - /seed, /calibrate, /stats, /clear
8. Made multi-provider - Nebius, OpenAI, OpenRouter, custom
9. Documented everything - README, GUIDE, HANDOVER

• Val: https://www.val.town/x/kamenxrider/slimarmor
• Endpoint: https://kamenxrider--95fbe492ffe111f0bee942dde27851f2.web.val.run
• Module: https://esm.town/v/kamenxrider/slimarmor/vectordb.ts

This document is for the next developer/AI continuing work on SlimArmor.