Search

Generates OpenAI embeddings for all public vals, and stores them in [Neon](https://neon.tech/), using the [pg_vector](https://neon.tech/docs/extensions/pgvector) extension.
- Create the `vals_embeddings` table in Neon if it doesn't already exist.
- Get all val names from the [database of public vals](https://www.val.town/v/sqlite/db), made by [Achille Lacoin](https://www.val.town/u/pomdtr).
- Get all val names from the `vals_embeddings` table and compute the difference (which ones are missing).
- Iterate through all missing vals, get their code, get embeddings from OpenAI, and store the result in Neon.
- Can now be searched using [janpaul123/semanticSearchNeon](https://www.val.town/v/janpaul123/semanticSearchNeon).

const openai = new OpenAI();
const queryEmbedding = (await openai.embeddings.create({
  model: "text-embedding-3-small",
const embedding = await openai.embeddings.create({
  model: "text-embedding-3-small",
const queryEmbeddingVal = (await openai.embeddings.create({
  model: "text-embedding-3-small",

Migrated from folder: semanticSearchPrototype/debugValEmbeddings

export default async function semanticSearchPublicVals(query) {
  const allValsBlobEmbeddingsMeta = (await blob.getJSON("allValsBlobEmbeddingsMeta")) ?? {};
  const allBatchDataIndexes = _.uniq(Object.values(allValsBlobEmbeddingsMeta).map(item => item.batchDataIndex));
  const embeddingsBatches = [];
  const allBatchDataIndexesPromises = [];
  for (const batchDataIndex of allBatchDataIndexes) {
    const embeddingsBatchBlobName = `allValsBlobEmbeddingsData_${batchDataIndex}`;
    const promise = blob.get(embeddingsBatchBlobName).then((response) => response.arrayBuffer());
    promise.then((data) => {
      embeddingsBatches[batchDataIndex as any] = data;
      console.log(`Loaded ${embeddingsBatchBlobName} (${data.byteLength} bytes)`);
    });
  for (const id in allValsBlobEmbeddingsMeta) {
    const meta = allValsBlobEmbeddingsMeta[id];
    const embedding =
      (new Float32Array(embeddingsBatches[meta.batchDataIndex], 256 * 4 * meta.valIndex, 256 * 4)) as any;
    db.add({ id, embedding, metadata: {} });
  const openai = new OpenAI();
  const embedding = await openai.embeddings.create({
    model: "text-embedding-3-small",

Generates OpenAI embeddings for all public vals, and stores them in  Val Town's [blob storage](https://docs.val.town/std/blob/).
- Get all val names from the [database of public vals](https://www.val.town/v/sqlite/db), made by [Achille Lacoin](https://www.val.town/u/pomdtr).
- Put val names in batches. Vals in the same batch will have their embeddings stored in the same blob, at different offsets.
- Iterate through all each batch, get code for all the vals, get embeddings from OpenAI, and store the result in a blob.
- When finished, save the metadata JSON to its own blob.

  const sqlite = createClient({
    url: "libsql://valsembeddings-jpvaltown.turso.io",
    authToken: Deno.env.get("TURSO_AUTH_TOKEN_VALSEMBEDDINGS"),
  });
  sqlite.execute("CREATE TABLE IF NOT EXISTS vals_embeddings (id TEXT NOT NULL, embedding BLOB NOT NULL)");
  sqlite.execute("CREATE VIRTUAL TABLE IF NOT EXISTS vss_vals_embeddings USING vss0(embedding(256))");
  const existingEmbeddingsIds = new Set(
    (await sqlite.execute("SELECT id FROM vals_embeddings")).rows.map((row) => row[0]),
  );
    const id = idForVal(val);
    if (!existingEmbeddingsIds.has(id)) {
      newVals.push(val);
    const embedding = await openai.embeddings.create({
      model: "text-embedding-3-small",
    sqlite.execute({
      sql: "INSERT INTO vals_embeddings (id, embedding) VALUES (:id, :embeddingBinary)",
      args: { id, embeddingBinary },
  sqlite.execute(
    "INSERT INTO vss_vals_embeddings (rowid, embedding) SELECT rowid, embedding FROM vals_embeddings WHERE rowid NOT IN (SELECT rowid FROM vss_vals_embeddings)",
  );

Generates OpenAI embeddings for all public vals, and stores them in [Turso](https://turso.tech/), using the [sqlite-vss](https://github.com/asg017/sqlite-vss) extension.
- Create the `vals_embeddings` and `vss_vals_embeddings` tables in Turso if they don't already exist.
- Get all val names from the [database of public vals](https://www.val.town/v/sqlite/db), made by [Achille Lacoin](https://www.val.town/u/pomdtr).
- Get all val names from the `vals_embeddings` table and compute the difference (which ones are missing).
- Iterate through all missing vals, get their code, get embeddings from OpenAI, and store the result in Turso.
- When finished, update the `vss_vals_embeddings` table so we can efficiently query them with the [sqlite-vss](https://github.com/asg017/sqlite-vss) extension.
  - This is blocked by a [bug in Turso](https://discord.com/channels/933071162680958986/1245378515679973420/1245378515679973420) that doesn't allow VSS indexes past a certain size.

  const { ChatOpenAI } = await import("npm:langchain/chat_models");
  const { OpenAIEmbeddings } = await import("npm:langchain/embeddings");
  const { createClient } = await import(
  const vectorStore = await SupabaseVectorStore.fromExistingIndex(
    new OpenAIEmbeddings({
      openAIApiKey: process.env.OPEN_API_KEY,

export let generateEmbeddings = async (
  req: express.Request,
  }
  const { OpenAIEmbeddings } = await import("npm:langchain/embeddings");
  const { createClient } = await import(
    splittedDocs,
    new OpenAIEmbeddings({
      openAIApiKey: process.env.OPEN_API_KEY,

Migrated from folder: linkedin_seeder/generateEmbeddings