Part of Val Town Semantic Search.

Generates OpenAI embeddings for all public vals, and stores them in Turso, using the sqlite-vss extension.

  • Create the vals_embeddings and vss_vals_embeddings tables in Turso if they don't already exist.
  • Get all val names from the database of public vals, made by Achille Lacoin.
  • Get all val names from the vals_embeddings table and compute the difference (which ones are missing).
  • Iterate through all missing vals, get their code, get embeddings from OpenAI, and store the result in Turso.
  • When finished, update the vss_vals_embeddings table so we can efficiently query them with the sqlite-vss extension.
    • This is blocked by a bug in Turso that doesn't allow VSS indexes past a certain size.
  • Can now be searched using janpaul123/semanticSearchTurso.

Migrated from folder: semanticSearchPrototype/indexValsTurso