Part of Val Town Semantic Search.

Uses Val Town's blob storage to search embeddings of all vals, by downloading them all and iterating through all of them to compute distance. Slow and terrible, but it works!

• Get metadata from blob storage: allValsBlob${dimensions}EmbeddingsMeta (currently allValsBlob1536EmbeddingsMeta), which has a list of all indexed vals and where their embedding is stored (batchDataIndex points to the blob, and valIndex represents the offset within the blob).
  • The blobs have been generated by janpaul123/indexValsBlobs. It is not run automatically.
• Get all blobs with embeddings pointed to by the metadata, e.g. allValsBlob1536EmbeddingsData_0 for batchDataIndex 0.
• Call OpenAI to generate an embedding for the search query.
• Go through all embeddings and compute cosine similarity with the embedding for the search query.
• Return list sorted by similarity.

Migrated from folder: semanticSearchPrototype/semanticSearchBlobs