Surprising embeddings

\n Embeddings. They're one of the parts of the LLM/AI wave that I sort of like.\n

\n An embedding model takes a bit of text and returns a big array of floating-point numbers that represent its\n position in a sort of _hyper-dimensional embedding space_. You interpret this array as sort of a big coordinate,\n like longitude & latitude & elevation & time and a thousand other dimensions, and you get sort of the word's place\n in the computer's… mind-brain, or whatever.\n

\n Embeddings are pretty cool when they work, because they sort of capture the idea of{\" \"}\n 'vibes', which makes them useful for search applications.\n Someone can search for 'lettuce' and get results for 'spinach' too because they're similar. It's amazing when it\n works. And pretty weird when it doesn't!\n

\n This project is based on this{\" \"}\n \n blog post I read last year\n {\" \"}\n about 'how RAG fails', which was unfortunately written by someone{\" \"}\n with a RAG alternative to sell.\n

\n But maybe maybe RAG is doomed, in some ways? Half the time it does something magical, and half the time it\n produces something deeply weird. It thinks that, for example, \"the 1900s\" are more similar to \"the 1700s\" than to\n \"20th century.\" Most of these, I think, are related to how the RAG is a little literal and focused on the style of\n a phrase as well as the content of it. On the other hand, all other search solutions, like fuzzy matching and most\n ranking algorithms, are also some sort of good-enough heuristic, so maybe RAG is totally fine.\n

\n Play around. You can enter up to 8 words or phrases. The thickness and opacity of the connections indicates how\n connected the model thinks the things are.\n

\n \n)\n\nconst examples =

\n Examples:{\" \"}\n {[\n {\n title: \"Apples and bananas are really different\",\n q: \"/?word=apples&word=oranges&word=bananas&word=grapefruit&word=blueberry&word=blackberry&word=raspberry&word=\",\n },\n {\n title: \"Lexus is way different from Tesla\",\n q: \"/?word=porsche&word=ferrari&word=tesla&word=kia&word=lexus&word=toyota&word=hyundai&word=\",\n },\n {\n title: \"Cats and dogs are more similar to each other than to canine and feline\",\n q: \"/?word=dog&word=cat&word=feline&word=canine&word=&word=&word=&word=\",\n },\n {\n title: \"Chess pieces wtf\",\n q: \"/?word=pawn&word=king&word=queen&word=rook&word=castle&word=&word=&word=\",\n },\n ].map((ex, i, arr) => {\n return (\n <>\n {ex.title}\n {i === arr.length - 1 ? \"\" : \", \"}\n \n )\n })}\n

\n\n// Based on https://github.com/Shawns2759/vector-similarity MIT\nfunction dotProduct(a: number[], b: number[]): number {\n return a.reduce((sum, value, index) => sum + value * b[index], 0)\n}\n\nfunction vectorMagnitude(a: number[]): number {\n return Math.sqrt(a.reduce((sum, value) => sum + value * value, 0))\n}\n\nfunction cosineSimilarity(a: number[], b: number[]): number {\n const dotProd = dotProduct(a, b)\n const magnitudeA = vectorMagnitude(a)\n const magnitudeB = vectorMagnitude(b)\n return dotProd / (magnitudeA * magnitudeB)\n}\n\nconst app = new Hono()\napp.get(\"/\", async (c) => {\n const r = blockAi(c.req.headers.get(\"User-Agent\"))\n if (r) return r\n\n if (working) {\n return c.text(\"Sorry, another embedding is being fetched, wait a moment and try again\")\n }\n\n working = true\n const inputQ = c.req.queries(\"word\")\n const items = (inputQ ?? []).filter(Boolean)\n const inputResult = v.safeParse(inputSchema, items)\n\n if (inputResult.success) {\n const input = inputResult.output\n const embeddings = await client.embed({\n input,\n model: \"voyage-3-lite\",\n })\n\n const zipped = input.map((word, i) => {\n return {\n word,\n embedding: embeddings.data?.at(i)?.embedding,\n }\n })\n\n const data = {\n nodes: zipped.map(z => {\n return {\n id: z.word,\n }\n }),\n links: zipped.flatMap((z, i) => {\n return zipped.slice(i + 1).map(z2 => {\n return {\n source: z.word,\n target: z2.word,\n value: cosineSimilarity(z.embedding, z2.embedding),\n }\n })\n }),\n }\n\n let nodePool = data.nodes.slice()\n let ordered = []\n\n const linkOrder = data.links.sort((l1, l2) => {\n return l2.value - l1.value\n })\n\n console.log(linkOrder)\n for (let link of linkOrder) {\n const s = nodePool.find(n => {\n return n.id === link.source\n })\n const t = nodePool.find(n => n.id === link.target)\n nodePool = nodePool.filter(n => n !== s && n !== t)\n console.log(s, t)\n if (s) ordered.push(s)\n if (t) ordered.push(t)\n }\n\n data.nodes = ordered.concat(nodePool)\n\n setTimeout(() => {\n working = false\n }, THROTTLE)\n\n return c.html(\n \n \n Surprising embeddings\n \n\n \n \n \n

\n Surprising embeddings\n \n Using Voyage AI voyage-3-lite model\n \n

\n \n \n \n

\n\n \n\n {examples}\n\n

{intro}

\n\n \n \n ,\n )\n }\n\n return c.html(\n \n \n Surprising embeddings\n \n\n \n \n \n

\n Surprising embeddings\n \n Using Voyage AI voyage-3-lite model\n \n

\n\n {examples}\n\n \n\n

{intro}

\n \n \n ,\n )\n})\n\nexport default app.fetch","surprising-embeddings",[],[99],{"_103":203,"_95":204,"_154":205,"_156":-5,"_157":139,"_158":206,"_160":161,"_105":207,"_100":208,"_163":164,"_165":209,"_167":22,"_168":-5,"_169":210,"_98":211},"caf606ce-da98-11ef-a3ba-e6cdfca9ef9f","visualization","83d70fb2-da85-11ef-a00c-e6cdfca9ef9f",306,["D",1737753524665],156,"import * as d3 from \"https://cdn.jsdelivr.net/npm/d3@7.9.0/+esm\"\n\nconst width = document.querySelector(\"main\").offsetWidth\nconst dataText = document.querySelector(\"#data\").innerHTML\nconsole.log(dataText)\n\nconst data = JSON.parse(\n dataText,\n)\n// Specify the dimensions of the chart.\nconst height = 700\n\n// Specify the color scale.\nconst color = d3.scaleOrdinal(d3.schemeCategory10)\n\n// The force simulation mutates links and nodes, so create a copy\n// so that re-evaluating this cell produces the same result.\nconst links = data.links.map((d) => ({ ...d }))\nconst nodes = data.nodes.map((d) => ({ ...d }))\n\nconst valueRange = d3.extent(data.links, d => d.value)\nconst scale = d3.scaleLinear().domain(valueRange).range([1, 40])\nconst opacityScale = d3.scaleLinear().domain(valueRange).range([0.5, 1])\nconst distanceScale = d3.scaleLinear().domain(valueRange).range([width, width / 10])\n\nconst positions = Array.from({ length: nodes.length }, (_, i) => {\n return {\n x: (Math.cos((i / nodes.length) * Math.PI * 2) * width / 2.5) + width / 2,\n y: (Math.sin((i / nodes.length) * Math.PI * 2) * width / 2.5) + width / 2,\n }\n})\n\n// Create the SVG container.\nconst svg = d3\n .create(\"svg\")\n .attr(\"width\", \"100%\")\n .attr(\"viewBox\", [0, 0, width, height])\n .attr(\"style\", \"max-width: 100%; aspect-ratio: 1/1;\")\n\n// Add a line for each link, and a circle for each node.\nconst link = svg\n .append(\"g\")\n .selectAll()\n .data(links)\n .join(\"line\")\n .attr(\"stroke-opacity\", d => opacityScale(d.value))\n .attr(\"stroke\", d => {\n return d3.interpolateLab(color(d.source), color(d.target))(0.5)\n })\n .attr(\"stroke-width\", (d) => scale(d.value))\n .attr(\"x1\", d => {\n const idx = nodes.findIndex(n => n.id === d.target)\n return positions[idx].x\n })\n .attr(\"y1\", d => {\n const idx = nodes.findIndex(n => n.id === d.target)\n return positions[idx].y\n })\n .attr(\"x2\", d => {\n const idx = nodes.findIndex(n => n.id === d.source)\n return positions[idx].x\n })\n .attr(\"y2\", d => {\n const idx = nodes.findIndex(n => n.id === d.source)\n return positions[idx].y\n })\n\nconst node = svg\n .append(\"g\")\n .selectAll()\n .data(nodes)\n .join(\"g\")\n .attr(\"transform\", (d, i) => `translate(${positions[i].x}, ${positions[i].y})`)\n\nnode\n .append(\"circle\")\n .attr(\"r\", 50)\n .attr(\"stroke\", \"#fff\")\n .attr(\"stroke-width\", 1.5)\n .attr(\"fill\", (d) => color(d.id))\n\nnode\n .append(\"text\")\n .attr(\"x\", 0)\n .attr(\"text-anchor\", \"middle\")\n .attr(\"font-size\", 14)\n .attr(\"font-weight\", \"bold\")\n .attr(\"fill\", \"white\")\n .attr(\"y\", \"0.31em\")\n .text(d => d.id)\n .clone(true).lower()\n .attr(\"fill\", \"none\")\n .attr(\"stroke\", \"black\")\n .attr(\"stroke-width\", 3)\n\nnode.append(\"title\").text((d) => d.id)\n\nnode.on(\"mouseenter\", e => {\n link.attr(\"opacity\", l => {\n const d = d3.select(e.target).data()[0]\n return (l.source === d.id || l.target === d.id) ? 1 : 0.1\n })\n})\n\nnode.on(\"mouseleave\", d => {\n link.attr(\"opacity\", d => opacityScale(d.value))\n})\n\ndocument.querySelector(\"#root\").appendChild(svg.node())\n\nconsole.log(data)",[],[204],"files",[151,172,181,192,202],"counts",{"_216":32,"_217":119,"_218":32,"_219":119},"pulls","branches","forks","envVars","contents",[],"file",{"_103":193,"_95":99,"_154":194,"_156":-5,"_157":139,"_158":195,"_160":161,"_105":224,"_100":197,"_163":188,"_165":198,"_167":22,"_168":199,"_98":225},["D",1737755227874],[99],"upstreamProject","upstreamBranch","emptyEnvVars","isNew","flags",[],"routes/_app.x.$handle.$name.code.$",{"_93":94,"_95":96,"_97":43,"_100":-7,"_98":99,"_101":234,"_149":239,"_212":255,"_222":256,"_220":258,"_259":-5,"_260":22,"_261":-5,"_262":-5,"_263":-5,"_264":265},{"_103":104,"_95":96,"_105":235,"_107":-5,"_108":22,"_109":110,"_111":112,"_113":-5,"_14":114,"_115":-5,"_116":22,"_117":-5,"_118":119,"_16":120,"_121":122,"_123":122,"_124":125,"_126":125,"_127":125,"_128":122,"_129":122,"_23":236,"_133":237,"_136":238,"_148":22},["D",1737736218273],{"_93":94,"_131":132},{"_93":94,"_135":132},{"_100":138,"_95":43,"_103":139,"_140":104,"_141":-5,"_142":-5,"_143":-5,"_144":-5,"_145":-5,"_146":96,"_147":94},[240,243,246,249,252],{"_103":152,"_95":153,"_154":155,"_156":-5,"_157":139,"_158":159,"_160":161,"_105":162,"_100":119,"_163":164,"_165":166,"_167":22,"_168":-5,"_169":241,"_98":242},[],[153],{"_103":173,"_95":174,"_154":175,"_156":-5,"_157":139,"_158":138,"_160":161,"_105":176,"_100":177,"_163":164,"_165":178,"_167":22,"_168":-5,"_169":244,"_98":245},[],[174],{"_103":182,"_95":183,"_154":184,"_156":-5,"_157":139,"_158":185,"_160":161,"_105":186,"_100":187,"_163":188,"_165":189,"_167":22,"_168":-5,"_169":247,"_98":248},[],[183],{"_103":193,"_95":99,"_154":194,"_156":-5,"_157":139,"_158":195,"_160":161,"_105":196,"_100":197,"_163":188,"_165":198,"_167":22,"_168":199,"_169":250,"_98":251},[],[99],{"_103":203,"_95":204,"_154":205,"_156":-5,"_157":139,"_158":206,"_160":161,"_105":207,"_100":208,"_163":164,"_165":209,"_167":22,"_168":-5,"_169":253,"_98":254},[],[204],[240,243,246,249,252],{"_103":193,"_95":99,"_154":194,"_156":-5,"_157":139,"_158":195,"_160":161,"_105":224,"_100":197,"_163":188,"_165":198,"_167":22,"_168":199,"_98":257},[99],[],"upstream","showCompareCallout","pull","ahead","behind","templates",[],"actionData","errors"]