Earlier this week, Tom MacWright posted Code Search is Hard. He describes the research he his doing to improve the code search experience of Val Town. It was a great read, and you might have seen it trending on Hacker News.

As Val Town's most active user (behind Steve Krouse, one of the founders of Val Town), I for sure agree with Tom that the search feature needs improvements. But while reading his post, I immediately thought of a different approach to the problem. And a few hours later, Val Town Search was born.

How does this new shiny search engine work? Well, it's quite simple.

1. I wrote a Deno script that fetches all vals from the Val Town API.

Create val
#!/usr/bin/env -S deno run -A
import * as path from "https://deno.land/std/path/mod.ts";
const dir = path.join(import.meta.dirname!, "..", "vals");
const blocklist = Deno.readTextFileSync(
  path.join(import.meta.dirname!, "blocklist.txt")
)
  .split("\n")
  .map((line) => line.trim())
  .filter((line) => line.length > 0);

let url = `https://api.val.town/v1/search/vals?limit=100&query=+`;
const vals = [];
while (true) {
  console.log("fetching", url);
  const resp = await fetch(url);
  if (!resp.ok) {
    console.error(resp.statusText);
    Deno.exit(1);
  }

  const { data, links } = await resp.json();
  vals.push(...data);

  if (!links.next) {
    break;
  }

  url = links.next;
}

Deno.removeSync(dir, { recursive: true });
Deno.mkdirSync(dir, { recursive: true });
for (const val of vals) {
  const slug = `${val.author.username}/${val.name}`;
  if (blocklist.includes(slug)) {
    console.log("skipping", slug);
    continue;
  }
  const userDir = path.join(dir, val.author.username);
  Deno.mkdirSync(userDir, { recursive: true });

  Deno.writeTextFileSync(path.join(userDir, `${val.name}.tsx`), val.code);
}

2. I pushed the data to a Github Repository (now private)
3. I added a Github Action that runs the script every hour to refresh the data.

Create val
#!/usr/bin/env -S deno run -A
import * as path from "https://deno.land/std/path/mod.ts";
const dir = path.join(import.meta.dirname!, "..", "vals");
const blocklist = Deno.readTextFileSync(
  path.join(import.meta.dirname!, "blocklist.txt")
)
  .split("\n")
  .map((line) => line.trim())
  .filter((line) => line.length > 0);

let url = `https://api.val.town/v1/search/vals?limit=100&query=+`;
const vals = [];
while (true) {
  console.log("fetching", url);
  const resp = await fetch(url);
  if (!resp.ok) {
    console.error(resp.statusText);
    Deno.exit(1);
  }

  const { data, links } = await resp.json();
  vals.push(...data);

  if (!links.next) {
    break;
  }

  url = links.next;
}

Deno.removeSync(dir, { recursive: true });
Deno.mkdirSync(dir, { recursive: true });
for (const val of vals) {
  const slug = `${val.author.username}/${val.name}`;
  if (blocklist.includes(slug)) {
    console.log("skipping", slug);
    continue;
  }
  const userDir = path.join(dir, val.author.username);
  Deno.mkdirSync(userDir, { recursive: true });

  Deno.writeTextFileSync(path.join(userDir, `${val.name}.tsx`), val.code);
}

4. I created a simple frontend on top of the Github Search API that allows you to search the data. It's hosted on Val Town (obviously).

That was it. I didn't have to build a complex search engine, I just used the tools that were available to me.

Is this a scalable solution for Val Town? Probably not.
Am I abusing the Github API? Maybe.
Does it work better than the current search feature of Val Town? Absolutely!

I hope that the val.town engineers will come up with a search feature that will put my little project to shame. But for now, you won't find a better way to search for vals than Val Town Search.

PS: This post was written / is served from Val Town