Hi

\",\n});\n```\n\n## Val Town Utility Functions\n\nVal Town provides several utility functions to help with common project tasks.\n\n### Importing Utilities\n\nAlways import utilities with version pins to avoid breaking changes:\n\n```ts\nimport {\n parseProject,\n readFile,\n serveFile,\n} from \"https://esm.town/v/std/utils@85-main/index.ts\";\n```\n\n### Available Utilities\n\n#### **serveFile** - Serve project files with proper content types\n\nFor example, in Hono:\n\n```ts\n// serve all files in frontend/ and shared/\napp.get(\"/frontend/*\", (c) => serveFile(c.req.path, import.meta.url));\napp.get(\"/shared/*\", (c) => serveFile(c.req.path, import.meta.url));\n```\n\n#### **readFile** - Read files from within the project:\n\n```ts\n// Read a file from the project\nconst fileContent = await readFile(\"/frontend/index.html\", import.meta.url);\n```\n\n#### **listFiles** - List all files in the project\n\n```ts\nconst files = await listFiles(import.meta.url);\n```\n\n#### **parseProject** - Extract information about the current project from import.meta.url\n\nThis is useful for including info for linking back to a val, ie in \"view source\"\nurls:\n\n```ts\nconst projectVal = parseProject(import.meta.url);\nconsole.log(projectVal.username); // Owner of the project\nconsole.log(projectVal.name); // Project name\nconsole.log(projectVal.version); // Version number\nconsole.log(projectVal.branch); // Branch name\nconsole.log(projectVal.links.self.project); // URL to the project page\n```\n\nHowever, it's _extremely importing_ to note that `parseProject` and other\nStandard Library utilities ONLY RUN ON THE SERVER. If you need access to this\ndata on the client, run it in the server and pass it to the client by splicing\nit into the HTML page or by making an API request for it.\n\n## Val Town Platform Specifics\n\n- **Redirects:** Use\n `return new Response(null, { status: 302, headers: { Location: \"/place/to/redirect\" }})`\n instead of `Response.redirect` which is broken\n- **Images:** Avoid external images or base64 images. Use emojis, unicode\n symbols, or icon fonts/libraries instead\n- **AI Image:** To inline generate an AI image use:\n `

`\n- **Storage:** DO NOT use the Deno KV module for storage\n- **Browser APIs:** DO NOT use the `alert()`, `prompt()`, or `confirm()` methods\n- **Weather Data:** Use open-meteo for weather data (doesn't require API keys)\n unless otherwise specified\n- **View Source:** Add a view source link by importing & using\n `import.meta.url.replace(\"ems.sh\", \"val.town)\"` (or passing this data to the\n client) and include `target=\"_top\"` attribute\n- **Error Debugging:** Add\n `` to HTML to capture\n client-side errors\n- **Error Handling:** Only use try...catch when there's a clear local\n resolution; Avoid catches that merely log or return 500s. Let errors bubble up\n with full context\n- **Environment Variables:** Use `Deno.env.get('keyname')` when you need to, but\n generally prefer APIs that don't require keys\n- **Imports:** Use `https://esm.sh` for npm and Deno dependencies to ensure\n compatibility on server and browser\n- **Storage Strategy:** Only use backend storage if explicitly required; prefer\n simple static client-side sites\n- **React Configuration:** When using React libraries, pin versions with\n `?deps=react@18.2.0,react-dom@18.2.0` and start the file with\n `/** @jsxImportSource https://esm.sh/react@18.2.0 */`\n- Ensure all React dependencies and sub-dependencies are pinned to the same\n version\n- **Styling:** Default to using TailwindCSS via\n `` unless otherwise\n specified\n\n## Project Structure and Design Patterns\n\n### Recommended Directory Structure\n\n```\n├── backend/\n│ ├── database/\n│ │ ├── migrations.ts # Schema definitions\n│ │ ├── queries.ts # DB query functions\n│ │ └── README.md\n│ └── routes/ # Route modules\n│ ├── [route].ts\n│ └── static.ts # Static file serving\n│ ├── index.ts # Main entry point\n│ └── README.md\n├── frontend/\n│ ├── components/\n│ │ ├── App.tsx\n│ │ └── [Component].tsx\n│ ├── favicon.svg\n│ ├── index.html # Main HTML template\n│ ├── index.tsx # Frontend JS entry point\n│ ├── README.md\n│ └── style.css\n├── README.md\n└── shared/\n ├── README.md\n └── utils.ts # Shared types and functions\n```\n\n### Backend (Hono) Best Practices\n\n- Hono is the recommended API framework\n- Main entry point should be `backend/index.ts`\n- **Static asset serving:** Use the utility functions to read and serve project\n files:\n ```ts\n import {\n readFile,\n serveFile,\n } from \"https://esm.town/v/std/utils@85-main/index.ts\";\n\n // serve all files in frontend/ and shared/\n app.get(\"/frontend/*\", (c) => serveFile(c.req.path, import.meta.url));\n app.get(\"/shared/*\", (c) => serveFile(c.req.path, import.meta.url));\n\n // For index.html, often you'll want to bootstrap with initial data\n app.get(\"/\", async (c) => {\n let html = await readFile(\"/frontend/index.html\", import.meta.url);\n\n // Inject data to avoid extra round-trips\n const initialData = await fetchInitialData();\n const dataScript = ``;\n\n html = html.replace(\"\", `${dataScript}`);\n return c.html(html);\n });\n ```\n- Create RESTful API routes for CRUD operations\n- Always include this snippet at the top-level Hono app to re-throwing errors to\n see full stack traces:\n ```ts\n // Unwrap Hono errors to see original error details\n app.onError((err, c) => {\n throw err;\n });\n ```\n\n### Database Patterns\n\n- Run migrations on startup or comment out for performance\n- Change table names when modifying schemas rather than altering\n- Export clear query functions with proper TypeScript typing\n\n## Common Gotchas and Solutions\n\n1. **Environment Limitations:**\n - Val Town runs on Deno in a serverless context, not Node.js\n - Code in `shared/` must work in both frontend and backend environments\n - Cannot use `Deno` keyword in shared code\n - Use `https://esm.sh` for imports that work in both environments\n\n2. **SQLite Peculiarities:**\n - Limited support for ALTER TABLE operations\n - Create new tables with updated schemas and copy data when needed\n - Always run table creation before querying\n\n3. **React Configuration:**\n - All React dependencies must be pinned to 18.2.0\n - Always include `@jsxImportSource https://esm.sh/react@18.2.0` at the top of\n React files\n - Rendering issues often come from mismatched React versions\n\n4. **File Handling:**\n - Val Town only supports text files, not binary\n - Use the provided utilities to read files across branches and forks\n - For files in the project, use `readFile` helpers\n\n5. **API Design:**\n - `fetch` handler is the entry point for HTTP vals\n - Run the Hono app with\n `export default app.fetch // This is the entry point for HTTP vals`",[],[287],"09c77ebc-ecab-11f0-ab70-42dde27851f2",".vtignore","019b9e55-c441-734d-8f29-de7ac3c2c2f3",["D",1767887979503],".git\n.vscode\n.cursorrules\n.DS_Store\nnode_modules\nvendor",[],[294],"09c76706-ecab-11f0-ab70-42dde27851f2","schema.ts","019b9e55-c441-734d-8f29-d4ced08f2322","019b9e55-c441-734d-8f29-dac8d0c4f41c",["D",1767887979503],"/**\n * Drizzle ORM Schema for Claude Plugins API\n */\nimport { sql } from \"npm:drizzle-orm@0.44.2\";\nimport { integer, sqliteTable, text } from \"npm:drizzle-orm@0.44.2/sqlite-core\";\n\n/**\n * Plugins table - stores all registered plugins\n */\nexport const plugins = sqliteTable(\"plugins\", {\n id: text(\"id\").primaryKey(),\n name: text(\"name\").notNull(),\n namespace: text(\"namespace\").notNull(),\n gitUrl: text(\"git_url\").notNull(),\n\n // Search fields\n description: text(\"description\"),\n keywords: text(\"keywords\"), // JSON stringified array\n category: text(\"category\"),\n skills: text(\"skills\"), // JSON stringified array\n\n // Display metadata\n version: text(\"version\").notNull().default(\"1.0.0\"),\n author: text(\"author\").notNull(), // Simple string, no complex object\n\n // All other marketplace data as single JSON blob\n metadata: text(\"metadata\"), // JSON: { homepage, license, commands, agents, mcpServers, etc. }\n\n // GitHub signals\n stars: integer(\"stars\").notNull().default(0),\n\n // Plugin status\n verified: integer(\"verified\", { mode: \"boolean\" }).notNull().default(false),\n downloads: integer(\"downloads\").notNull().default(0),\n\n // Timestamps\n createdAt: text(\"created_at\").notNull().default(sql`CURRENT_TIMESTAMP`),\n updatedAt: text(\"updated_at\").notNull().default(sql`CURRENT_TIMESTAMP`),\n});\n\n/**\n * Plugin stats table - tracks download statistics\n */\nexport const pluginStats = sqliteTable(\"plugin_stats\", {\n pluginId: text(\"plugin_id\")\n .primaryKey()\n .references(() => plugins.id),\n downloadsTotal: integer(\"downloads_total\").notNull().default(0),\n downloadsWeek: integer(\"downloads_week\").notNull().default(0),\n downloadsMonth: integer(\"downloads_month\").notNull().default(0),\n lastDownloadedAt: text(\"last_downloaded_at\"),\n});\n\n/**\n * Search Buckets tabs - stores all search buckets\n */\nexport const searchBuckets = sqliteTable(\"search_buckets\", {\n id: text(\"id\").primaryKey(),\n name: text(\"name\").notNull(),\n buckets: text(\"buckets\").notNull(), // JSON stringified array\n\n // Timestamps\n createdAt: text(\"created_at\").notNull().default(sql`CURRENT_TIMESTAMP`),\n updatedAt: text(\"updated_at\").notNull().default(sql`CURRENT_TIMESTAMP`),\n});\n\n/**\n * Skills table - stores all registered Claude Code Skills\n */\nexport const skills = sqliteTable(\"skills\", {\n id: text(\"id\").primaryKey(),\n name: text(\"name\").notNull(),\n namespace: text(\"namespace\").notNull(), // @owner/repo/skillName\n sourceUrl: text(\"source_url\").notNull(), // GitHub tree URL\n sourceType: text(\"source_type\").notNull().default(\"github\"), // \"github\" | \"r2\"\n\n // From SKILL.md YAML frontmatter\n description: text(\"description\").notNull(), // Max 200 chars\n version: text(\"version\"),\n dependencies: text(\"dependencies\"),\n\n // Metadata\n author: text(\"author\").notNull(),\n stars: integer(\"stars\").notNull().default(0),\n installs: integer(\"installs\").notNull().default(0),\n verified: integer(\"verified\", { mode: \"boolean\" }).notNull().default(false),\n deleted: integer(\"deleted\", { mode: \"boolean\" }).notNull().default(false),\n\n // JSON blob for extra data\n metadata: text(\"metadata\"), // {repoOwner, repoName, directoryPath, rawFileUrl, license, ...}\n\n // Timestamps\n createdAt: text(\"created_at\").notNull().default(sql`CURRENT_TIMESTAMP`),\n updatedAt: text(\"updated_at\").notNull().default(sql`CURRENT_TIMESTAMP`),\n firstCommitAt: text(\"first_commit_at\"), // When SKILL.md was first committed (for deduplication)\n contentHash: text(\"content_hash\"), // SHA-256 hash of SKILL.md content (for duplicate detection)\n});\n\n/**\n * Skill stats table - tracks install statistics\n */\nexport const skillStats = sqliteTable(\"skill_stats\", {\n skillId: text(\"skill_id\")\n .primaryKey()\n .references(() => skills.id),\n installsTotal: integer(\"installs_total\").notNull().default(0),\n installsWeek: integer(\"installs_week\").notNull().default(0),\n installsMonth: integer(\"installs_month\").notNull().default(0),\n lastInstalledAt: text(\"last_installed_at\"),\n});\n\n// Type exports for TypeScript\nexport type Plugin = typeof plugins.$inferSelect;\nexport type NewPlugin = typeof plugins.$inferInsert;\n\nexport type PluginStats = typeof pluginStats.$inferSelect;\nexport type NewPluginStats = typeof pluginStats.$inferInsert;\n\nexport type Skill = typeof skills.$inferSelect;\nexport type NewSkill = typeof skills.$inferInsert;\n\nexport type SkillStats = typeof skillStats.$inferSelect;\nexport type NewSkillStats = typeof skillStats.$inferInsert;",[],[308,301],"database","09c7635a-ecab-11f0-ab70-42dde27851f2","run-migrations.ts","019b9e55-c441-734d-8f29-d3442fe746fc",["D",1767887979503],"import { runMigrations } from \"./db.ts\";\n\n(async function run() {\n try {\n await runMigrations();\n console.log(\"Migrations completed successfully\");\n } catch (error) {\n console.error(\"Error running migrations:\", error);\n throw error;\n }\n})();",[],[308,310],"09c73fa6-ecab-11f0-ab70-42dde27851f2","db.ts","019b9e55-c441-734d-8f29-cd503c7ba9ad",["D",1767887979503],"/**\n * Database connection and utilities\n */\nimport { sqlite } from \"https://esm.town/v/std/sqlite\";\nimport { sql } from \"npm:drizzle-orm@0.44.2\";\nimport { drizzle } from \"npm:drizzle-orm@0.44.2/libsql\";\nimport * as schema from \"./schema.ts\";\n\n// Initialize Drizzle with Val Town SQLite (without schema to avoid circular refs)\nexport const db = drizzle(sqlite as any);\n\nasync function runCreateSearchBuckets() {\n console.log(\"Running create search buckets\");\n try {\n // Check if tables exist by attempting to query them\n const tableCheck = await db\n .select({ count: sql`count(*)` })\n .from(schema.searchBuckets)\n .limit(1)\n .catch(() => null);\n\n if (tableCheck === null) {\n console.log(\"Creating table for Search Buckets...\");\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS search_buckets (\n id TEXT PRIMARY KEY,\n name TEXT NOT NULL,\n buckets TEXT,\n created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,\n updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP\n )\n `);\n console.log(\"Search buckets created successfully\");\n }\n } catch (e) {\n console.error(\"Table creation failed\");\n throw e;\n }\n}\n\n/**\n * Run database migrations to create tables and indexes\n */\nasync function runCreateTablesAndIndexes() {\n console.log(\"Running create tables and indexes...\");\n\n try {\n // Check if tables exist by attempting to query them\n const tableCheck = await db\n .select({ count: sql`count(*)` })\n .from(schema.plugins)\n .limit(1)\n .catch(() => null);\n\n if (tableCheck === null) {\n // Tables don't exist, create them using schema-driven SQL\n console.log(\"Creating database schema...\");\n\n // Create tables using Drizzle schema definitions\n // This ensures schema and database stay in sync\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS plugins (\n id TEXT PRIMARY KEY,\n name TEXT NOT NULL,\n namespace TEXT NOT NULL,\n git_url TEXT NOT NULL,\n description TEXT,\n keywords TEXT,\n category TEXT,\n version TEXT NOT NULL DEFAULT '1.0.0',\n author TEXT NOT NULL,\n metadata TEXT,\n stars INTEGER NOT NULL DEFAULT 0,\n verified INTEGER NOT NULL DEFAULT 0,\n downloads INTEGER NOT NULL DEFAULT 0,\n first_commit_at TEXT,\n content_hash TEXT,\n created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,\n updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP\n )\n `);\n\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS plugin_stats (\n plugin_id TEXT PRIMARY KEY,\n downloads_total INTEGER NOT NULL DEFAULT 0,\n downloads_week INTEGER NOT NULL DEFAULT 0,\n downloads_month INTEGER NOT NULL DEFAULT 0,\n last_downloaded_at TEXT,\n FOREIGN KEY (plugin_id) REFERENCES plugins(id) ON DELETE CASCADE\n )\n `);\n\n // Create indexes for query performance\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_plugins_namespace ON plugins(namespace)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_plugins_name ON plugins(name)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_plugins_category ON plugins(category)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_plugins_content_hash ON plugins(content_hash)`,\n );\n await sqlite.execute(\n `CREATE UNIQUE INDEX IF NOT EXISTS idx_plugins_namespace_name ON plugins(namespace, name)`,\n );\n\n console.log(\"Schema created successfully\");\n }\n\n console.log(\"Creating database completed successfully\");\n } catch (error) {\n console.error(\"Migration failed:\", error);\n throw error;\n }\n}\n\nasync function runAddSkillsMigration(): Promise {\n console.log(\"Running migration: add `skills` column to `plugins`…\");\n const pragma = await sqlite.execute(`PRAGMA table_info(plugins);`);\n const hasSkills = Array.isArray(pragma?.rows)\n ? pragma.rows.some((r: any) => r.name === \"skills\")\n : false;\n\n if (hasSkills) {\n console.log(\"Column `skills` already exists. Skipping ALTER TABLE.\");\n } else {\n console.log(\"Adding `skills` column (TEXT, nullable) to `plugins`…\");\n await sqlite.execute(`ALTER TABLE plugins ADD COLUMN skills TEXT;`);\n console.log(\"Column `skills` added.\");\n }\n\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_plugins_skills ON plugins(skills);`,\n );\n\n console.log(\"Migration completed: add `skills` column.\");\n}\n\nasync function runAddSkillsTableMigration(): Promise {\n console.log(\"Running migration: create skills and skill_stats tables...\");\n\n try {\n // Check if skills table exists\n const skillsTableCheck = await sqlite\n .execute(\n `SELECT name FROM sqlite_master WHERE type='table' AND name='skills';`,\n )\n .catch(() => null);\n\n if (!skillsTableCheck || skillsTableCheck.rows.length === 0) {\n console.log(\"Creating skills table...\");\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS skills (\n id TEXT PRIMARY KEY,\n name TEXT NOT NULL,\n namespace TEXT NOT NULL,\n source_url TEXT NOT NULL,\n source_type TEXT NOT NULL DEFAULT 'github',\n description TEXT NOT NULL,\n version TEXT,\n dependencies TEXT,\n author TEXT NOT NULL,\n stars INTEGER NOT NULL DEFAULT 0,\n installs INTEGER NOT NULL DEFAULT 0,\n verified INTEGER NOT NULL DEFAULT 0,\n metadata TEXT,\n first_commit_at TEXT,\n content_hash TEXT,\n created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,\n updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP\n )\n `);\n\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS skill_stats (\n skill_id TEXT PRIMARY KEY,\n installs_total INTEGER NOT NULL DEFAULT 0,\n installs_week INTEGER NOT NULL DEFAULT 0,\n installs_month INTEGER NOT NULL DEFAULT 0,\n last_installed_at TEXT,\n FOREIGN KEY (skill_id) REFERENCES skills(id) ON DELETE CASCADE\n )\n `);\n\n // Create indexes\n await sqlite.execute(\n `CREATE UNIQUE INDEX IF NOT EXISTS idx_skills_namespace ON skills(namespace)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_skills_name ON skills(name)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_skills_author ON skills(author)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_skills_content_hash ON skills(content_hash)`,\n );\n\n console.log(\"Skills tables created successfully\");\n } else {\n console.log(\"Skills table already exists. Skipping.\");\n }\n } catch (error) {\n console.error(\"Skills table migration failed:\", error);\n throw error;\n }\n\n console.log(\"Migration completed: skills tables\");\n}\n\nasync function runAddDeletedColumnMigration(): Promise {\n console.log(\"Running migration: add `deleted` and `deleted_at` to skills...\");\n try {\n const skillsTableCheck = await sqlite\n .execute(\n `SELECT name FROM sqlite_master WHERE type='table' AND name='skills';`,\n )\n .catch(() => null);\n\n if (!skillsTableCheck) {\n throw new Error(\"Run runAddSkillsTableMigration before this migration\");\n }\n\n const pragma = await sqlite.execute(`PRAGMA table_info(skills);`);\n const cols = new Set(\n (pragma?.rows ?? []).map((r: { name: string }) =>\n (r?.name ?? \"\").toLowerCase()\n ),\n );\n if (!cols.has(\"deleted\")) {\n console.log(\"Adding `deleted` column to skills...\");\n await sqlite.execute(\n `ALTER TABLE skills ADD COLUMN deleted INTEGER NOT NULL DEFAULT 0;`,\n );\n } else {\n console.log(\"`deleted` column already exists. Skipping.\");\n }\n } catch (e) {\n console.error(\"Skills table migration failed:\", error);\n throw error;\n }\n}\n\nexport async function runMigrations(): Promise {\n // await runCreateTablesAndIndexes();\n // await runAddSkillsMigration();\n // await runAddSkillsTableMigration();\n // await runCreateSearchBuckets()\n await runAddDeletedColumnMigration();\n}\n",[],[308,317],"09c77d5e-ecab-11f0-ab70-42dde27851f2",["D",1767887979503],[240,241,242],[308],"09c6fec4-ecab-11f0-ab70-42dde27851f2","skills-indexer.ts","019b9e55-c441-734d-8f29-c4323ccefae0","019b9e55-c441-734d-8f29-cab74d2ac8c5",["D",1767887979503],"interval","import { parseSkillMd, transformToDbSkill } from \"./skill-parser.ts\";\nimport { Bucket, GitHubClient } from \"../clients/github.ts\";\nimport { db } from \"../database/db.ts\";\nimport { skills, skillStats } from \"../database/schema.ts\";\nimport { eq, sql } from \"npm:drizzle-orm@0.44.2\";\nimport { blob } from \"https://esm.town/v/std/blob\";\nimport {\n type Logger,\n withLogger,\n} from \"https://esm.town/v/kamalnrf/axiom-logger/standalone.ts\";\n\nconst IGNORE_DEDUPLICATE_NAMESPACES = [\n \"@anthropics/skills/mcp-builder\",\n \"@anthropics/claude-code/frontend-design\",\n \"@steveyegge/beads/beads\",\n \"@openshift/hypershift/effective-go\",\n];\n\n/**\n * Compute SHA-256 hash of content\n */\nasync function computeHash(content: string): Promise {\n const encoder = new TextEncoder();\n const data = encoder.encode(content);\n const hashBuffer = await crypto.subtle.digest(\"SHA-256\", data);\n const hashArray = Array.from(new Uint8Array(hashBuffer));\n return hashArray.map((b) => b.toString(16).padStart(2, \"0\")).join(\"\");\n}\n\ntype SkillsIndexerRunState = {\n lastIndexProcessed: number; // exclusive end of last batch (next run starts here)\n batchSize: number;\n totalBuckets: number;\n sortedBy: \"count_desc_min_asc\";\n startedAt: string;\n completedAt?: string;\n};\nconst LAST_RUN_STATE_KEY = \"skills-indexer:last-run-state\";\nexport async function getLastRunState(): Promise {\n try {\n return await blob.getJSON(LAST_RUN_STATE_KEY);\n } catch {\n return null;\n }\n}\n\nfunction computeEndIndexByItemCount(\n buckets: Array,\n start: number,\n maxItems: number,\n): number {\n let total = 0;\n let i = start;\n while (i < buckets.length && total + buckets[i].count <= maxItems) {\n total += buckets[i].count;\n i++;\n }\n // If the very first bucket exceeds maxItems, still process at least one bucket\n return i === start ? Math.min(start + 1, buckets.length) : i;\n}\n\nexport async function setLastRunState(\n state: SkillsIndexerRunState,\n): Promise {\n await blob.setJSON(LAST_RUN_STATE_KEY, state);\n}\n\nconst github = new GitHubClient();\n\ntype IndexSkillsArgs = {\n minSize: number;\n maxSize: number;\n};\n\nasync function indexSkills({ minSize, maxSize }: IndexSkillsArgs, log: Logger) {\n let indexed = 0;\n let invalid = 0;\n let failed = 0;\n let skippedDuplicates = 0;\n\n // Search for SKILL.md files in skills/ directories (exclude archived repos)\n const searchQuery =\n `filename:SKILL.md language:Markdown \"---\" \"name:\" \"description:\" size:${minSize}..${maxSize}`;\n log.info(`Searching: ${searchQuery}`);\n\n const firstPage = await github.searchCode(searchQuery);\n log.info(`Found ${firstPage.total_count} SKILL.md files`);\n\n // Calculate total pages (max 10 pages = 1000 results)\n const maxPages = Math.min(Math.ceil(firstPage.total_count / 100), 10);\n if (firstPage.total_count > 1000) {\n log.warn(\n `Found ${firstPage.total_count} results. GitHub limits to 1000, only first 1000 will be indexed.`,\n );\n }\n\n // Process all pages\n for (let page = 1; page <= maxPages; page++) {\n log.info(`\\nProcessing page ${page}/${maxPages}...`);\n\n const searchResults = page === 1\n ? firstPage\n : await github.searchCode(searchQuery, page);\n\n log.info(`Processing ${searchResults.items.length} items...`);\n\n for (const item of searchResults.items) {\n try {\n // Fetch SKILL.md content\n const content = await github.fetchFileContent(\n item.repository.full_name,\n item.path,\n );\n\n // Compute content hash for deduplication\n const contentHash = await computeHash(content);\n\n // Parse and validate SKILL.md\n const skillData = parseSkillMd(content, item.repository.full_name);\n if (!skillData) {\n log.warn(\n `Invalid SKILL.md from ${item.repository.full_name}/${item.path}`,\n );\n invalid++;\n continue;\n }\n\n // Fetch repository metadata (stars)\n const repoMetadata = await github.fetchRepoMetadata(\n item.repository.full_name,\n );\n\n // Extract directory path (remove SKILL.md from path)\n const directoryPath = item.path.replace(/\\/SKILL\\.md$/, \"\");\n\n // Build raw file URL\n const rawFileUrl =\n `https://raw.githubusercontent.com/${item.repository.full_name}/${repoMetadata.default_branch}/${item.path}`;\n\n // Extract repo name from full_name\n const [repoOwner, repoName] = item.repository.full_name.split(\"/\");\n\n // Skip forks\n if (repoMetadata.fork) {\n log.info(`Skipping forked repository ${repoOwner}/${repoName}`);\n continue;\n }\n\n // Build namespace for this skill\n const skillDirName =\n item.path.replace(/\\/SKILL\\.md$/, \"\").split(\"/\").filter(Boolean)\n .pop() || \"unknown\";\n const namespace = `@${repoOwner}/${repoName}/${skillDirName}`;\n\n // Check for existing skill with same namespace\n const existing = await db\n .select()\n .from(skills)\n .where(eq(skills.namespace, namespace))\n .get();\n\n // If there is no change in a skills hash, skip\n if (existing.contentHash === contentHash) {\n indexed++;\n continue;\n }\n\n // Check for duplicate content (same hash, different namespace)\n // Only check if contentHash is valid (not null)\n let duplicateByHash = null;\n if (contentHash) {\n duplicateByHash = await db\n .select()\n .from(skills)\n .where(\n sql`${skills.contentHash} = ${contentHash} AND ${skills.namespace} != ${namespace} AND ${skills.deleted} = 0`,\n )\n .get();\n }\n\n // Fetch first commit date (needed for dedup comparison)\n let firstCommitAt: string | null = existing?.firstCommitAt || null;\n if (!firstCommitAt) {\n firstCommitAt = await github.fetchFirstCommitDate(\n item.repository.full_name,\n item.path,\n );\n if (firstCommitAt) {\n log.info(\n `First commit for ${item.repository.full_name}/${item.path}: ${firstCommitAt}`,\n );\n }\n }\n\n const isIgnored = IGNORE_DEDUPLICATE_NAMESPACES.includes(namespace);\n\n // Handle duplicate detection\n if (duplicateByHash && firstCommitAt && !isIgnored) {\n const duplicateFirstCommit = duplicateByHash.firstCommitAt;\n\n if (duplicateFirstCommit) {\n const currentDate = new Date(firstCommitAt);\n const duplicateDate = new Date(duplicateFirstCommit);\n\n if (currentDate > duplicateDate) {\n // This skill is newer than existing duplicate - mark as deleted and skip\n log.info(\n `Duplicate detected: ${namespace} (${firstCommitAt}) is newer than ${duplicateByHash.namespace} (${duplicateFirstCommit}). Marking as deleted.`,\n );\n\n // Still upsert but mark as deleted\n const dbSkill = transformToDbSkill(\n skillData,\n repoOwner,\n repoName,\n directoryPath,\n item.repository.html_url,\n repoMetadata.stargazers_count,\n rawFileUrl,\n repoMetadata.default_branch,\n firstCommitAt,\n contentHash,\n );\n\n const skillId = existing?.id || dbSkill.id;\n\n await db\n .insert(skills)\n .values({\n ...dbSkill,\n id: skillId,\n deleted: true,\n })\n .onConflictDoUpdate({\n target: [skills.namespace],\n set: {\n deleted: sql`1`,\n contentHash: sql`excluded.content_hash`,\n firstCommitAt:\n sql`COALESCE(excluded.first_commit_at, skills.first_commit_at)`,\n updatedAt: sql`CURRENT_TIMESTAMP`,\n },\n })\n .run();\n\n skippedDuplicates++;\n continue;\n } else {\n // This skill is older - mark the other one as deleted\n log.info(\n `Duplicate detected: ${duplicateByHash.namespace} (${duplicateFirstCommit}) is newer than ${namespace} (${firstCommitAt}). Marking duplicate as deleted.`,\n );\n\n await db\n .update(skills)\n .set({\n deleted: true,\n updatedAt: sql`CURRENT_TIMESTAMP`,\n })\n .where(eq(skills.id, duplicateByHash.id))\n .run();\n }\n }\n }\n\n // Transform to database format\n const dbSkill = transformToDbSkill(\n skillData,\n repoOwner,\n repoName,\n directoryPath,\n item.repository.html_url,\n repoMetadata.stargazers_count,\n rawFileUrl,\n repoMetadata.default_branch,\n firstCommitAt,\n contentHash,\n );\n\n // Use existing ID if skill already exists\n const skillId = existing?.id || dbSkill.id;\n\n // Upsert skill\n await db\n .insert(skills)\n .values({\n ...dbSkill,\n id: skillId,\n deleted: false,\n })\n .onConflictDoUpdate({\n target: [skills.namespace],\n set: {\n name: sql`excluded.name`,\n description: sql`excluded.description`,\n version: sql`excluded.version`,\n dependencies: sql`excluded.dependencies`,\n sourceUrl: sql`excluded.source_url`,\n stars: sql`excluded.stars`,\n metadata: sql`excluded.metadata`,\n updatedAt: sql`CURRENT_TIMESTAMP`,\n firstCommitAt:\n sql`COALESCE(excluded.first_commit_at, skills.first_commit_at)`,\n contentHash: sql`excluded.content_hash`,\n },\n })\n .run();\n\n // Ensure stats entry exists\n await db\n .insert(skillStats)\n .values({ skillId })\n .onConflictDoNothing()\n .run();\n\n indexed++;\n } catch (error) {\n log.error(\n `Failed to process ${item.repository.full_name}/${item.path}:`,\n error,\n );\n failed++;\n }\n }\n }\n\n return {\n indexed,\n invalid,\n failed,\n skippedDuplicates,\n };\n}\n\nexport default withLogger({\n valName: \"skills-indexer\",\n console: \"warn\",\n type: \"CRON\",\n}, async (log) => {\n const startTime = Date.now();\n\n try {\n log.info(\"GitHub Skills indexer started\");\n const skillsSearchBuckets = await blob.getJSON(\n \"skills-search-buckets\",\n );\n log.info(\"skillSearchBuckets =>\", { skillsSearchBuckets });\n const sortedBuckets = skillsSearchBuckets.sort((a, b) => {\n if (b.count !== a.count) return b.count - a.count;\n return a.min - b.min;\n });\n\n const BATCH_SIZE = 2000;\n const prevState = await getLastRunState();\n const startIdx = Math.min(\n Math.max(prevState?.lastIndexProcessed ?? 0, 0),\n sortedBuckets.length,\n );\n const endIdx = computeEndIndexByItemCount(\n sortedBuckets,\n startIdx,\n BATCH_SIZE,\n );\n\n const runState: SkillsIndexerRunState = {\n lastIndexProcessed: endIdx,\n batchSize: BATCH_SIZE,\n totalBuckets: sortedBuckets.length,\n sortedBy: \"count_desc_min_asc\",\n startedAt: (new Date(startTime)).toISOString(),\n };\n console.log(\"Run State =>\", runState);\n await setLastRunState(runState);\n const currentBatch = sortedBuckets.slice(startIdx, endIdx);\n console.log(\n `Processing buckets ${startIdx}..${endIdx} of ${sortedBuckets.length} (batch size ${currentBatch.length})`,\n );\n log.info(\n `Processing buckets ${startIdx}..${endIdx} of ${sortedBuckets.length} (batch size ${currentBatch.length})`,\n );\n\n let indexed = 0;\n let invalid = 0;\n let failed = 0;\n let skippedDuplicates = 0;\n\n const results = await Promise.all(\n currentBatch.map(async ({ min, max }) => {\n try {\n return await indexSkills({ minSize: min, maxSize: max }, log);\n } catch (e) {\n console.error(\"indexSkills failed:\", e);\n return { indexed: 0, invalid: 0, failed: 1, skippedDuplicates: 0 };\n }\n }),\n );\n\n for (const res of results) {\n indexed += res.indexed;\n invalid += res.invalid;\n failed += res.failed;\n skippedDuplicates += res.skippedDuplicates;\n }\n\n const duration = ((Date.now() - startTime) / 1000).toFixed(2);\n console.log(\n `Skills indexer completed in ${duration}s: ✓ Indexed ${indexed} skills, ${invalid} invalid, ${failed} failed, ${skippedDuplicates} duplicates marked.`,\n );\n\n // Update Final Run State\n const completedState: SkillsIndexerRunState = {\n ...runState,\n completedAt: new Date().toISOString(),\n };\n\n if (endIdx >= sortedBuckets.length) {\n const resetState: SkillsIndexerRunState = {\n ...completedState,\n lastIndexProcessed: 0,\n };\n await setLastRunState(resetState);\n log.info(\"All buckets processed. Checkpoint reset to 0.\");\n } else {\n await setLastRunState(completedState);\n }\n } catch (e) {\n log.error(\"Skills indexer CRON job failed:\", e);\n throw e;\n }\n});",[],[336,328],"cron","09c6e812-ecab-11f0-ab70-42dde27851f2","skill-parser.ts","019b9e55-c441-734d-8f29-c0400099bfdc",["D",1767887979503],"import { z } from \"npm:zod@4.1.12\";\n\n/**\n * Zod schema for SKILL.md YAML frontmatter\n * Based on Claude Code Skills documentation\n */\nconst SkillFrontmatterSchema = z\n .object({\n name: z.string().min(1).max(64), // Required, max 64 chars\n description: z.string().min(1),\n version: z.string().optional(),\n dependencies: z.string().optional(),\n })\n .passthrough(); // Allow other fields\n\nexport type SkillFrontmatter = z.infer;\n\n/**\n * Parse and validate SKILL.md content\n * Extracts YAML frontmatter and validates required fields\n */\nexport function parseSkillMd(\n content: string,\n repoFullName: string,\n): SkillFrontmatter | null {\n try {\n // Extract YAML frontmatter\n const frontmatterMatch = content.match(/^---\\n([\\s\\S]*?)\\n---/);\n\n if (!frontmatterMatch) {\n console.warn(\n `No YAML frontmatter found in SKILL.md from ${repoFullName}`,\n );\n return null;\n }\n\n const yamlContent = frontmatterMatch[1];\n\n // Simple YAML parser (handles basic key: value pairs)\n const frontmatter: Record = {};\n const lines = yamlContent.split(\"\\n\");\n\n for (const line of lines) {\n const match = line.match(/^(\\w+):\\s*(.+)$/);\n if (match) {\n const [, key, value] = match;\n // Remove quotes if present\n frontmatter[key] = value.replace(/^[\"']|[\"']$/g, \"\").trim();\n }\n }\n\n // Validate with Zod\n const result = SkillFrontmatterSchema.safeParse(frontmatter);\n\n if (!result.success) {\n console.error(`Invalid SKILL.md frontmatter from ${repoFullName}:`);\n console.error(\n `Validation errors: ${JSON.stringify(result.error.errors, null, 2)}`,\n );\n return null;\n }\n\n return result.data;\n } catch (error) {\n console.error(\n `Failed to parse SKILL.md from ${repoFullName}:`,\n error instanceof Error ? error.message : String(error),\n );\n return null;\n }\n}\n\n/**\n * Transform skill data into database record\n */\nexport function transformToDbSkill(\n skillData: SkillFrontmatter,\n repoOwner: string,\n repoName: string,\n directoryPath: string,\n repoUrl: string,\n repoStars: number,\n rawFileUrl: string,\n defaultBranch: string = \"main\",\n firstCommitAt: string | null = null,\n contentHash: string | null = null,\n): {\n id: string;\n name: string;\n namespace: string;\n sourceUrl: string;\n sourceType: string;\n description: string;\n version: string | null;\n dependencies: string | null;\n author: string;\n stars: number;\n installs: number;\n verified: boolean;\n metadata: string;\n createdAt: string;\n updatedAt: string;\n firstCommitAt: string | null;\n contentHash: string | null;\n} {\n // Extract skill directory name from path\n const skillDirName = directoryPath.split(\"/\").filter(Boolean).pop() ||\n \"unknown\";\n\n // Build namespace: @owner/repo/skillName\n const namespace = `@${repoOwner}/${repoName}/${skillDirName}`;\n\n // Build source URL (GitHub tree URL to skill directory)\n const sourceUrl = `${repoUrl}/tree/${defaultBranch}/${directoryPath}`;\n\n // Metadata blob\n const metadata = {\n repoOwner,\n repoName,\n directoryPath,\n rawFileUrl,\n };\n\n return {\n id: crypto.randomUUID(),\n name: skillData.name,\n namespace,\n sourceUrl,\n sourceType: \"github\",\n description: skillData.description,\n version: skillData.version || null,\n dependencies: skillData.dependencies || null,\n author: repoOwner,\n stars: repoStars,\n installs: 0,\n verified: false,\n metadata: JSON.stringify(metadata),\n createdAt: new Date().toISOString(),\n updatedAt: new Date().toISOString(),\n firstCommitAt,\n contentHash,\n };\n}",[],[336,338],"09c6da98-ecab-11f0-ab70-42dde27851f2","search-buckets.ts","019b9e55-c441-734d-8f29-bf3709843da4",["D",1767887979503],"import { GitHubClient } from \"../clients/github.ts\";\nimport { blob } from \"https://esm.town/v/std/blob\";\n\nconst MIN_SIZE = 500;\nconst MAX_SIZE = 150_000;\nconst BASE_QUERY =\n `filename:SKILL.md language:Markdown \"---\" \"name:\" \"description:\"`;\nconst BUCKET_NAME = \"skills-search-buckets\";\n\nconst MAX_PER_BUCKET = 950;\nconst MIN_SPAN = 200;\n\n// Helper to read previously stored buckets (returns [] if none)\nasync function getStoredBuckets(): Promise<\n Array<{ min: number; max: number; count: number }>\n> {\n try {\n const existing = await blob.getJSON<\n Array<{ min: number; max: number; count: number }>\n >(BUCKET_NAME);\n return Array.isArray(existing) ? existing : [];\n } catch {\n return [];\n }\n}\n\nexport default async function (interval: Interval) {\n const startTime = Date.now();\n\n try {\n console.log(\"GitHub Search Buckets for skills started\");\n const github = new GitHubClient();\n\n // 1) Load stored buckets. If empty, create initial buckets for full range.\n let buckets = await getStoredBuckets();\n if (buckets.length === 0) {\n console.log(\n \"No stored buckets found. Creating initial buckets from full size range.\",\n );\n buckets = await github.findBuckets(BASE_QUERY, {\n min: MIN_SIZE,\n max: MAX_SIZE,\n maxPerBucket: MAX_PER_BUCKET,\n minSpan: MIN_SPAN,\n });\n } else {\n console.log(\n `Loaded ${buckets.length} stored buckets. Refining oversized ones if needed…`,\n );\n }\n\n // Sort by Size Range\n buckets.sort((a, b) => (b.max - b.min) - (a.max - a.min));\n console.log(\"Existing Buckets =>\", buckets);\n\n // 2) Iterate stored (or newly created) buckets.\n // For any bucket with count > MAX_PER_BUCKET, reuse findBuckets over that bucket’s range to downsize.\n const refined: Array<{ min: number; max: number; count: number }> = [];\n for (const b of buckets) {\n const subs = await github.findBuckets(BASE_QUERY, {\n min: b.min,\n max: b.max,\n maxPerBucket: MAX_PER_BUCKET,\n minSpan: MIN_SPAN,\n });\n refined.push(...subs);\n }\n\n // 3) Persist the updated buckets\n await blob.setJSON(BUCKET_NAME, refined);\n\n const duration = ((Date.now() - startTime) / 1000).toFixed(2);\n console.log(\n `Search Buckets completed in ${duration}s, and created ${refined.length} buckets`,\n );\n } catch (e) {\n console.error(\"Skills indexer CRON job failed:\", e);\n throw e;\n }\n}",[],[336,345],"09c6d21e-ecab-11f0-ab70-42dde27851f2","refresh-skills.ts","019b9e55-c441-734d-8f29-bac2c47635e2",["D",1767887979503],"import { db } from \"../database/db.ts\";\nimport { skills } from \"../database/schema.ts\";\nimport { blob } from \"https://esm.town/v/std/blob\";\nimport { eq } from \"npm:drizzle-orm@0.44.2\";\n\ntype SkillsRefreshRunState = {\n limit: number; // exclusive end of last batch (next run starts here)\n offset: number;\n startedAt: string;\n completedAt?: string;\n};\nconst LAST_RUN_STATE_KEY = \"refresh-skills:last-run-state\";\nexport async function getLastRunState(): Promise {\n try {\n return await blob.getJSON(LAST_RUN_STATE_KEY);\n } catch {\n return null;\n }\n}\n\nconst isDeleted = async (rawFileUrl: string) => {\n const result = await fetch(rawFileUrl, { method: \"HEAD\" });\n if (result.status === 404) {\n return true;\n }\n\n return false;\n};\n\nasync function markSkillDeleted(skillId: string): Promise {\n return await db\n .update(skills)\n .set({ deleted: 1, updated_at: new Date().toISOString() })\n .where(eq(skills.id, skillId));\n}\n\n// Learn more: https://docs.val.town/vals/cron/\nexport default async function (interval: Interval) {\n const state = await getLastRunState();\n const data = await db\n .select()\n .from(skills)\n .where(eq(skills.deleted, 0))\n .all();\n\n return Promise.all(data.map(async (skill) => {\n const metadata = JSON.parse(skill.metadata);\n if (metadata.rawFileUrl === null) {\n return;\n }\n\n const isDeletedSkill = await isDeleted(metadata.rawFileUrl);\n if (isDeletedSkill) {\n await markSkillDeleted(skill.id);\n }\n }));\n}",[],[336,352],"09c6bac2-ecab-11f0-ab70-42dde27851f2","plugins-indexer.ts","019b9e55-c441-734d-8f29-b674ff480c64",["D",1767887979503],"import { parseMarketplace, transformToDbPlugins } from \"./parser.ts\";\nimport { GitHubClient } from \"../clients/github.ts\";\nimport { db } from \"../database/db.ts\";\nimport { plugins, pluginStats } from \"../database/schema.ts\";\nimport { sql } from \"npm:drizzle-orm@0.44.2\";\n\n// Learn more: https://docs.val.town/vals/cron/\nexport default async function (interval: Interval) {\n const startTime = Date.now();\n\n try {\n console.log(\"GitHub plugin indexer started\");\n const github = new GitHubClient();\n const searchQuery = \"filename:marketplace.json path:.claude-plugin\";\n\n const firstPage = await github.searchCode(searchQuery);\n console.log(`Found ${firstPage.total_count} marketplace.json files`);\n\n let indexed = 0;\n let failed = 0;\n\n // Calculate total pages (max 10 pages = 1000 results)\n const maxPages = Math.min(Math.ceil(firstPage.total_count / 100), 10);\n if (firstPage.total_count > 1000) {\n console.warn(\n `Found ${firstPage.total_count} results. GitHub limits to 1000, only first 1000 will be indexed.`,\n );\n }\n\n // Process all pages\n for (let page = 1; page <= maxPages; page++) {\n console.log(`\\nProcessing page ${page}/${maxPages}...`);\n\n const searchResults = page === 1\n ? firstPage\n : await github.searchCode(searchQuery, page);\n\n console.log(`Processing ${searchResults.items.length} items...`);\n\n for (const item of searchResults.items) {\n try {\n //console.log(`Processing ${item.repository.full_name}...`);\n\n // Fetch repository metadata (stars, updated_at)\n const repoMetadata = await github.fetchRepoMetadata(\n item.repository.full_name,\n );\n\n // Fetch marketplace.json content\n const content = await github.fetchFileContent(\n item.repository.full_name,\n item.path,\n );\n\n // Parse and validate\n const marketplace = parseMarketplace(\n content,\n item.repository.full_name,\n );\n if (!marketplace) {\n console.warn(\n `Invalid marketplace.json from ${item.repository.full_name}`,\n );\n failed++;\n continue;\n }\n\n // Transform to database format with repo metadata\n const dbPlugins = transformToDbPlugins(\n marketplace,\n item.repository.owner.login,\n item.repository.html_url,\n repoMetadata.stargazers_count,\n );\n\n // Upsert each plugin\n for (const plugin of dbPlugins) {\n try {\n // Check if plugin exists to get existing ID\n const existing = await db\n .select()\n .from(plugins)\n .where(\n sql`${plugins.namespace} = ${plugin.namespace} AND ${plugins.name} = ${plugin.name}`,\n )\n .get();\n\n // Use existing ID if plugin already exists, otherwise use new UUID\n const pluginId = existing?.id || plugin.id;\n\n // Upsert plugin with correct ID\n await db\n .insert(plugins)\n .values({\n ...plugin,\n id: pluginId,\n })\n .onConflictDoUpdate({\n target: [plugins.namespace, plugins.name],\n set: {\n description: sql`excluded.description`,\n keywords: sql`excluded.keywords`,\n category: sql`excluded.category`,\n version: sql`excluded.version`,\n author: sql`excluded.author`,\n metadata: sql`excluded.metadata`,\n gitUrl: sql`excluded.git_url`,\n stars: sql`excluded.stars`,\n skills: sql`COALESCE(excluded.skills, ${plugin.skills})`,\n updatedAt: sql`CURRENT_TIMESTAMP`,\n },\n })\n .run();\n\n // Ensure stats entry exists (using the correct plugin ID)\n await db\n .insert(pluginStats)\n .values({ pluginId })\n .onConflictDoNothing()\n .run();\n\n indexed++;\n } catch (dbError) {\n console.error(\n `Failed to upsert plugin ${plugin.namespace}/${plugin.name}:`,\n dbError,\n );\n }\n }\n } catch (error) {\n console.error(\n `Failed to process ${item.repository.full_name}:`,\n error,\n );\n failed++;\n }\n }\n }\n } catch (e) {\n console.error(\"Indexer CRON job failed:\", e);\n // so Val.town will notify\n throw e;\n }\n}",[],[336,359],"09c69b46-ecab-11f0-ab70-42dde27851f2","parser.ts","019b9e55-c441-734d-8f29-b0060ec17350",["D",1767887979503],"import { z } from \"npm:zod@4.1.12\";\n\n/**\n * Zod schema for marketplace.json structure\n */\nconst PluginSchema = z\n .object({\n name: z.string().min(1),\n description: z.string().optional(),\n version: z.string().optional(),\n category: z.string().optional(),\n skills: z.array(z.string()).optional(),\n })\n .passthrough(); // Allow ALL other fields\n\nconst MarketplaceSchema = z\n .object({\n name: z.string().min(1),\n plugins: z.array(PluginSchema),\n })\n .passthrough();\n\nexport type MarketplaceData = z.infer;\nexport type PluginData = z.infer;\n\n/**\n * Parse and validate marketplace.json content\n * Handles both standard marketplace format and non-standard single plugin format\n */\nexport function parseMarketplace(\n jsonString: string,\n repoFullName: string,\n): MarketplaceData | null {\n try {\n const jsonData = JSON.parse(jsonString);\n\n // First, try standard marketplace.json format\n const result = MarketplaceSchema.safeParse(jsonData);\n\n if (result.success) {\n return result.data;\n }\n\n // If standard format fails, check if this is a single plugin definition\n // (no marketplace wrapper, just a plugin object)\n if (jsonData.name && !jsonData.plugins) {\n console.log(\n `Detected single plugin format in ${repoFullName}, attempting to wrap...`,\n );\n\n // Try to wrap it as a marketplace with one plugin\n const wrapped = {\n name: jsonData.name,\n owner: jsonData.author || jsonData.owner || { name: \"Unknown\" },\n plugins: [jsonData],\n };\n\n const wrappedResult = MarketplaceSchema.safeParse(wrapped);\n if (wrappedResult.success) {\n console.log(`Successfully wrapped single plugin from ${repoFullName}`);\n return wrappedResult.data;\n }\n }\n\n // Still failed - log the errors\n console.error(`Invalid marketplace.json from ${repoFullName}:`);\n console.error(\n `Validation errors: ${JSON.stringify(result.error.errors, null, 2)}`,\n );\n return null;\n } catch (error) {\n console.error(\n `Failed to parse JSON from ${repoFullName}:`,\n error instanceof Error ? error.message : String(error),\n );\n console.error(`Stack: ${error instanceof Error ? error.stack : \"\"}`);\n return null;\n }\n}\n\n/**\n * Transform marketplace data into database plugin records\n */\nexport function transformToDbPlugins(\n marketplace: MarketplaceData,\n repoOwner: string,\n repoUrl: string,\n repoStars: number = 0,\n): Array<{\n id: string;\n name: string;\n namespace: string;\n gitUrl: string;\n description: string;\n keywords: string;\n skills: string;\n category: string | null;\n version: string;\n author: string;\n metadata: string;\n stars: number;\n verified: boolean;\n downloads: number;\n createdAt: string;\n updatedAt: string;\n}> {\n return marketplace.plugins.map((plugin: any) => {\n // Safe getter helper\n const get = (obj: any, key: string, defaultValue: any = null) => {\n return obj?.[key] ?? defaultValue;\n };\n\n // Simple author extraction - just get a string\n let author = \"Unknown\";\n const pluginAuthor = get(plugin, \"author\");\n if (typeof pluginAuthor === \"string\") {\n author = pluginAuthor;\n } else if (pluginAuthor?.name) {\n author = pluginAuthor.name;\n } else {\n const marketplaceOwner = get(marketplace, \"owner\");\n const marketplacePublisher = get(marketplace, \"publisher\");\n if (typeof marketplacePublisher === \"string\") {\n author = marketplacePublisher;\n } else if (marketplaceOwner?.name) {\n author = marketplaceOwner.name;\n } else {\n author = repoOwner;\n }\n }\n\n // Simple version - use plugin version, fallback to marketplace, then default\n let version = get(plugin, \"version\");\n if (!version || !/^\\d+\\.\\d+\\.\\d+/.test(version)) {\n const marketplaceVersion = get(marketplace, \"version\");\n version = marketplaceVersion && /^\\d+\\.\\d+\\.\\d+/.test(marketplaceVersion)\n ? marketplaceVersion\n : \"1.0.0\";\n }\n\n // Extract keywords from various sources\n const keywords = get(plugin, \"keywords\", []);\n const tags = get(plugin, \"tags\", []);\n const allKeywords = [...keywords, ...tags].filter(\n (k: any) => typeof k === \"string\",\n );\n\n // Everything else goes into metadata\n const metadata = {\n homepage: get(plugin, \"homepage\"),\n repository: get(plugin, \"repository\"),\n license: get(plugin, \"license\"),\n commands: get(plugin, \"commands\"),\n agents: get(plugin, \"agents\"),\n mcpServers: get(plugin, \"mcpServers\"),\n // Include any other fields that might be useful later\n ...(plugin.bugs && { bugs: plugin.bugs }),\n ...(plugin.features && { features: plugin.features }),\n ...(plugin.requirements && { requirements: plugin.requirements }),\n };\n\n return {\n id: crypto.randomUUID(),\n name: plugin.name,\n namespace: `@${repoOwner}/${marketplace.name}`,\n gitUrl: repoUrl,\n description: get(plugin, \"description\", \"\"),\n keywords: JSON.stringify(allKeywords),\n category: get(plugin, \"category\"),\n skills: JSON.stringify(plugin.skills || []),\n version: version,\n author: author,\n metadata: JSON.stringify(metadata),\n stars: repoStars,\n verified: false,\n downloads: 0,\n createdAt: new Date().toISOString(),\n updatedAt: new Date().toISOString(),\n };\n });\n}",[],[336,366],"09c73e02-ecab-11f0-ab70-42dde27851f2",["D",1767887979503],[233,234,235,236,237,238],[336],"09c668f6-ecab-11f0-ab70-42dde27851f2","github.ts","019b9e55-c441-734d-8f29-aaf22323c357","019b9e55-c441-734d-8f29-af409caf1c8b",["D",1767887979503],"import {\n getToken,\n syncTokenHealth,\n} from \"https://esm.town/v/kamalnrf/token-pool/lib/pool.ts\";\nimport { z } from \"npm:zod@3.24.2\";\n\nconst REQUESTS_PER_INTERVAL = 80;\nconst INTERVAL_MS = 1000;\n\nexport type Bucket = {\n min: number;\n max: number;\n count: number;\n};\n\n/**\n * GitHub code search response types\n */\nconst GitHubSearchItemSchema = z.object({\n name: z.string(),\n path: z.string(),\n sha: z.string(),\n url: z.string().url(),\n html_url: z.string().url(),\n repository: z.object({\n id: z.number(),\n full_name: z.string(),\n html_url: z.string().url(),\n owner: z.object({\n login: z.string(),\n }),\n }),\n});\n\nconst GitHubRepoSchema = z.object({\n id: z.number(),\n full_name: z.string(),\n stargazers_count: z.number(),\n updated_at: z.string(),\n html_url: z.string().url(),\n fork: z.boolean(),\n default_branch: z.string(),\n});\n\nconst GitHubSearchResponseSchema = z.object({\n total_count: z.number(),\n incomplete_results: z.boolean(),\n items: z.array(GitHubSearchItemSchema),\n});\n\nexport type GitHubSearchItem = z.infer;\nexport type GitHubSearchResponse = z.infer;\nexport type GitHubRepo = z.infer;\n\n/**\n * GitHub API client with rate limit handling\n */\nexport class GitHubClient {\n private baseUrl = \"https://api.github.com\";\n private apiVersion = \"2022-11-28\";\n private tokens = REQUESTS_PER_INTERVAL;\n private lastRefill = Date.now();\n\n /**\n * Search for files matching a path pattern\n */\n async searchCode(\n query: string,\n page: number = 1,\n ): Promise {\n const url = `${this.baseUrl}/search/code?q=${\n encodeURIComponent(query)\n }&per_page=100&page=${page}`;\n\n const response = await this.fetchWithRetry(url);\n\n // Check rate limit\n const remaining = parseInt(\n response.headers.get(\"x-ratelimit-remaining\") || \"10\",\n );\n const resetTime = parseInt(\n response.headers.get(\"x-ratelimit-reset\") || \"0\",\n );\n\n console.log(\n `GitHub rate limit: ${remaining}/10 remaining, resets at ${\n new Date(resetTime * 1000).toISOString()\n }`,\n );\n\n if (remaining <= 2) {\n const waitSeconds = resetTime - Math.floor(Date.now() / 1000) + 5;\n if (waitSeconds > 0) {\n console.warn(\n `Rate limit low. Waiting ${waitSeconds}s before continuing...`,\n );\n await this.sleep(waitSeconds * 1000);\n }\n }\n\n if (!response.ok) {\n throw new Error(\n `GitHub search failed: ${response.status} ${response.statusText}`,\n );\n }\n\n const data = await response.json();\n return GitHubSearchResponseSchema.parse(data);\n }\n\n /**\n * Fetch raw file content from repository\n */\n async fetchFileContent(repoFullName: string, path: string): Promise {\n const url = `${this.baseUrl}/repos/${repoFullName}/contents/${path}`;\n\n const response = await this.fetchWithRetry(url);\n\n if (!response.ok) {\n throw new Error(\n `Failed to fetch ${path} from ${repoFullName}: ${response.status}`,\n );\n }\n\n const data = await response.json();\n\n // Decode base64 content\n if (data.encoding === \"base64\" && data.content) {\n return atob(data.content.replace(/\\n/g, \"\"));\n }\n\n throw new Error(`Unexpected content encoding: ${data.encoding}`);\n }\n\n /**\n * Fetch repository metadata (stars, updated_at, etc.)\n */\n async fetchRepoMetadata(repoFullName: string): Promise {\n const url = `${this.baseUrl}/repos/${repoFullName}`;\n\n const response = await this.fetchWithRetry(url);\n\n if (!response.ok) {\n throw new Error(\n `Failed to fetch repo metadata for ${repoFullName}: ${response.status}`,\n );\n }\n\n const data = await response.json();\n return GitHubRepoSchema.parse(data);\n }\n\n async findBuckets(\n baseQuery: string,\n {\n min,\n max,\n maxPerBucket = 950,\n minSpan = 100,\n }: {\n min: number;\n max: number;\n maxPerBucket?: number;\n minSpan?: number;\n },\n ): Promise> {\n const query = `${baseQuery} size:${min}..${max}`;\n const count = (await this.searchCode(query)).total_count ?? 0;\n console.log(\"Query =>\", {\n query,\n count,\n });\n\n if (count === 0) return [];\n\n if (count <= maxPerBucket || (max - min) <= minSpan) {\n return [{ min, max, count }];\n }\n\n // Split the range and recurse\n const mid = Math.floor((min + max) / 2);\n const left = await this.findBuckets(\n baseQuery,\n { min: min, max: mid, maxPerBucket, minSpan },\n );\n const right = await this.findBuckets(\n baseQuery,\n { min: mid + 1, max: max, maxPerBucket, minSpan },\n );\n return [...left, ...right];\n }\n\n /**\n * Fetch with exponential backoff retry\n */\n private async fetchWithRetry(\n url: string,\n maxRetries: number = 3,\n ): Promise {\n // Throttling requests to Github before any attempts\n await this.waitForToken();\n\n let lastError: Error | null = null;\n\n for (let attempt = 0; attempt <= maxRetries; attempt++) {\n try {\n const t = await getToken();\n if (!t?.token) {\n throw new Error(\"No Github token found\");\n }\n\n const response = await fetch(url, {\n headers: {\n Accept: \"application/vnd.github+json\",\n Authorization: `Bearer ${t.token}`,\n \"X-GitHub-Api-Version\": this.apiVersion,\n },\n });\n\n await syncTokenHealth(t.id, response);\n // Handle rate limiting with Retry-After\n if (response.status === 429 || response.status === 403) {\n console.log(\n `Rate limited. with ${t.id}, retrying with new token`,\n );\n continue;\n }\n\n return response;\n } catch (error) {\n lastError = error as Error;\n\n if (attempt < maxRetries) {\n const delay = Math.min(1000 * Math.pow(2, attempt), 10000);\n console.log(\n `Request failed, retrying in ${delay}ms... (attempt ${\n attempt + 1\n }/${maxRetries})`,\n );\n await this.sleep(delay);\n }\n }\n }\n\n throw lastError || new Error(\"Request failed after retries\");\n }\n\n private async waitForToken(): Promise {\n const now = Date.now();\n const elapsed = now - this.lastRefill;\n if (elapsed >= INTERVAL_MS) {\n const intervals = Math.floor(elapsed / INTERVAL_MS);\n this.tokens = Math.min(\n REQUESTS_PER_INTERVAL,\n this.tokens + intervals * REQUESTS_PER_INTERVAL,\n );\n this.lastRefill = now;\n }\n\n if (this.tokens > 0) {\n this.tokens--;\n return;\n }\n\n const delay = Math.max(0, INTERVAL_MS - (Date.now() - this.lastRefill));\n await this.sleep(delay);\n // After waiting, consume a token\n if (this.tokens > 0) this.tokens--;\n else this.tokens = Math.max(0, this.tokens - 1);\n }\n\n private sleep(ms: number): Promise {\n return new Promise((resolve) => setTimeout(resolve, ms));\n }\n\n /**\n * Fetch the date when a file was first committed using GitHub REST API.\n * Uses Link header pagination to find the last page (oldest commit).\n * Returns ISO date string or null if unable to determine.\n */\n async fetchFirstCommitDate(\n repoFullName: string,\n filePath: string,\n ): Promise {\n try {\n // First request: get commits for this file with per_page=1 to check Link header\n const url = `${this.baseUrl}/repos/${repoFullName}/commits?path=${encodeURIComponent(filePath)}&per_page=1`;\n const response = await this.fetchWithRetry(url);\n\n if (!response.ok) {\n console.error(\n `Failed to fetch commits for ${repoFullName}/${filePath}: ${response.status}`,\n );\n return null;\n }\n\n // Parse Link header to find last page\n const linkHeader = response.headers.get(\"Link\");\n let lastPage = 1;\n\n if (linkHeader) {\n // Link header format: ; rel=\"next\", ; rel=\"last\"\n const lastMatch = linkHeader.match(/<[^>]+[?&]page=(\\d+)[^>]*>;\\s*rel=\"last\"/);\n if (lastMatch) {\n lastPage = parseInt(lastMatch[1], 10);\n }\n }\n\n // If only one page, the first response already has the oldest commit\n if (lastPage === 1) {\n const commits = await response.json();\n if (commits.length === 0) {\n return null;\n }\n // Single page means this is also the oldest\n const oldestCommit = commits[0];\n return oldestCommit?.commit?.author?.date || oldestCommit?.commit?.committer?.date || null;\n }\n\n // Fetch the last page to get the oldest commit\n const lastPageUrl = `${this.baseUrl}/repos/${repoFullName}/commits?path=${encodeURIComponent(filePath)}&per_page=1&page=${lastPage}`;\n const lastPageResponse = await this.fetchWithRetry(lastPageUrl);\n\n if (!lastPageResponse.ok) {\n console.error(\n `Failed to fetch last page of commits for ${repoFullName}/${filePath}: ${lastPageResponse.status}`,\n );\n return null;\n }\n\n const lastPageCommits = await lastPageResponse.json();\n if (lastPageCommits.length === 0) {\n return null;\n }\n\n const oldestCommit = lastPageCommits[0];\n return oldestCommit?.commit?.author?.date || oldestCommit?.commit?.committer?.date || null;\n } catch (error) {\n console.error(\n `Failed to fetch first commit date for ${repoFullName}/${filePath}:`,\n error,\n );\n return null;\n }\n }\n}",[],[384,377],"clients","09c695f6-ecab-11f0-ab70-42dde27851f2",["D",1767887979503],[231],[384],["D",1767887979503],[],[18,76],["D",1767887979503],[],[18,88],["D",1767887979503],[],[18,94],["D",1767887979503],[],[18,99],["D",1767887979503],[226,227,228,229],[18],["D",1767887979503],["D",1789629047764],{"_11":12,"_155":82},{"_11":12,"_151":-5,"_81":82,"_31":80,"_152":153,"_154":30},{"_39":40,"_13":16,"_33":36,"_140":103,"_141":71,"_142":143,"_144":107,"_145":146,"_147":148,"_149":14,"_150":12},"ctx",{"_52":-5,"_176":442,"_443":444,"_445":446,"_447":30,"_448":30,"_449":30,"_450":24,"_451":-5,"_452":-7,"_453":-7,"_454":26,"_455":30,"_456":30,"_457":-5,"_458":30,"_459":30,"_460":30,"_461":-5},"ENV",{"_413":414,"_415":416,"_417":16,"_418":419,"_420":421,"_422":423,"_424":425,"_426":427,"_428":429,"_430":431,"_432":433,"_434":435,"_436":437,"_438":439,"_440":441},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","899124c68db8496248a17ee95f074b807335ea7b","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]

History · api