Hi

\"\n});\n```\n\n## Val Town Utility Functions\n\nVal Town provides several utility functions to help with common project tasks.\n\n### Importing Utilities\n\nAlways import utilities with version pins to avoid breaking changes:\n\n```ts\nimport { parseProject, readFile, serveFile } from \"https://esm.town/v/std/utils@85-main/index.ts\";\n```\n\n### Available Utilities\n\n\n#### **serveFile** - Serve project files with proper content types\n\nFor example, in Hono:\n\n```ts\n// serve all files in frontend/ and shared/\napp.get(\"/frontend/*\", c => serveFile(c.req.path, import.meta.url));\napp.get(\"/shared/*\", c => serveFile(c.req.path, import.meta.url));\n```\n\n#### **readFile** - Read files from within the project:\n\n```ts\n// Read a file from the project\nconst fileContent = await readFile(\"/frontend/index.html\", import.meta.url);\n```\n\n#### **listFiles** - List all files in the project\n\n```ts\nconst files = await listFiles(import.meta.url);\n```\n\n#### **parseProject** - Extract information about the current project from import.meta.url\n\nThis is useful for including info for linking back to a val, ie in \"view source\" urls:\n\n```ts\nconst projectVal = parseProject(import.meta.url);\nconsole.log(projectVal.username); // Owner of the project\nconsole.log(projectVal.name); // Project name\nconsole.log(projectVal.version); // Version number\nconsole.log(projectVal.branch); // Branch name\nconsole.log(projectVal.links.self.project); // URL to the project page\n```\n\nHowever, it's *extremely importing* to note that `parseProject` and other Standard Library utilities ONLY RUN ON THE SERVER.\nIf you need access to this data on the client, run it in the server and pass it to the client by splicing it into the HTML page\nor by making an API request for it.\n\n## Val Town Platform Specifics\n\n- **Redirects:** Use `return new Response(null, { status: 302, headers: { Location: \"/place/to/redirect\" }})` instead of `Response.redirect` which is broken\n- **Images:** Avoid external images or base64 images. Use emojis, unicode symbols, or icon fonts/libraries instead\n- **AI Image:** To inline generate an AI image use: `

`\n- **Storage:** DO NOT use the Deno KV module for storage\n- **Browser APIs:** DO NOT use the `alert()`, `prompt()`, or `confirm()` methods\n- **Weather Data:** Use open-meteo for weather data (doesn't require API keys) unless otherwise specified\n- **View Source:** Add a view source link by importing & using `import.meta.url.replace(\"ems.sh\", \"val.town)\"` (or passing this data to the client) and include `target=\"_top\"` attribute\n- **Error Debugging:** Add `` to HTML to capture client-side errors\n- **Error Handling:** Only use try...catch when there's a clear local resolution; Avoid catches that merely log or return 500s. Let errors bubble up with full context\n- **Environment Variables:** Use `Deno.env.get('keyname')` when you need to, but generally prefer APIs that don't require keys\n- **Imports:** Use `https://esm.sh` for npm and Deno dependencies to ensure compatibility on server and browser\n- **Storage Strategy:** Only use backend storage if explicitly required; prefer simple static client-side sites\n- **React Configuration:** When using React libraries, pin versions with `?deps=react@18.2.0,react-dom@18.2.0` and start the file with `/** @jsxImportSource https://esm.sh/react@18.2.0 */`\n- Ensure all React dependencies and sub-dependencies are pinned to the same version\n- **Styling:** Default to using TailwindCSS via `` unless otherwise specified\n\n## Project Structure and Design Patterns\n\n### Recommended Directory Structure\n```\n├── backend/\n│ ├── database/\n│ │ ├── migrations.ts # Schema definitions\n│ │ ├── queries.ts # DB query functions\n│ │ └── README.md\n│ └── routes/ # Route modules\n│ ├── [route].ts\n│ └── static.ts # Static file serving\n│ ├── index.ts # Main entry point\n│ └── README.md\n├── frontend/\n│ ├── components/\n│ │ ├── App.tsx\n│ │ └── [Component].tsx\n│ ├── favicon.svg\n│ ├── index.html # Main HTML template\n│ ├── index.tsx # Frontend JS entry point\n│ ├── README.md\n│ └── style.css\n├── README.md\n└── shared/\n ├── README.md\n └── utils.ts # Shared types and functions\n```\n\n### Backend (Hono) Best Practices\n\n- Hono is the recommended API framework\n- Main entry point should be `backend/index.ts`\n- **Static asset serving:** Use the utility functions to read and serve project files:\n ```ts\n import { readFile, serveFile } from \"https://esm.town/v/std/utils@85-main/index.ts\";\n \n // serve all files in frontend/ and shared/\n app.get(\"/frontend/*\", c => serveFile(c.req.path, import.meta.url));\n app.get(\"/shared/*\", c => serveFile(c.req.path, import.meta.url));\n \n // For index.html, often you'll want to bootstrap with initial data\n app.get(\"/\", async c => {\n let html = await readFile(\"/frontend/index.html\", import.meta.url);\n \n // Inject data to avoid extra round-trips\n const initialData = await fetchInitialData();\n const dataScript = ``;\n \n html = html.replace(\"\", `${dataScript}`);\n return c.html(html);\n });\n ```\n- Create RESTful API routes for CRUD operations\n- Always include this snippet at the top-level Hono app to re-throwing errors to see full stack traces:\n ```ts\n // Unwrap Hono errors to see original error details\n app.onError((err, c) => {\n throw err;\n });\n ```\n\n### Database Patterns\n- Run migrations on startup or comment out for performance\n- Change table names when modifying schemas rather than altering\n- Export clear query functions with proper TypeScript typing\n\n## Common Gotchas and Solutions\n\n1. **Environment Limitations:** \n - Val Town runs on Deno in a serverless context, not Node.js\n - Code in `shared/` must work in both frontend and backend environments\n - Cannot use `Deno` keyword in shared code\n - Use `https://esm.sh` for imports that work in both environments\n\n2. **SQLite Peculiarities:**\n - Limited support for ALTER TABLE operations\n - Create new tables with updated schemas and copy data when needed\n - Always run table creation before querying\n\n3. **React Configuration:**\n - All React dependencies must be pinned to 18.2.0\n - Always include `@jsxImportSource https://esm.sh/react@18.2.0` at the top of React files\n - Rendering issues often come from mismatched React versions\n\n4. **File Handling:**\n - Val Town only supports text files, not binary\n - Use the provided utilities to read files across branches and forks\n - For files in the project, use `readFile` helpers\n\n5. **API Design:**\n - `fetch` handler is the entry point for HTTP vals\n - Run the Hono app with `export default app.fetch // This is the entry point for HTTP vals`\n\n","deleted","label","children",[],[154],{"_102":173,"_95":174,"_155":175,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":176,"_99":32,"_163":164,"_165":177,"_167":22,"_168":-5,"_169":178,"_98":179},"645c9766-a919-11f0-8842-0224a6c84d84","README.md","0199e383-e9b8-700d-adb7-a5423a1880f6",["D",1760458697070],"# GitHub Engagement to Clay\n\n![clay-data.png](https://imagedelivery.net/iHX6Ovru0O7AjmyT5yZRoA/cbeb5c23-1b9a-42e7-b176-b94c7ec42400/public)\n\nAutomatically ingest GitHub accounts of everyone who interacts with your repo\ninto Clay.\n\nThis val tracks anyone who:\n\n- Created an issue\n- Reacted to an issue\n- Commented on an issue\n- Reacted to an issue's comments\n- Starred a repo\n- Forked a repo\n\n## Setup\n\n1. Click `Remix` on the top-right to get a copy of it\n1. Set up a Clay workbook with a `Webhook` column\n1. Copy your Clay workbook's `Webhook` URL\n ![CleanShot 2025-08-08 at 13.23.47@2x.png](https://imagedelivery.net/iHX6Ovru0O7AjmyT5yZRoA/48cd9c4d-2990-480f-9d07-fb09e2282500/public)\n\n1. Set that as `CLAY_WEBHOOK_URL` in this val's `Environment variables` on the\n left sidebar\n1. In [config.ts](./config.ts), configure the following settings:\n\n **Required:**\n - `GITHUB_REPO` - The repository to track (format: `\"owner/repo\"`)\n\n **Optional (for testing or limiting API usage):**\n - `ISSUE_LIMIT` - Maximum number of issues to fetch (`undefined` to fetch all\n issues)\n - `MAX_STARGAZERS` - Maximum number of stargazers to fetch (`undefined` to\n fetch all)\n - `MAX_FORKS` - Maximum number of forks to fetch (`undefined` to fetch all)\n - `MAX_COMMENTS_PER_ISSUE` - Maximum comments to fetch per issue (`undefined`\n to fetch all)\n - `FETCH_REACTIONS` - Set to `true` to track reactions on issues and comments\n (Note that this will dramatically slow down the val. Default: `false`)\n\n **Optional (deduplication & data management):**\n - `ENABLE_DEDUPLICATION` - Prevents sending duplicate users to Clay by\n tracking them in a database (default: `true`)\n - `MAX_ISSUE_NUMBERS_PER_INTERACTION` - Limits the number of issue numbers\n stored per interaction type to keep Clay payloads manageable (default:\n `20`)\n\n1. That's it! The cron will run on your repo every 30 minutes from now on. To\n test it out immediately, navigate to [main.ts](./main.ts) and click `Run`.\n\n## Files\n\n- [config.ts](config.ts) - Configuration settings for the scraper\n- [database.ts](database.ts) - SQLite database for tracking sent users\n- [clay.ts](clay.ts) - Send data to Clay\n- [github.ts](github.ts) - Collects engaged users and fetches usernames\n- [main.ts](main.ts) - Cron trigger that orchestrates the scraping and Clay\n integration\n\n## User Deduplication\n\nThis scraper automatically tracks which users have been sent to Clay to prevent\nduplicates. The feature is controlled by the `ENABLE_DEDUPLICATION` flag in\n[config.ts](./config.ts) (enabled by default).\n\n**How it works:**\n\n- GitHub usernames are stored in a SQLite database after being successfully sent\n to Clay\n- On subsequent runs, the scraper checks the database and only sends new users\n- The database persists across all cron runs, so users are only sent once\n- Each user is tracked by their GitHub username and source repository\n\n**Database details:**\n\n- Users are tracked in the `tracked_users` table\n- Stores: username, source repo, and timestamp of first encounter\n- The database automatically initializes on the first run\n\n**To disable deduplication:** Set `ENABLE_DEDUPLICATION = false` in\n[config.ts](./config.ts) to send all engaged users on every run (useful for\ntesting or if you want to manage deduplication in Clay instead).\n\n## Rate Limiting\n\nOn larger repos, you may get rate-limited by GitHub. To mitigate this, Val Town\nuses a [proxied fetch](https://docs.val.town/std/fetch/) that reroutes requests\nusing a proxy vendor so that requests obtain different IP addresses. It also\nautomatically retries failed requests several times. Note that Note that using\n`std/fetch` will be significantly slower than directly calling the Javascript\nFetch API due to extra network hops.",[],[174],{"_102":181,"_95":182,"_155":183,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":184,"_99":32,"_163":185,"_165":186,"_167":22,"_168":-5,"_169":187,"_98":188},"645ca2f6-a919-11f0-8842-0224a6c84d84","clay.ts","0199e383-e9b8-700d-adb7-a9d122ac5d8b",["D",1760458697070],"script","// Helper function to add delay between batches\nfunction delay(ms: number): Promise {\n return new Promise((resolve) => setTimeout(resolve, ms));\n}\n\n// Send data to Clay in batches to respect rate limits\n// Clay limits: 10 records per second per workspace, max burst 20\nexport async function sendToClay(\n webhookUrl: string,\n data: any[],\n): Promise {\n const BATCH_SIZE = 10;\n const BATCH_DELAY_MS = 1000; // 1 second between batches\n\n // Split data into batches of 10\n for (let i = 0; i < data.length; i += BATCH_SIZE) {\n const batch = data.slice(i, i + BATCH_SIZE);\n console.log(` Sending batch ${Math.floor(i / BATCH_SIZE) + 1} (${batch.length} records)...`);\n\n // Send all records in this batch in parallel\n const requests = batch.map(async (record) => {\n const response = await fetch(webhookUrl, {\n method: \"POST\",\n headers: { \"Content-Type\": \"application/json\" },\n body: JSON.stringify(record),\n });\n\n if (!response.ok) {\n const errorText = await response.text();\n throw new Error(\n `Clay request failed for user ${record.github_username}: ${response.status} ${errorText}`,\n );\n }\n\n return response;\n });\n\n // Wait for all requests in this batch to complete\n await Promise.all(requests);\n\n // If there are more batches, wait 1 second before sending the next batch\n if (i + BATCH_SIZE < data.length) {\n await delay(BATCH_DELAY_MS);\n }\n }\n}\n",[],[182],{"_102":190,"_95":191,"_155":192,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":193,"_99":32,"_163":185,"_165":194,"_167":22,"_168":-5,"_169":195,"_98":196},"645ca774-a919-11f0-8842-0224a6c84d84","config.ts","0199e383-e9b8-700d-adb7-ae9b6f300c46",["D",1760458697070],"// GitHub repository to track (format: \"owner/repo\")\nexport const GITHUB_REPO = \"sourcebot-dev/sourcebot\";\n\n// Optional: Limit number of issues to fetch (undefined = fetch all)\nexport const ISSUE_LIMIT = 3 as number | undefined;\n\n// Optional: Limit number of stargazers to fetch (undefined = fetch all)\nexport const MAX_STARGAZERS = 3 as number | undefined;\n\n// Optional: Limit number of forks to fetch (undefined = fetch all)\nexport const MAX_FORKS = 3 as number | undefined;\n\n// Optional: Limit comments per issue (undefined = fetch all)\nexport const MAX_COMMENTS_PER_ISSUE = 3 as number | undefined;\n\n// Optional: Skip fetching reactions (faster processing)\nexport const FETCH_REACTIONS = false;\n\n// Optional: Enable user deduplication to prevent sending duplicate users to Clay\n// When enabled, users are tracked in a SQLite database and only new users are sent\nexport const ENABLE_DEDUPLICATION = true;\n\n// Maximum number of issue numbers to track per interaction type (e.g., issues_created, issues_commented_on)\n// For users who interact with many issues, this limits array sizes to keep Clay payloads manageable\n// When exceeded, only the most recent 20 issue numbers are stored\nexport const MAX_ISSUE_NUMBERS_PER_INTERACTION = 20;\n",[],[191],{"_102":198,"_95":199,"_155":200,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":201,"_99":32,"_163":185,"_165":202,"_167":22,"_168":-5,"_169":203,"_98":204},"645caaa8-a919-11f0-8842-0224a6c84d84","database.ts","0199e383-e9b8-700d-adb7-b0335289cb5b",["D",1760458697070],"import { sqlite } from \"https://esm.town/v/stevekrouse/sqlite\";\n\nconst TABLE_NAME = \"tracked_users\";\n\n/**\n * Initialize the database and create the tracked_users table if it doesn't exist\n */\nexport async function initDatabase(): Promise {\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLE_NAME} (\n username TEXT PRIMARY KEY,\n source_repo TEXT NOT NULL,\n first_seen TEXT NOT NULL\n )\n `);\n}\n\n/**\n * Check if a user is already tracked in the database\n */\nexport async function isUserTracked(username: string): Promise {\n const result = await sqlite.execute(\n `SELECT 1 FROM ${TABLE_NAME} WHERE username = ? LIMIT 1`,\n [username],\n );\n return result.rows.length > 0;\n}\n\n/**\n * Add multiple users to the tracked users database\n */\nexport async function addTrackedUsers(\n users: Array<{ username: string; sourceRepo: string }>,\n): Promise {\n if (users.length === 0) return;\n\n const timestamp = new Date().toISOString();\n\n // Use a transaction for bulk insert\n for (const user of users) {\n await sqlite.execute(\n `INSERT OR IGNORE INTO ${TABLE_NAME} (username, source_repo, first_seen) VALUES (?, ?, ?)`,\n [user.username, user.sourceRepo, timestamp],\n );\n }\n}\n\n/**\n * Get statistics about tracked users\n */\nexport async function getStats(): Promise<{\n totalUsers: number;\n byRepo: Record;\n}> {\n const totalResult = await sqlite.execute(\n `SELECT COUNT(*) as count FROM ${TABLE_NAME}`,\n );\n const totalUsers = totalResult.rows[0]?.count || 0;\n\n const repoResult = await sqlite.execute(\n `SELECT source_repo, COUNT(*) as count FROM ${TABLE_NAME} GROUP BY source_repo`,\n );\n\n const byRepo: Record = {};\n for (const row of repoResult.rows) {\n byRepo[row.source_repo] = row.count;\n }\n\n return { totalUsers, byRepo };\n}\n",[],[199],{"_102":206,"_95":207,"_155":208,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":209,"_99":32,"_163":185,"_165":210,"_167":22,"_168":-5,"_169":211,"_98":212},"645caf08-a919-11f0-8842-0224a6c84d84","github.ts","0199e383-e9b8-700d-adb7-b54fb58e73fe",["D",1760458697070],"import { fetch } from \"https://esm.town/v/std/fetch\";\nimport { MAX_ISSUE_NUMBERS_PER_INTERACTION } from \"./config.ts\";\n\nexport interface InteractionDetails {\n starred: boolean;\n forked: boolean;\n issues_created?: {\n count: number;\n issue_numbers: number[];\n };\n issue_comments?: {\n count: number;\n issues_commented_on: number[];\n };\n issue_reactions?: {\n count: number;\n issues_reacted_to: number[];\n };\n comment_reactions?: {\n count: number;\n comments_reacted_to_on_issues: number[];\n };\n}\n\nexport interface EngagementSummary {\n total_interactions: number;\n first_seen?: string;\n last_seen?: string;\n}\n\nexport interface EngagedUser {\n github_username: string;\n engagement_summary: EngagementSummary;\n interactions: InteractionDetails;\n}\n\nexport async function fetchAllIssues(\n owner: string,\n repo: string,\n limit?: number,\n): Promise {\n const issues: any[] = [];\n let page = 1;\n const perPage = limit && limit < 100 ? limit : 100;\n\n const headers: Record = {\n Accept: \"application/vnd.github.v3+json\",\n \"User-Agent\": \"GitHubEngagementTracker/1.0\",\n };\n\n while (true) {\n const remainder = limit\n ? Math.min(perPage, limit - issues.length)\n : perPage;\n\n if (remainder <= 0) break;\n\n const response = await fetch(\n `https://api.github.com/repos/${owner}/${repo}/issues?page=${page}&per_page=${remainder}&state=all`,\n { headers },\n );\n\n if (!response.ok) {\n throw new Error(\n `GitHub API error: ${response.status} ${response.statusText}`,\n );\n }\n\n const data = await response.json();\n\n const filteredIssues = data\n .filter((item) => !item.pull_request)\n .map((item) => ({\n number: item.number,\n title: item.title,\n body: item.body || \"\",\n author_username: item.user.login,\n created_at: item.created_at,\n updated_at: item.updated_at,\n state: item.state,\n labels: item.labels.map((label) => label.name),\n html_url: item.html_url,\n }));\n\n issues.push(...filteredIssues);\n\n if (data.length < remainder) break;\n\n page++;\n }\n\n return issues;\n}\n\nexport async function fetchIssueComments(\n owner: string,\n repo: string,\n issueNumber: number,\n limit?: number,\n): Promise {\n const comments: any[] = [];\n let page = 1;\n const perPage = limit && limit < 100 ? limit : 100;\n\n const headers: Record = {\n Accept: \"application/vnd.github.v3+json\",\n \"User-Agent\": \"GitHubEngagementTracker/1.0\",\n };\n\n while (true) {\n const remainder = limit\n ? Math.min(perPage, limit - comments.length)\n : perPage;\n\n if (remainder <= 0) break;\n\n const response = await fetch(\n `https://api.github.com/repos/${owner}/${repo}/issues/${issueNumber}/comments?page=${page}&per_page=${remainder}`,\n { headers },\n );\n\n if (!response.ok) {\n if (response.status === 404) return comments;\n throw new Error(\n `GitHub API error fetching comments: ${response.status} ${response.statusText}`,\n );\n }\n\n const data = await response.json();\n\n const mappedComments = data.map((item: any) => ({\n id: item.id,\n body: item.body || \"\",\n author_username: item.user.login,\n created_at: item.created_at,\n issue_number: issueNumber,\n }));\n\n comments.push(...mappedComments);\n\n if (data.length < remainder) break;\n page++;\n }\n\n return comments;\n}\n\nexport async function fetchIssueReactions(\n owner: string,\n repo: string,\n issueNumber: number,\n): Promise {\n const headers: Record = {\n Accept: \"application/vnd.github.v3+json\",\n \"User-Agent\": \"GitHubEngagementTracker/1.0\",\n };\n\n const response = await fetch(\n `https://api.github.com/repos/${owner}/${repo}/issues/${issueNumber}/reactions`,\n { headers },\n );\n\n if (!response.ok) {\n if (response.status === 404) return [];\n throw new Error(\n `GitHub API error fetching issue reactions: ${response.status} ${response.statusText}`,\n );\n }\n\n const data = await response.json();\n\n return data.map((item: any) => ({\n content: item.content,\n user_username: item.user.login,\n created_at: item.created_at,\n }));\n}\n\nexport async function fetchCommentReactions(\n owner: string,\n repo: string,\n commentId: number,\n): Promise {\n const headers: Record = {\n Accept: \"application/vnd.github.v3+json\",\n \"User-Agent\": \"GitHubEngagementTracker/1.0\",\n };\n\n const response = await fetch(\n `https://api.github.com/repos/${owner}/${repo}/issues/comments/${commentId}/reactions`,\n { headers },\n );\n\n if (!response.ok) {\n if (response.status === 404) return [];\n throw new Error(\n `GitHub API error fetching comment reactions: ${response.status} ${response.statusText}`,\n );\n }\n\n const data = await response.json();\n\n return data.map((item: any) => ({\n content: item.content,\n user_username: item.user.login,\n created_at: item.created_at,\n }));\n}\n\nexport async function fetchStargazers(\n owner: string,\n repo: string,\n limit?: number,\n): Promise {\n const stargazers: string[] = [];\n let page = 1;\n const perPage = limit && limit < 100 ? limit : 100;\n\n const headers: Record = {\n Accept: \"application/vnd.github.v3+json\",\n \"User-Agent\": \"GitHubEngagementTracker/1.0\",\n };\n\n while (true) {\n const remainder = limit\n ? Math.min(perPage, limit - stargazers.length)\n : perPage;\n\n if (remainder <= 0) break;\n\n const response = await fetch(\n `https://api.github.com/repos/${owner}/${repo}/stargazers?page=${page}&per_page=${remainder}`,\n { headers },\n );\n\n if (!response.ok) {\n if (response.status === 404) return stargazers;\n throw new Error(\n `GitHub API error fetching stargazers: ${response.status} ${response.statusText}`,\n );\n }\n\n const data = await response.json();\n\n for (const item of data) {\n stargazers.push(item.login);\n }\n\n if (data.length < remainder) break;\n page++;\n }\n\n return stargazers;\n}\n\nexport async function fetchForks(\n owner: string,\n repo: string,\n limit?: number,\n): Promise {\n const forkers: string[] = [];\n let page = 1;\n const perPage = limit && limit < 100 ? limit : 100;\n\n const headers: Record = {\n Accept: \"application/vnd.github.v3+json\",\n \"User-Agent\": \"GitHubEngagementTracker/1.0\",\n };\n\n while (true) {\n const remainder = limit\n ? Math.min(perPage, limit - forkers.length)\n : perPage;\n\n if (remainder <= 0) break;\n\n const response = await fetch(\n `https://api.github.com/repos/${owner}/${repo}/forks?page=${page}&per_page=${remainder}`,\n { headers },\n );\n\n if (!response.ok) {\n if (response.status === 404) return forkers;\n throw new Error(\n `GitHub API error fetching forks: ${response.status} ${response.statusText}`,\n );\n }\n\n const data = await response.json();\n\n for (const item of data) {\n forkers.push(item.owner.login);\n }\n\n if (data.length < remainder) break;\n page++;\n }\n\n return forkers;\n}\n\ninterface FetchConfig {\n issueLimit?: number;\n maxStargazers?: number;\n maxForks?: number;\n maxCommentsPerIssue?: number;\n fetchReactions: boolean;\n}\n\n// Helper interface for accumulating user interaction data\ninterface UserInteractionAccumulator {\n starred: boolean;\n forked: boolean;\n issuesCreated: Set;\n issuesCommentedOn: Set;\n issuesReactedTo: Set;\n issuesWithCommentReactions: Set;\n timestamps: string[];\n}\n\nfunction addIssueNumber(set: Set, issueNumber: number): void {\n if (set.size < MAX_ISSUE_NUMBERS_PER_INTERACTION) {\n set.add(issueNumber);\n }\n}\n\nfunction addTimestamp(timestamps: string[], timestamp: string): void {\n if (timestamp) {\n timestamps.push(timestamp);\n }\n}\n\nexport async function fetchEngagedUsers(\n owner: string,\n repo: string,\n config: FetchConfig,\n): Promise {\n console.log(`Fetching engaged users for ${owner}/${repo}...`);\n\n const issues = await fetchAllIssues(owner, repo, config.issueLimit);\n console.log(`Found ${issues.length} issues`);\n\n // Map to accumulate interaction data per user\n const userInteractions = new Map();\n\n function ensureUser(username: string): UserInteractionAccumulator {\n if (!userInteractions.has(username)) {\n userInteractions.set(username, {\n starred: false,\n forked: false,\n issuesCreated: new Set(),\n issuesCommentedOn: new Set(),\n issuesReactedTo: new Set(),\n issuesWithCommentReactions: new Set(),\n timestamps: [],\n });\n }\n return userInteractions.get(username)!;\n }\n\n // Collect issue creators\n for (const issue of issues) {\n const user = ensureUser(issue.author_username);\n addIssueNumber(user.issuesCreated, issue.number);\n addTimestamp(user.timestamps, issue.created_at);\n }\n\n // Fetch comments and reactions for each issue\n for (const issue of issues) {\n // Fetch issue comments\n try {\n const comments = await fetchIssueComments(\n owner,\n repo,\n issue.number,\n config.maxCommentsPerIssue,\n );\n\n for (const comment of comments) {\n const user = ensureUser(comment.author_username);\n addIssueNumber(user.issuesCommentedOn, issue.number);\n addTimestamp(user.timestamps, comment.created_at);\n\n // Fetch reactions for this comment (if enabled)\n if (config.fetchReactions) {\n try {\n const commentReactions = await fetchCommentReactions(\n owner,\n repo,\n comment.id,\n );\n\n for (const reaction of commentReactions) {\n const reactorUser = ensureUser(reaction.user_username);\n addIssueNumber(reactorUser.issuesWithCommentReactions, issue.number);\n addTimestamp(reactorUser.timestamps, reaction.created_at);\n }\n } catch (error) {\n console.log(\n ` Error fetching reactions for comment ${comment.id}:`,\n error.message,\n );\n }\n }\n }\n } catch (error) {\n console.log(\n ` Error fetching comments for issue #${issue.number}:`,\n error.message,\n );\n }\n\n // Fetch issue reactions (if enabled)\n if (config.fetchReactions) {\n try {\n const issueReactions = await fetchIssueReactions(\n owner,\n repo,\n issue.number,\n );\n\n for (const reaction of issueReactions) {\n const user = ensureUser(reaction.user_username);\n addIssueNumber(user.issuesReactedTo, issue.number);\n addTimestamp(user.timestamps, reaction.created_at);\n }\n } catch (error) {\n console.log(\n ` Error fetching reactions for issue #${issue.number}:`,\n error.message,\n );\n }\n }\n }\n\n // Fetch stargazers\n try {\n const stargazers = await fetchStargazers(owner, repo, config.maxStargazers);\n console.log(`Found ${stargazers.length} stargazers`);\n for (const username of stargazers) {\n const user = ensureUser(username);\n user.starred = true;\n }\n } catch (error) {\n console.log(` Error fetching stargazers:`, error.message);\n }\n\n // Fetch forkers\n try {\n const forkers = await fetchForks(owner, repo, config.maxForks);\n console.log(`Found ${forkers.length} forkers`);\n for (const username of forkers) {\n const user = ensureUser(username);\n user.forked = true;\n }\n } catch (error) {\n console.log(` Error fetching forkers:`, error.message);\n }\n\n // Convert accumulated data to EngagedUser format\n const engagedUsers: EngagedUser[] = [];\n\n for (const [username, data] of userInteractions.entries()) {\n // Calculate total interactions\n let totalInteractions = 0;\n if (data.starred) totalInteractions += 1;\n if (data.forked) totalInteractions += 1;\n totalInteractions += data.issuesCreated.size;\n totalInteractions += data.issuesCommentedOn.size;\n totalInteractions += data.issuesReactedTo.size;\n totalInteractions += data.issuesWithCommentReactions.size;\n\n // Get first and last seen from timestamps\n const sortedTimestamps = data.timestamps.sort();\n const firstSeen = sortedTimestamps[0];\n const lastSeen = sortedTimestamps[sortedTimestamps.length - 1];\n\n // Build interactions object\n const interactions: InteractionDetails = {\n starred: data.starred,\n forked: data.forked,\n };\n\n if (data.issuesCreated.size > 0) {\n interactions.issues_created = {\n count: data.issuesCreated.size,\n issue_numbers: Array.from(data.issuesCreated).sort((a, b) => b - a), // Descending\n };\n }\n\n if (data.issuesCommentedOn.size > 0) {\n interactions.issue_comments = {\n count: data.issuesCommentedOn.size,\n issues_commented_on: Array.from(data.issuesCommentedOn).sort((a, b) => b - a),\n };\n }\n\n if (data.issuesReactedTo.size > 0) {\n interactions.issue_reactions = {\n count: data.issuesReactedTo.size,\n issues_reacted_to: Array.from(data.issuesReactedTo).sort((a, b) => b - a),\n };\n }\n\n if (data.issuesWithCommentReactions.size > 0) {\n interactions.comment_reactions = {\n count: data.issuesWithCommentReactions.size,\n comments_reacted_to_on_issues: Array.from(data.issuesWithCommentReactions).sort((a, b) => b - a),\n };\n }\n\n engagedUsers.push({\n github_username: username,\n engagement_summary: {\n total_interactions: totalInteractions,\n first_seen: firstSeen,\n last_seen: lastSeen,\n },\n interactions,\n });\n }\n\n console.log(`Total unique engaged users: ${engagedUsers.length}`);\n return engagedUsers;\n}\n",[],[207],{"_102":214,"_95":215,"_155":216,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":217,"_99":32,"_163":218,"_165":219,"_167":22,"_168":-5,"_169":220,"_98":221},"645cbde0-a919-11f0-8842-0224a6c84d84","main.ts","0199e383-e9b8-700d-adb7-ba7eb3c754d9",["D",1760458697070],"interval","import { fetchEngagedUsers } from \"./github.ts\";\nimport { sendToClay } from \"./clay.ts\";\nimport {\n GITHUB_REPO,\n ISSUE_LIMIT,\n MAX_STARGAZERS,\n MAX_FORKS,\n MAX_COMMENTS_PER_ISSUE,\n FETCH_REACTIONS,\n ENABLE_DEDUPLICATION,\n} from \"./config.ts\";\nimport {\n initDatabase,\n isUserTracked,\n addTrackedUsers,\n getStats,\n} from \"./database.ts\";\n\nexport default async function () {\n // Make sure to set your Clay Webhook URL in the Environment Variables!\n const clayWebhookUrl = Deno.env.get(\"CLAY_WEBHOOK_URL\");\n\n if (!clayWebhookUrl) {\n throw new Error(\"CLAY_WEBHOOK_URL environment variable is required\");\n }\n\n const [owner, repo] = String(GITHUB_REPO).split(\"/\");\n\n // Initialize database for user tracking\n if (ENABLE_DEDUPLICATION) {\n await initDatabase();\n console.log(\"Database initialized for user deduplication\");\n }\n\n // Fetch engaged users\n const engagedUsers = await fetchEngagedUsers(owner, repo, {\n issueLimit: ISSUE_LIMIT,\n maxStargazers: MAX_STARGAZERS,\n maxForks: MAX_FORKS,\n maxCommentsPerIssue: MAX_COMMENTS_PER_ISSUE,\n fetchReactions: FETCH_REACTIONS,\n });\n console.log(`Found ${engagedUsers.length} engaged users`);\n\n // Filter out already-tracked users (if deduplication is enabled)\n let newUsers = engagedUsers;\n if (ENABLE_DEDUPLICATION) {\n newUsers = [];\n for (const user of engagedUsers) {\n const isTracked = await isUserTracked(user.github_username);\n if (!isTracked) {\n newUsers.push(user);\n }\n }\n const alreadyTracked = engagedUsers.length - newUsers.length;\n console.log(\n `✓ Detected ${alreadyTracked} duplicate user${alreadyTracked !== 1 ? \"s\" : \"\"} (already in database)`,\n );\n console.log(\n `✓ Found ${newUsers.length} new unique user${newUsers.length !== 1 ? \"s\" : \"\"} to send to Clay`,\n );\n\n // Show database stats\n const stats = await getStats();\n console.log(\n `Database stats: ${stats.totalUsers} total users tracked across all repos`,\n );\n }\n\n // Format data for Clay integration - one record per user\n const userData = newUsers.map((user) => ({\n github_username: user.github_username,\n source_repo: GITHUB_REPO,\n engagement_summary: user.engagement_summary,\n interactions: user.interactions,\n }));\n\n // Log data\n if (userData.length > 0 && userData.length <= 10) {\n console.log(`\\n=== User Data (for Clay integration) ===`);\n userData.forEach((record) => {\n console.log(JSON.stringify(record, null, 2));\n });\n } else if (userData.length > 10) {\n console.log(\n `\\n=== Sample User Data (showing first 3 of ${userData.length}) ===`,\n );\n userData.slice(0, 3).forEach((record) => {\n console.log(JSON.stringify(record, null, 2));\n });\n }\n\n console.log(`\\n→ Sending ${userData.length} user records to Clay...`);\n\n // Send to Clay\n if (userData.length > 0) {\n await sendToClay(clayWebhookUrl, userData);\n console.log(`✓ Successfully sent ${userData.length} records to Clay`);\n\n // Add newly sent users to the tracking database\n if (ENABLE_DEDUPLICATION) {\n await addTrackedUsers(\n newUsers.map((user) => ({\n username: user.github_username,\n sourceRepo: GITHUB_REPO,\n })),\n );\n console.log(`✓ Added ${newUsers.length} users to tracking database`);\n }\n } else {\n console.log(`⚠ No new user records to send`);\n }\n\n console.log(`\\n✓ Cron job completed successfully`);\n}\n",[],[215],"files",[152,172,180,189,197,205,213],"counts",{"_226":32,"_227":135,"_228":32,"_229":230},"pulls","branches","forks","envVars",{"_231":135,"_232":32},"count","emptyCount","contents",[235,238,241,244,247,250,253],{"_102":153,"_95":154,"_155":156,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":236,"_99":32,"_163":164,"_165":166,"_167":22,"_168":-5,"_98":237},["D",1760458697070],[154],{"_102":173,"_95":174,"_155":175,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":239,"_99":32,"_163":164,"_165":177,"_167":22,"_168":-5,"_98":240},["D",1760458697070],[174],{"_102":181,"_95":182,"_155":183,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":242,"_99":32,"_163":185,"_165":186,"_167":22,"_168":-5,"_98":243},["D",1760458697070],[182],{"_102":190,"_95":191,"_155":192,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":245,"_99":32,"_163":185,"_165":194,"_167":22,"_168":-5,"_98":246},["D",1760458697070],[191],{"_102":198,"_95":199,"_155":200,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":248,"_99":32,"_163":185,"_165":202,"_167":22,"_168":-5,"_98":249},["D",1760458697070],[199],{"_102":206,"_95":207,"_155":208,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":251,"_99":32,"_163":185,"_165":210,"_167":22,"_168":-5,"_98":252},["D",1760458697070],[207],{"_102":214,"_95":215,"_155":216,"_157":-5,"_158":136,"_159":135,"_160":161,"_104":254,"_99":32,"_163":218,"_165":219,"_167":22,"_168":-5,"_98":255},["D",1760458697070],[215],"upstreamProject",{"_102":144,"_95":258,"_110":259,"_111":260,"_108":109,"_23":261},"GitHub-To-Clay","Scrape users from any GitHub repo, and ingest into Clay","https://imagedelivery.net/iHX6Ovru0O7AjmyT5yZRoA/7d1bd1e4-9009-406c-8bf6-a238d334e100/square512",{"_93":262,"_128":263},"whp","https://imagedelivery.net/iHX6Ovru0O7AjmyT5yZRoA/0dc740ff-fa57-445c-3eec-80fd20daa500/square512","upstreamBranch",{"_102":139,"_104":266,"_267":268,"_95":43,"_137":144,"_99":269,"_138":-5,"_140":-5,"_270":271,"_106":-5,"_149":22,"_108":109,"_148":262,"_16":146,"_147":258},["D",1760276213018],"updatedAt",["D",1760458811326],28,"lockfileKey","1760460120024-esm.town_v_whp_GitHub-To-Clay@28-main","emptyEnvVars","isNew","flags",[],"routes/_app.x.$handle.$name.compare.$branch",{"_93":94,"_95":96,"_97":43,"_100":278,"_283":32,"_284":135,"_285":286,"_292":22,"_293":282,"_294":295,"_298":-5},{"_102":103,"_95":96,"_104":279,"_106":-5,"_107":22,"_108":109,"_110":-5,"_111":-5,"_14":112,"_113":-5,"_114":22,"_115":-5,"_116":32,"_16":117,"_118":119,"_120":119,"_121":122,"_123":122,"_124":122,"_125":119,"_126":119,"_23":280,"_130":281,"_133":282,"_149":22},["D",1760458697070],{"_93":94,"_128":129},{"_93":94,"_132":129},{"_99":135,"_95":43,"_102":136,"_137":103,"_138":139,"_140":141,"_142":109,"_143":144,"_145":146,"_147":96,"_148":94},"ahead","behind","diff",{"_287":288,"_167":289,"_290":291},"added",[],[],"modified",[],"hasChanges","sourceBranch","targetBranch",{"_102":139,"_104":296,"_267":297,"_95":43,"_137":144,"_99":269,"_138":-5,"_140":-5,"_270":271,"_106":-5,"_149":22,"_108":109,"_148":262,"_16":146,"_147":258},["D",1760276213018],["D",1760458811326],"pull","actionData","errors"]

charmaine

GitHub-To-Clay-remix

Mergecharmaine/mainintowhp/main