Hi

\"\n});\n```\n\n## Val Town Utility Functions\n\nVal Town provides several utility functions to help with common project tasks.\n\n### Importing Utilities\n\nAlways import utilities with version pins to avoid breaking changes:\n\n```ts\nimport { parseProject, readFile, serveFile } from \"https://esm.town/v/std/utils@85-main/index.ts\";\n```\n\n### Available Utilities\n\n\n#### **serveFile** - Serve project files with proper content types\n\nFor example, in Hono:\n\n```ts\n// serve all files in frontend/ and shared/\napp.get(\"/frontend/*\", c => serveFile(c.req.path, import.meta.url));\napp.get(\"/shared/*\", c => serveFile(c.req.path, import.meta.url));\n```\n\n#### **readFile** - Read files from within the project:\n\n```ts\n// Read a file from the project\nconst fileContent = await readFile(\"/frontend/index.html\", import.meta.url);\n```\n\n#### **listFiles** - List all files in the project\n\n```ts\nconst files = await listFiles(import.meta.url);\n```\n\n#### **parseProject** - Extract information about the current project from import.meta.url\n\nThis is useful for including info for linking back to a val, ie in \"view source\" urls:\n\n```ts\nconst projectVal = parseProject(import.meta.url);\nconsole.log(projectVal.username); // Owner of the project\nconsole.log(projectVal.name); // Project name\nconsole.log(projectVal.version); // Version number\nconsole.log(projectVal.branch); // Branch name\nconsole.log(projectVal.links.self.project); // URL to the project page\n```\n\nHowever, it's *extremely importing* to note that `parseProject` and other Standard Library utilities ONLY RUN ON THE SERVER.\nIf you need access to this data on the client, run it in the server and pass it to the client by splicing it into the HTML page\nor by making an API request for it.\n\n## Val Town Platform Specifics\n\n- **Redirects:** Use `return new Response(null, { status: 302, headers: { Location: \"/place/to/redirect\" }})` instead of `Response.redirect` which is broken\n- **Images:** Avoid external images or base64 images. Use emojis, unicode symbols, or icon fonts/libraries instead\n- **AI Image:** To inline generate an AI image use: `

`\n- **Storage:** DO NOT use the Deno KV module for storage\n- **Browser APIs:** DO NOT use the `alert()`, `prompt()`, or `confirm()` methods\n- **Weather Data:** Use open-meteo for weather data (doesn't require API keys) unless otherwise specified\n- **View Source:** Add a view source link by importing & using `import.meta.url.replace(\"ems.sh\", \"val.town)\"` (or passing this data to the client) and include `target=\"_top\"` attribute\n- **Error Debugging:** Add `` to HTML to capture client-side errors\n- **Error Handling:** Only use try...catch when there's a clear local resolution; Avoid catches that merely log or return 500s. Let errors bubble up with full context\n- **Environment Variables:** Use `Deno.env.get('keyname')` when you need to, but generally prefer APIs that don't require keys\n- **Imports:** Use `https://esm.sh` for npm and Deno dependencies to ensure compatibility on server and browser\n- **Storage Strategy:** Only use backend storage if explicitly required; prefer simple static client-side sites\n- **React Configuration:** When using React libraries, pin versions with `?deps=react@18.2.0,react-dom@18.2.0` and start the file with `/** @jsxImportSource https://esm.sh/react@18.2.0 */`\n- Ensure all React dependencies and sub-dependencies are pinned to the same version\n- **Styling:** Default to using TailwindCSS via `` unless otherwise specified\n\n## Project Structure and Design Patterns\n\n### Recommended Directory Structure\n```\n├── backend/\n│ ├── database/\n│ │ ├── migrations.ts # Schema definitions\n│ │ ├── queries.ts # DB query functions\n│ │ └── README.md\n│ └── routes/ # Route modules\n│ ├── [route].ts\n│ └── static.ts # Static file serving\n│ ├── index.ts # Main entry point\n│ └── README.md\n├── frontend/\n│ ├── components/\n│ │ ├── App.tsx\n│ │ └── [Component].tsx\n│ ├── favicon.svg\n│ ├── index.html # Main HTML template\n│ ├── index.tsx # Frontend JS entry point\n│ ├── README.md\n│ └── style.css\n├── README.md\n└── shared/\n ├── README.md\n └── utils.ts # Shared types and functions\n```\n\n### Backend (Hono) Best Practices\n\n- Hono is the recommended API framework\n- Main entry point should be `backend/index.ts`\n- **Static asset serving:** Use the utility functions to read and serve project files:\n ```ts\n import { readFile, serveFile } from \"https://esm.town/v/std/utils@85-main/index.ts\";\n \n // serve all files in frontend/ and shared/\n app.get(\"/frontend/*\", c => serveFile(c.req.path, import.meta.url));\n app.get(\"/shared/*\", c => serveFile(c.req.path, import.meta.url));\n \n // For index.html, often you'll want to bootstrap with initial data\n app.get(\"/\", async c => {\n let html = await readFile(\"/frontend/index.html\", import.meta.url);\n \n // Inject data to avoid extra round-trips\n const initialData = await fetchInitialData();\n const dataScript = ``;\n \n html = html.replace(\"\", `${dataScript}`);\n return c.html(html);\n });\n ```\n- Create RESTful API routes for CRUD operations\n- Always include this snippet at the top-level Hono app to re-throwing errors to see full stack traces:\n ```ts\n // Unwrap Hono errors to see original error details\n app.onError((err, c) => {\n throw err;\n });\n ```\n\n### Database Patterns\n- Run migrations on startup or comment out for performance\n- Change table names when modifying schemas rather than altering\n- Export clear query functions with proper TypeScript typing\n\n## Common Gotchas and Solutions\n\n1. **Environment Limitations:** \n - Val Town runs on Deno in a serverless context, not Node.js\n - Code in `shared/` must work in both frontend and backend environments\n - Cannot use `Deno` keyword in shared code\n - Use `https://esm.sh` for imports that work in both environments\n\n2. **SQLite Peculiarities:**\n - Limited support for ALTER TABLE operations\n - Create new tables with updated schemas and copy data when needed\n - Always run table creation before querying\n\n3. **React Configuration:**\n - All React dependencies must be pinned to 18.2.0\n - Always include `@jsxImportSource https://esm.sh/react@18.2.0` at the top of React files\n - Rendering issues often come from mismatched React versions\n\n4. **File Handling:**\n - Val Town only supports text files, not binary\n - Use the provided utilities to read files across branches and forks\n - For files in the project, use `readFile` helpers\n\n5. **API Design:**\n - `fetch` handler is the entry point for HTTP vals\n - Run the Hono app with `export default app.fetch // This is the entry point for HTTP vals`\n\n",[],[300],"267864c6-55f3-11f0-ab57-f69ea79377d9","shared","267865ac-55f3-11f0-ab57-f69ea79377d9",3,["D",1751316325925],[],[308],"f9ae8824-8a7e-11f0-b5f3-0224a6c84d84","index.tsx","fde87e54-56bc-11f0-b92d-f69ea79377d9","265c7fb8-55f3-11f0-894e-f69ea79377d9",77,["D",1757093840781],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React from \"https://esm.sh/react@18.2.0\";\nimport { createRoot } from \"https://esm.sh/react-dom@18.2.0/client\";\nimport {\n BrowserRouter,\n Navigate,\n Route,\n Routes,\n} from \"https://esm.sh/react-router-dom@6.22.3?deps=react@18.2.0\";\nimport {\n QueryClient,\n QueryClientProvider,\n} from \"https://esm.sh/@tanstack/react-query@4.29.5?deps=react@18.2.0,react-dom@18.2.0\";\nimport App from \"./components/App.tsx\";\nimport LoginPage from \"./components/LoginPage.tsx\";\nimport Dashboard from \"./components/Dashboard.tsx\";\nimport SummaryDetailView from \"./components/SummaryDetailView.tsx\";\nimport { AuthProvider, RequireAuth } from \"./components/AuthContext.tsx\";\n\n// Create a React Query client\nconst queryClient = new QueryClient({\n defaultOptions: {\n queries: {\n staleTime: 5 * 60 * 1000, // 5 minutes\n cacheTime: 10 * 60 * 1000, // 10 minutes\n },\n },\n});\n\nconst root = createRoot(document.getElementById(\"root\")!);\n\nroot.render(\n \n \n \n \n \n }>\n } />\n \n \n \n }\n />\n \n \n \n }\n />\n } />\n \n \n \n \n \n ,\n);\n",[],[323,315],"frontend","fdecd9b8-56bc-11f0-b053-f69ea79377d9","index.html","268930b2-55f3-11f0-a013-f69ea79377d9",27,["D",1751403016237],"\n\n\n \n \n OffX - Twitter Feed Summarizer\n \n \n\n\n

\n \n\n ",[],[323,325],"fae74498-8a8c-11f0-a4af-0224a6c84d84","api.ts","fae7456a-8a8c-11f0-a4af-0224a6c84d84","fad584e2-8a8c-11f0-9d6e-0224a6c84d84",84,["D",1757099855793],"/**\n * Shared API utility functions with authentication handling\n */\n\nconst API_BASE = \"/api\";\n\n/**\n * Get authentication headers from sessionStorage\n */\nexport function getAuthHeaders(): HeadersInit {\n const auth = sessionStorage.getItem(\"auth\");\n if (!auth) {\n throw new Error(\"No authentication found. Please log in.\");\n }\n \n return {\n \"Authorization\": auth,\n \"Content-Type\": \"application/json\",\n };\n}\n\n/**\n * Check if user is authenticated\n */\nexport function isAuthenticated(): boolean {\n return !!sessionStorage.getItem(\"auth\");\n}\n\n/**\n * Clear authentication and redirect to login\n */\nexport function clearAuthAndRedirect(): void {\n sessionStorage.removeItem(\"auth\");\n // Trigger a page reload to redirect to login via React Router\n window.location.href = \"/\";\n}\n\n/**\n * Authenticated fetch wrapper that automatically includes auth headers\n */\nexport async function authenticatedFetch(\n endpoint: string, \n options: RequestInit = {}\n): Promise {\n try {\n const headers = {\n ...getAuthHeaders(),\n ...options.headers,\n };\n\n const response = await fetch(`${API_BASE}${endpoint}`, {\n ...options,\n headers,\n });\n\n // Handle authentication failures\n if (response.status === 401) {\n clearAuthAndRedirect();\n throw new Error(\"Authentication failed. Please log in again.\");\n }\n\n return response;\n } catch (error) {\n // If auth headers couldn't be retrieved, clear auth and redirect\n if (error instanceof Error && error.message.includes(\"No authentication found\")) {\n clearAuthAndRedirect();\n }\n throw error;\n }\n}\n\n/**\n * Authenticated fetch with JSON response parsing\n */\nexport async function authenticatedFetchJson(\n endpoint: string, \n options: RequestInit = {}\n): Promise {\n const response = await authenticatedFetch(endpoint, options);\n \n if (!response.ok) {\n throw new Error(`Request failed: ${response.status} ${response.statusText}`);\n }\n \n return response.json();\n}\n\n/**\n * Authenticated GET request\n */\nexport function authenticatedGet(endpoint: string): Promise {\n return authenticatedFetchJson(endpoint, { method: \"GET\" });\n}\n\n/**\n * Authenticated POST request\n */\nexport function authenticatedPost(endpoint: string, data: any): Promise {\n return authenticatedFetchJson(endpoint, {\n method: \"POST\",\n body: JSON.stringify(data),\n });\n}\n\n/**\n * Authenticated PUT request\n */\nexport function authenticatedPut(endpoint: string, data: any): Promise {\n return authenticatedFetchJson(endpoint, {\n method: \"PUT\",\n body: JSON.stringify(data),\n });\n}\n\n/**\n * Authenticated DELETE request\n */\nexport function authenticatedDelete(endpoint: string): Promise {\n return authenticatedFetch(endpoint, { method: \"DELETE\" }).then(() => {});\n}",[],[323,341,333],"utils","fad5841a-8a8c-11f0-9d6e-0224a6c84d84",83,["D",1757099855673],[248],[323,341],"fafab24e-8a8c-11f0-ae36-0224a6c84d84","TopicManagement.tsx","83e5340a-8a7c-11f0-81bc-0224a6c84d84","069859e8-55f5-11f0-9cfc-f69ea79377d9",87,["D",1757099855878],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React, { useState } from \"https://esm.sh/react@18.2.0\";\nimport {\n useMutation,\n useQuery,\n useQueryClient,\n} from \"https://esm.sh/@tanstack/react-query@4.29.5?deps=react@18.2.0,react-dom@18.2.0\";\nimport {\n authenticatedGet,\n authenticatedPost,\n authenticatedPut,\n authenticatedDelete,\n} from \"../utils/api.ts\";\n\n// Types\ninterface Topic {\n id: number;\n title: string;\n description?: string;\n created_at: string;\n updated_at: string;\n}\n\ninterface CreateTopicData {\n title: string;\n description?: string;\n}\n\ninterface UpdateTopicData {\n title?: string;\n description?: string;\n}\n\n// API functions\nconst fetchTopics = async (): Promise => {\n return authenticatedGet(\"/topics\");\n};\n\nconst createTopic = async (data: CreateTopicData): Promise => {\n return authenticatedPost(\"/topics\", data);\n};\n\nconst updateTopic = async (\n id: number,\n data: UpdateTopicData,\n): Promise => {\n return authenticatedPut(`/topics/${id}`, data);\n};\n\nconst deleteTopic = async (id: number): Promise => {\n return authenticatedDelete(`/topics/${id}`);\n};\n\nexport default function TopicManagement() {\n const queryClient = useQueryClient();\n const [showCreateForm, setShowCreateForm] = useState(false);\n const [editingTopic, setEditingTopic] = useState(null);\n const [deleteConfirm, setDeleteConfirm] = useState(null);\n\n // Query for topics\n const {\n data: topics = [],\n isLoading,\n error,\n } = useQuery({\n queryKey: [\"topics\"],\n queryFn: fetchTopics,\n });\n\n // Mutations\n const createMutation = useMutation({\n mutationFn: createTopic,\n onSuccess: () => {\n queryClient.invalidateQueries({ queryKey: [\"topics\"] });\n setShowCreateForm(false);\n },\n });\n\n const updateMutation = useMutation<\n Topic,\n Error,\n { id: number; data: UpdateTopicData }\n >({\n mutationFn: ({ id, data }: { id: number; data: UpdateTopicData }) =>\n updateTopic(id, data),\n onSuccess: () => {\n queryClient.invalidateQueries({ queryKey: [\"topics\"] });\n setEditingTopic(null);\n },\n });\n\n const deleteMutation = useMutation({\n mutationFn: deleteTopic,\n onSuccess: () => {\n queryClient.invalidateQueries({ queryKey: [\"topics\"] });\n setDeleteConfirm(null);\n },\n });\n\n // Form handlers\n const handleCreateSubmit = (e: React.FormEvent) => {\n e.preventDefault();\n const formData = new FormData(e.currentTarget);\n const title = formData.get(\"title\") as string;\n const description = formData.get(\"description\") as string;\n\n if (!title.trim()) return;\n\n createMutation.mutate({\n title: title.trim(),\n description: description.trim() || undefined,\n });\n };\n\n const handleUpdateSubmit = (e: React.FormEvent) => {\n e.preventDefault();\n if (!editingTopic) return;\n\n const formData = new FormData(e.currentTarget);\n const title = formData.get(\"title\") as string;\n const description = formData.get(\"description\") as string;\n\n updateMutation.mutate({\n id: editingTopic.id,\n data: {\n title: title.trim(),\n description: description.trim() || undefined,\n },\n });\n };\n\n const handleDelete = (id: number) => {\n deleteMutation.mutate(id);\n };\n\n if (isLoading) {\n return (\n

\n );\n }\n\n if (error) {\n return (\n

Error Loading Topics

\n {error instanceof Error ? error.message : \"Unknown error occurred\"}\n

\n );\n }\n\n return (\n

Topic Management

\n setShowCreateForm(true)}\n className=\"bg-blue-500 hover:bg-blue-600 text-white px-4 py-2 rounded-lg transition-colors\"\n >\n Add Topic\n \n

\n\n {/* Create Form Modal */}\n {showCreateForm && (\n

Create New Topic

\n \n {createMutation.error && (\n

\n Error:{\" \"}\n {createMutation.error.message || \"Failed to create topic\"}\n

\n )}\n

\n )}\n\n {/* Edit Form Modal */}\n {editingTopic && (\n

Edit Topic

\n \n {updateMutation.error && (\n

\n Error:{\" \"}\n {updateMutation.error.message || \"Failed to update topic\"}\n

\n )}\n

\n )}\n\n {/* Delete Confirmation Modal */}\n {deleteConfirm && (\n

\n Confirm Delete\n

\n Are you sure you want to delete this topic? This action cannot be\n undone.\n

\n handleDelete(deleteConfirm)}\n disabled={deleteMutation.isLoading}\n className=\"flex-1 bg-red-500 hover:bg-red-600 disabled:bg-red-300 text-white py-2 rounded-md transition-colors\"\n >\n {deleteMutation.isLoading ? \"Deleting...\" : \"Delete\"}\n \n setDeleteConfirm(null)}\n className=\"flex-1 bg-gray-300 hover:bg-gray-400 text-gray-800 py-2 rounded-md transition-colors\"\n >\n Cancel\n \n

\n {deleteMutation.error && (\n

\n Error:{\" \"}\n {deleteMutation.error.message || \"Failed to delete topic\"}\n

\n )}\n

\n )}\n\n {/* Topics List */}\n {topics.length === 0\n ? (\n

No topics created yet

Get started by creating your first topic!

\n )\n : (\n

\n {topics.map((topic) => (\n \n

\n {topic.title}\n

\n #{topic.id}\n

\n\n {topic.description && (\n

\n {topic.description}\n

\n )}\n\n

\n Created: {new Date(topic.created_at).toLocaleDateString()}\n {topic.updated_at !== topic.created_at && (\n <>\n
\n Updated: {new Date(topic.updated_at).toLocaleDateString()}\n \n )}\n

\n\n

\n setEditingTopic(topic)}\n className=\"text-blue-500 hover:text-blue-700 text-sm font-medium transition-colors\"\n >\n Edit\n \n setDeleteConfirm(topic.id)}\n className=\"text-red-500 hover:text-red-700 text-sm font-medium transition-colors\"\n >\n Delete\n \n

\n ))}\n

\n )}\n \n );\n}\n",[],[323,356,348],"components","fb1ac6ec-8a8c-11f0-9b1d-0224a6c84d84","SummaryDetailView.tsx","83d4e280-8a7c-11f0-9d13-0224a6c84d84",91,["D",1757099855805],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React, { useState } from \"https://esm.sh/react@18.2.0\";\nimport {\n useNavigate,\n useParams,\n} from \"https://esm.sh/react-router-dom@6.22.3?deps=react@18.2.0\";\nimport {\n useQuery,\n useQueryClient,\n} from \"https://esm.sh/@tanstack/react-query@4.29.5?deps=react@18.2.0,react-dom@18.2.0\";\nimport ReactMarkdown from \"https://esm.sh/react-markdown@8.0.7?deps=react@18.2.0\";\nimport { authenticatedGet, authenticatedPost } from \"../utils/api.ts\";\n\ninterface Summarization {\n id: number;\n topic_id: number;\n content: string;\n prompt: string;\n post_count: number;\n created_at: string;\n time_period_start: string;\n time_period_end: string;\n topic_title?: string;\n}\n\ninterface SummarizationResponse {\n data: Summarization;\n}\n\ninterface RegenerateResponse {\n success: boolean;\n status: \"success\" | \"no_data\" | \"error\";\n message?: string;\n}\n\nconst API_BASE = \"/api\";\n\nexport default function SummaryDetailView() {\n const { id } = useParams<{ id: string }>();\n const navigate = useNavigate();\n const queryClient = useQueryClient();\n const [isRegenerating, setIsRegenerating] = useState(false);\n const [regenerateMessage, setRegenerateMessage] = useState(\n null,\n );\n\n // Fetch the specific summarization\n const { data: summarizationData, isLoading, error } = useQuery<\n SummarizationResponse\n >({\n queryKey: [\"summarization\", id],\n queryFn: async () => {\n if (!id) {\n throw new Error(\"No summarization ID provided\");\n }\n\n return authenticatedGet(`/summarizations/${id}`);\n },\n enabled: !!id, // Only run the query if we have an ID\n });\n\n const handleBack = () => {\n navigate(\"/dashboard\");\n };\n\n const handleRegenerate = async () => {\n if (!summarizationData?.data?.id) {\n setRegenerateMessage(\"Cannot regenerate: summarization ID not available\");\n return;\n }\n\n setIsRegenerating(true);\n setRegenerateMessage(null);\n\n try {\n const result = await authenticatedPost(\n `/summarizations/${summarizationData.data.id}/regenerate`,\n {}\n );\n\n if (!result.success) {\n throw new Error(result.message || \"Failed to regenerate summary\");\n }\n\n if (result.status === \"success\") {\n setRegenerateMessage(\n \"✅ Summary regenerated successfully! Refreshing...\",\n );\n\n // Invalidate queries to refetch updated data\n await queryClient.invalidateQueries([\"summarization\", id]);\n await queryClient.invalidateQueries([\"summarizations\"]);\n\n setTimeout(() => {\n setRegenerateMessage(null);\n }, 3000);\n } else if (result.status === \"no_data\") {\n setRegenerateMessage(\n \"⚠️ No recent posts found to regenerate summary from\",\n );\n } else {\n setRegenerateMessage(`⚠️ ${result.message || \"Unknown status\"}`);\n }\n } catch (error) {\n console.error(\"Error regenerating summary:\", error);\n setRegenerateMessage(\n `❌ Error: ${\n error instanceof Error ? error.message : \"Unknown error occurred\"\n }`,\n );\n } finally {\n setIsRegenerating(false);\n }\n };\n\n const formatDate = (dateString: string) => {\n return new Date(dateString).toLocaleDateString(\"en-US\", {\n year: \"numeric\",\n month: \"short\",\n day: \"numeric\",\n hour: \"2-digit\",\n minute: \"2-digit\",\n });\n };\n\n if (isLoading) {\n return (\n

\n );\n }\n\n if (error) {\n return (\n

\n \n ← Back to Summaries\n \n

\n\n

Error loading summarization

\n {error instanceof Error ? error.message : \"Unknown error occurred\"}\n

\n );\n }\n\n if (!summarizationData) {\n return (\n

\n \n ← Back to Summaries\n \n

\n\n

No summarization found.

\n );\n }\n\n const summarization = summarizationData.data;\n\n return (\n

\n {/* Navigation */}\n

\n \n ← Back to Summaries\n \n

\n\n {/* Header */}\n

\n {summarization.topic_title || `Topic ${summarization.topic_id}`}\n

\n\n

\n Created: {formatDate(summarization.created_at)}\n Posts: {summarization.post_count}\n \n Period: {formatDate(summarization.time_period_start)} -{\" \"}\n {formatDate(summarization.time_period_end)}\n \n

\n\n {/* Content */}\n

Summary Content

\n {summarization.content\n ? (\n (\n

\n {children}\n

\n ),\n h2: ({ children }: { children: React.ReactNode }) => (\n

\n {children}\n

\n ),\n h3: ({ children }: { children: React.ReactNode }) => (\n

\n {children}\n

\n ),\n p: ({ children }: { children: React.ReactNode }) => (\n

\n {children}\n

\n ),\n ul: ({ children }: { children: React.ReactNode }) => (\n

\n {children}\n \n ),\n ol: ({ children }: { children: React.ReactNode }) => (\n

\n {children}\n \n ),\n li: ({ children }: { children: React.ReactNode }) =>

{children}

,\n a: ({ href, children }: { href?: string; children: React.ReactNode }) => (\n \n {children}\n \n ),\n img: ({ src, alt }: { src?: string; alt?: string }) => (\n

\n {\n const target = e.target as HTMLImageElement;\n target.style.display = \"none\";\n const errorDiv = document.createElement(\"div\");\n errorDiv.className =\n \"text-gray-500 italic text-sm border border-gray-200 p-3 rounded\";\n errorDiv.textContent = `Failed to load image: ${\n alt || src\n }`;\n target.parentNode?.insertBefore(errorDiv, target);\n }}\n />\n

\n ),\n blockquote: ({ children }: { children: React.ReactNode }) => (\n

\n {children}\n

\n ),\n code: ({ children, inline }: { children: React.ReactNode; inline?: boolean }) => (\n inline\n ? (\n \n {children}\n \n )\n : (\n \n {children}\n \n )\n ),\n }}\n >\n {summarization.content}\n \n )\n :

No content available

}\n

\n\n {/* Additional media support for embedded tweets, videos, etc. */}\n

\n {/* Look for tweet URLs and render them with appropriate styling */}\n {summarization.content &&\n /https:\\/\\/(?:x|twitter)\\.com\\/\\w+\\/status\\/\\d+/.test(\n summarization.content,\n ) && (\n

🐦 Twitter Links Detected

\n This summary contains Twitter/X post links. Click the links\n above to view the original posts.\n

\n )}\n\n {/* Look for YouTube URLs */}\n {summarization.content &&\n /https:\\/\\/(?:www\\.)?(?:youtube\\.com\\/watch\\?v=|youtu\\.be\\/)[\\w-]+/\n .test(summarization.content) &&\n (\n

📺 YouTube Links Detected

\n This summary contains YouTube video links. Click the links\n above to view the videos.\n

\n )}\n\n {/* Look for image URLs */}\n {summarization.content &&\n /https?:\\/\\/[^\\s]+\\.(?:jpg|jpeg|png|gif|webp)/i.test(\n summarization.content,\n ) && (\n

🖼️ Images Detected

\n This summary contains image links that should be displayed\n above.\n

\n )}\n

\n\n {/* Actions */}\n

\n {regenerateMessage && (\n \n {regenerateMessage}\n

\n )}\n\n

\n \n {isRegenerating\n ? (\n \n \n \n \n \n \n \n Regenerating...\n \n )\n : (\n \"Re-run Summarization\"\n )}\n \n

\n \n );\n}\n",[],[323,356,358],"fb12fdb8-8a8c-11f0-9d5b-0224a6c84d84","SummarizationsList.tsx","83fee2ce-8a7c-11f0-b5d4-0224a6c84d84",90,["D",1757099855915],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React, { useState } from \"https://esm.sh/react@18.2.0\";\nimport { useQuery } from \"https://esm.sh/@tanstack/react-query@4.29.5?deps=react@18.2.0,react-dom@18.2.0\";\nimport { useNavigate } from \"https://esm.sh/react-router-dom@6.22.3?deps=react@18.2.0\";\nimport ReactPaginate from \"https://esm.sh/react-paginate@8.2.0\";\nimport { authenticatedFetch, authenticatedGet } from \"../utils/api.ts\";\n\ninterface Summarization {\n id: number;\n topic_id: number;\n content: string;\n prompt: string;\n post_count: number;\n created_at: string;\n time_period_start: string;\n time_period_end: string;\n topic_title?: string;\n}\n\ninterface PaginationInfo {\n page: number;\n limit: number;\n totalCount: number;\n totalPages: number;\n hasNext: boolean;\n hasPrev: boolean;\n}\n\ninterface SummarizationsResponse {\n data: Summarization[];\n pagination: PaginationInfo;\n}\n\ninterface Topic {\n id: number;\n title: string;\n description?: string;\n created_at: string;\n updated_at: string;\n}\n\ninterface TopicResponse {\n data: Topic[];\n}\n\nconst API_BASE = \"/api\";\n\nexport default function SummarizationsList() {\n const [currentPage, setCurrentPage] = useState(1);\n const [searchTerm, setSearchTerm] = useState(\"\");\n const [selectedTopicId, setSelectedTopicId] = useState(\"\");\n const navigate = useNavigate();\n const itemsPerPage = 10;\n\n // Fetch summarizations with pagination\n const { data: summarizationsData, isLoading, error } = useQuery<\n SummarizationsResponse\n >({\n queryKey: [\"summarizations\", currentPage, selectedTopicId, searchTerm],\n queryFn: async () => {\n const params = new URLSearchParams({\n page: currentPage.toString(),\n limit: itemsPerPage.toString(),\n });\n\n if (selectedTopicId) {\n params.append(\"topic_id\", selectedTopicId);\n }\n\n if (searchTerm) {\n params.append(\"search\", searchTerm);\n }\n\n const response = await authenticatedFetch(`/summarizations?${params}`);\n\n if (!response.ok) {\n throw new Error(\"Failed to fetch summarizations\");\n }\n\n return response.json();\n },\n });\n\n // Fetch topics for filter dropdown\n const { data: topicsData } = useQuery({\n queryKey: [\"topics\"],\n queryFn: () => authenticatedGet(\"/topics\"),\n });\n\n const handlePageChange = (selectedItem: { selected: number }) => {\n setCurrentPage(selectedItem.selected + 1);\n };\n\n const handleSearchChange = (e: React.ChangeEvent) => {\n setSearchTerm(e.target.value);\n setCurrentPage(1); // Reset to first page when searching\n };\n\n const handleTopicFilterChange = (e: React.ChangeEvent) => {\n setSelectedTopicId(e.target.value);\n setCurrentPage(1); // Reset to first page when filtering\n };\n\n const formatDate = (dateString: string) => {\n return new Date(dateString).toLocaleDateString(\"en-US\", {\n year: \"numeric\",\n month: \"short\",\n day: \"numeric\",\n hour: \"2-digit\",\n minute: \"2-digit\",\n });\n };\n\n const getExcerpt = (content: string, maxLength: number = 150) => {\n if (content.length <= maxLength) return content;\n return content.substring(0, maxLength) + \"...\";\n };\n\n const handleViewDetails = (summarizationId: number) => {\n navigate(`/summary/${summarizationId}`);\n };\n\n if (isLoading) {\n return (\n

\n {[...Array(5)].map((_, i) => (\n

\n ))}\n

\n );\n }\n\n if (error) {\n return (\n

Error loading summarizations

\n {error instanceof Error ? error.message : \"Unknown error occurred\"}\n

\n );\n }\n\n const summarizations = summarizationsData?.data || [];\n const pagination = summarizationsData?.pagination;\n\n return (\n

Summarizations

\n\n {/* Search and Filter Controls */}\n

\n \n

\n \n \n {topicsData?.data?.map((topic: any) => (\n \n ))}\n \n

\n\n {/* Summarizations List */}\n {summarizations.length === 0\n ? (\n

No summarizations found.

\n {(searchTerm || selectedTopicId) && (\n {\n setSearchTerm(\"\");\n setSelectedTopicId(\"\");\n setCurrentPage(1);\n }}\n className=\"mt-2 text-blue-600 hover:text-blue-800\"\n >\n Clear filters\n \n )}\n

\n )\n : (\n

\n {summarizations.map((summarization) => (\n \n

\n {summarization.topic_title ||\n `Topic ${summarization.topic_id}`}\n

\n {formatDate(summarization.created_at)} •{\" \"}\n {summarization.post_count} posts\n

\n handleViewDetails(summarization.id)}\n className=\"bg-blue-500 hover:bg-blue-600 text-white px-3 py-1 rounded text-sm\"\n >\n View Details\n \n

\n\n

\n {getExcerpt(summarization.content)}\n

\n\n

\n Period: {formatDate(summarization.time_period_start)} -{\" \"}\n {formatDate(summarization.time_period_end)}\n

\n ))}\n

\n )}\n\n {/* Pagination */}\n {pagination && pagination.totalPages > 1 && (\n

\n \n

\n )}\n\n {/* Results Summary */}\n {pagination && (\n

\n Showing {Math.min(\n (currentPage - 1) * itemsPerPage + 1,\n pagination.totalCount,\n )} to {Math.min(currentPage * itemsPerPage, pagination.totalCount)} of\n {\" \"}\n {pagination.totalCount} results\n

\n )}\n \n );\n}\n",[],[323,356,366],"fe22a7c8-56bc-11f0-8697-f69ea79377d9","LoginPage.tsx","fe22a8ea-56bc-11f0-8697-f69ea79377d9",30,["D",1751403016145],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React, { useState } from \"https://esm.sh/react@18.2.0\";\nimport {\n useLocation,\n useNavigate,\n} from \"https://esm.sh/react-router-dom@6.22.3?deps=react@18.2.0\";\nimport { useAuth } from \"./AuthContext.tsx\";\n\ninterface LocationState {\n from?: {\n pathname: string;\n };\n}\n\nexport default function LoginPage() {\n const navigate = useNavigate();\n const location = useLocation();\n const auth = useAuth();\n\n const [username, setUsername] = useState(\"\");\n const [password, setPassword] = useState(\"\");\n const [error, setError] = useState(\"\");\n const [loading, setLoading] = useState(false);\n\n const locationState = location.state as LocationState;\n const from = locationState?.from?.pathname || \"/dashboard\";\n\n const handleSubmit = async (e: React.FormEvent) => {\n e.preventDefault();\n setError(\"\");\n setLoading(true);\n\n try {\n const result = await auth.signin(username, password);\n\n if (result) {\n // Send them back to the page they tried to visit when they were\n // redirected to the login page. Use { replace: true } so we don't create\n // another entry in the history stack for the login page.\n navigate(from, { replace: true });\n } else {\n setError(\"Invalid username or password\");\n }\n } catch (error) {\n setError(\"An unexpected error occurred. Please try again.\");\n console.error(\"Login error:\", error);\n } finally {\n setLoading(false);\n }\n };\n\n // If user is already logged in, redirect to dashboard\n React.useEffect(() => {\n if (auth.user) {\n navigate(\"/dashboard\", { replace: true });\n }\n }, [auth.user, navigate]);\n\n return (\n

Welcome to OffX

\n This application helps you summarize your Twitter feed using AI.\n

\n\n

Dashboard Access

\n Login to access your dashboard and manage your summaries.\n

\n\n \n

\n );\n}\n",[],[323,356,374],"84d4378a-8a7c-11f0-87c1-0224a6c84d84","Dashboard.tsx","fde198a0-56bc-11f0-b7ab-f69ea79377d9",70,["D",1757092785713],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React, { useState } from \"https://esm.sh/react@18.2.0\";\nimport { useAuth } from \"./AuthContext.tsx\";\nimport { useNavigate } from \"https://esm.sh/react-router-dom@6.22.3?deps=react@18.2.0\";\nimport TopicManagement from \"./TopicManagement.tsx\";\nimport Configuration from \"./Configuration.tsx\";\nimport SummarizationsList from \"./SummarizationsList.tsx\";\n\nexport default function Dashboard() {\n const auth = useAuth();\n const navigate = useNavigate();\n const [activeTab, setActiveTab] = useState<\"overview\" | \"topics\" | \"summaries\" | \"config\">(\"overview\");\n\n const handleLogout = () => {\n auth.signout();\n navigate(\"/\");\n };\n\n const tabs = [\n { id: \"overview\" as const, label: \"Overview\", icon: \"📊\" },\n { id: \"topics\" as const, label: \"Topics\", icon: \"📝\" },\n { id: \"summaries\" as const, label: \"Summaries\", icon: \"📋\" },\n { id: \"config\" as const, label: \"Configuration\", icon: \"⚙️\" },\n ];\n\n return (\n

Dashboard

Welcome, {auth.user}!

\n \n

\n\n {/* Tab Navigation */}\n

\n {tabs.map((tab) => (\n setActiveTab(tab.id)}\n className={`${\n activeTab === tab.id\n ? \"border-blue-500 text-blue-600\"\n : \"border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300\"\n } whitespace-nowrap py-2 px-1 border-b-2 font-medium text-sm flex items-center gap-2`}\n >\n {tab.icon}\n {tab.label}\n \n ))}\n

\n\n {/* Tab Content */}\n {activeTab === \"overview\" && (\n

Recent Summaries

No summaries generated yet.

\n \n

Quick Stats

System statistics will appear here.

\n )}\n\n {activeTab === \"topics\" && }\n\n {activeTab === \"summaries\" && }\n\n {activeTab === \"config\" && }\n

\n );\n}\n",[],[323,356,382],"faf3763c-8a8c-11f0-bce7-0224a6c84d84","Configuration.tsx","83dde36c-8a7c-11f0-af15-0224a6c84d84",86,["D",1757099855866],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React from \"https://esm.sh/react@18.2.0\";\nimport { useQuery } from \"https://esm.sh/@tanstack/react-query@4.29.5?deps=react@18.2.0,react-dom@18.2.0\";\nimport { authenticatedGet } from \"../utils/api.ts\";\n\n// Types\ninterface AppConfiguration {\n email: {\n enabled: boolean;\n recipientEmail: string | null;\n frequency: string;\n maxSummaries: number;\n jobNotifications: boolean;\n adminEmail: string | null;\n };\n twitter: {\n summarization: {\n timeWindowHours: number;\n maxTweets: number;\n includeMedia: boolean;\n includeSentiment: boolean;\n emailNotifications: boolean;\n };\n filtering: {\n keywords: string[];\n excludeKeywords: string[];\n };\n };\n\n system: {\n environment: string;\n lastUpdated: string;\n };\n}\n\ninterface SystemStatus {\n services: {\n database: { status: string; message: string };\n email: { status: string; message: string };\n twitter: { status: string; message: string };\n };\n configuration: {\n requiredEnvVars: { [key: string]: boolean };\n optionalEnvVars: { [key: string]: boolean };\n };\n lastChecked: string;\n}\n\n// API functions\nconst fetchConfiguration = async (): Promise => {\n return authenticatedGet(\"/config\");\n};\n\nconst fetchSystemStatus = async (): Promise => {\n return authenticatedGet(\"/config/status\");\n};\n\n// Helper components\nconst StatusBadge: React.FC<{ status: string }> = ({ status }) => {\n const getStatusColor = (status: string) => {\n switch (status.toLowerCase()) {\n case \"healthy\":\n case \"configured\":\n return \"bg-green-100 text-green-800\";\n case \"disabled\":\n return \"bg-yellow-100 text-yellow-800\";\n case \"unhealthy\":\n case \"misconfigured\":\n return \"bg-red-100 text-red-800\";\n default:\n return \"bg-gray-100 text-gray-800\";\n }\n };\n\n return (\n \n {status}\n \n );\n};\n\nconst ConfigSection: React.FC<{ title: string; children: React.ReactNode }> = (\n { title, children },\n) => (\n

{title}

\n {children}\n

\n);\n\nconst ConfigRow: React.FC<\n { label: string; value: React.ReactNode; description?: string }\n> = ({\n label,\n value,\n description,\n}) => (\n

{label}

\n {description && (\n

{description}

\n )}\n

{value}

\n);\n\nexport default function Configuration() {\n const {\n data: config,\n isLoading: configLoading,\n error: configError,\n } = useQuery({\n queryKey: [\"configuration\"],\n queryFn: fetchConfiguration,\n refetchInterval: 30000, // Refresh every 30 seconds\n });\n\n const {\n data: status,\n isLoading: statusLoading,\n error: statusError,\n } = useQuery({\n queryKey: [\"system-status\"],\n queryFn: fetchSystemStatus,\n refetchInterval: 30000, // Refresh every 30 seconds\n });\n\n if (configLoading || statusLoading) {\n return (\n

\n {[1, 2, 3, 4].map((i) => (\n

\n ))}\n

\n );\n }\n\n if (configError || statusError) {\n return (\n

\n Error Loading Configuration\n

\n {configError?.message || statusError?.message ||\n \"Unknown error occurred\"}\n

\n );\n }\n\n if (!config || !status) {\n return (\n

No configuration data available

\n );\n }\n\n return (\n

\n {/* Header */}\n

\n System Configuration\n

\n Current application settings and system status\n

\n Last updated:{\" \"}\n {new Date(config.system.lastUpdated).toLocaleString()}\n

\n Environment:{\" \"}\n {config.system.environment}\n

\n\n {/* System Status */}\n \n

\n }\n description={status.services.database.message}\n />\n }\n description={status.services.email.message}\n />\n }\n description={status.services.twitter.message}\n />\n \n \n\n {/* Email Configuration */}\n \n

\n \n \n \n \n \n \n \n \n\n {/* Twitter & Summarization */}\n \n

\n \n \n \n \n \n \n \n\n {/* Filtering */}\n \n

\n 0\n ? config.twitter.filtering.keywords.join(\", \")\n : \"None\"}\n description=\"Only process tweets containing these keywords\"\n />\n 0\n ? config.twitter.filtering.excludeKeywords.join(\", \")\n : \"None\"}\n description=\"Skip tweets containing these keywords\"\n />\n \n \n\n {/* Environment Variables Status */}\n \n

\n Required Variables\n

{key}: \n \n {configured ? \"Set\" : \"Missing\"}\n \n

\n Optional Variables\n

{key}: \n \n {configured ? \"Set\" : \"Default\"}\n \n

\n \n\n {/* Help Text */}\n

\n \n \n \n

\n Configuration Management\n

\n To modify these settings, update the environment variables in Val\n Town's interface. Changes will be reflected here after the next\n restart or refresh.\n

\n );\n}\n",[],[323,356,390],"fde4fd38-56bc-11f0-aabd-f69ea79377d9","AuthContext.tsx","fde4fdec-56bc-11f0-aabd-f69ea79377d9",25,["D",1751403016176],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React, {\n createContext,\n useContext,\n useState,\n} from \"https://esm.sh/react@18.2.0\";\nimport {\n Navigate,\n useLocation,\n} from \"https://esm.sh/react-router-dom@6.22.3?deps=react@18.2.0\";\n\ninterface AuthContextType {\n user: string | null;\n signin: (username: string, password: string) => Promise;\n signout: () => void;\n}\n\nconst AuthContext = createContext(null!);\n\nexport function AuthProvider({ children }: { children: React.ReactNode }) {\n const [user, setUser] = useState(null);\n\n const signin = async (username: string, password: string) => {\n try {\n // Create basic auth header\n const authHeader = \"Basic \" + btoa(username + \":\" + password);\n\n const response = await fetch(\"/api/login\", {\n headers: {\n \"Authorization\": authHeader,\n },\n });\n\n if (response.ok) {\n setUser(username);\n // Store auth in sessionStorage for persistence\n sessionStorage.setItem(\"auth\", authHeader);\n return true;\n } else {\n return false;\n }\n } catch (error) {\n console.error(\"Login error:\", error);\n return false;\n }\n };\n\n const signout = () => {\n setUser(null);\n sessionStorage.removeItem(\"auth\");\n };\n\n // Check if we have auth in sessionStorage on mount\n React.useEffect(() => {\n const auth = sessionStorage.getItem(\"auth\");\n if (auth) {\n // Verify the auth is still valid\n fetch(\"/api/login\", {\n headers: {\n \"Authorization\": auth,\n },\n }).then((response) => {\n if (response.ok) {\n // Extract username from Basic auth\n try {\n const decoded = atob(auth.replace(\"Basic \", \"\"));\n const username = decoded.split(\":\")[0];\n setUser(username);\n } catch (e) {\n console.error(\"Failed to decode auth\", e);\n sessionStorage.removeItem(\"auth\");\n }\n } else {\n sessionStorage.removeItem(\"auth\");\n }\n }).catch(() => {\n sessionStorage.removeItem(\"auth\");\n });\n }\n }, []);\n\n const value = { user, signin, signout };\n\n return {children};\n}\n\nexport function useAuth() {\n return useContext(AuthContext);\n}\n\nexport function RequireAuth({ children }: { children: JSX.Element }) {\n const auth = useAuth();\n const location = useLocation();\n\n if (!auth.user) {\n // Redirect to the login page but save the current location\n return ;\n }\n\n return children;\n}\n",[],[323,356,398],"faee2a4c-8a8c-11f0-a0a1-0224a6c84d84","App.tsx","fddc183a-56bc-11f0-824f-f69ea79377d9",85,["D",1757099855801],"/** @jsxImportSource https://esm.sh/react@18.2.0 */\nimport React from \"https://esm.sh/react@18.2.0\";\nimport { Outlet } from \"https://esm.sh/react-router-dom@6.22.3?deps=react@18.2.0\";\n\nexport default function App(): React.ReactElement {\n return (\n

OffX

Twitter Feed Summarizer

\n\n

\n \n

\n\n \n

\n );\n}\n",[],[323,356,406],"0698589e-55f5-11f0-9cfc-f69ea79377d9",10,["D",1751317131440],[239,240,241,242,243,244,245,246],[323,356],"265c7ee6-55f3-11f0-894e-f69ea79377d9",["D",1751316325744],[238,247,249,250],[323],"bad7c436-8ad7-11f0-8087-0224a6c84d84","twitter-test.http.ts","8487434e-8a7c-11f0-af15-0224a6c84d84",["D",1757131960563],"http","import { initializeTwitterClient, storeTweets } from \"./twitter-api.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\nimport { getRecentPosts } from \"./database/queries.ts\";\nimport { TwitterProcessor } from \"./twitter-processor.ts\";\n\n/**\n * HTTP endpoint to test Twitter API integration\n * This endpoint allows manual triggering of the Twitter fetch process\n * and returns diagnostic information\n */\nexport default async function (req: Request): Promise {\n // Check for basic auth\n const authHeader = req.headers.get(\"Authorization\");\n if (!authHeader || !authHeader.startsWith(\"Basic \")) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n const base64Credentials = authHeader.split(\" \")[1];\n const credentials = atob(base64Credentials);\n const [username, password] = credentials.split(\":\");\n\n const expectedUsername = Deno.env.get(\"AUTH_USERNAME\");\n const expectedPassword = Deno.env.get(\"AUTH_PASSWORD\");\n\n if (username !== expectedUsername || password !== expectedPassword) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n // Process the request\n try {\n const url = new URL(req.url);\n const action = url.searchParams.get(\"action\") || \"status\";\n\n // Initialize database\n await initializeDatabase();\n\n let responseData: any = {\n status: \"ok\",\n timestamp: new Date().toISOString(),\n };\n\n if (action === \"fetch\") {\n // Get processing options from query params\n const includeRetweets = url.searchParams.get(\"retweets\") === \"true\";\n const includeReplies = url.searchParams.get(\"replies\") === \"true\";\n const keywords = url.searchParams.get(\"keywords\")?.split(\",\").map((k) =>\n k.trim()\n ) || undefined;\n\n // Create processor config\n const processorConfig = {\n includeRetweets,\n includeReplies,\n keywords,\n };\n\n // Fetch tweets from Twitter API\n const twitterClient = initializeTwitterClient();\n const rawTweets = await twitterClient.fetchHomeTimeline(10);\n\n // Process tweets\n const processor = new TwitterProcessor(processorConfig);\n const processedTweets = processor.processTweets(rawTweets);\n\n // Store tweets\n const newTweetsCount = await storeTweets(processedTweets);\n\n responseData = {\n ...responseData,\n action: \"fetch\",\n fetchedCount: rawTweets.length,\n processedCount: processedTweets.length,\n newCount: newTweetsCount,\n processorConfig,\n tweets: processedTweets.slice(0, 5), // Return first 5 tweets as preview\n };\n } else if (action === \"recent\") {\n // Get recent posts from database\n const limit = parseInt(url.searchParams.get(\"limit\") || \"10\", 10);\n const recentPosts = await getRecentPosts(limit);\n\n responseData = {\n ...responseData,\n action: \"recent\",\n count: recentPosts.length,\n posts: recentPosts,\n };\n } else if (action === \"config\") {\n // Get current processor configuration from environment\n const config = TwitterProcessor.loadConfigFromEnv();\n\n responseData = {\n ...responseData,\n action: \"config\",\n config,\n };\n }\n\n return new Response(JSON.stringify(responseData, null, 2), {\n status: 200,\n headers: { \"Content-Type\": \"application/json\" },\n });\n } catch (error: unknown) {\n console.error(\"Error in Twitter test endpoint:\", error);\n\n return new Response(\n JSON.stringify(\n {\n status: \"error\",\n message: error instanceof Error ? error.message : String(error),\n timestamp: new Date().toISOString(),\n },\n null,\n 2,\n ),\n {\n status: 500,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n}\n",[],[176,423],"a5decd36-8a7d-11f0-aa86-0224a6c84d84","twitter-summarization.ts","845e360c-8a7c-11f0-9d13-0224a6c84d84",76,["D",1757093270676],"import { OpenAI } from \"https://esm.town/v/std/openai\";\nimport {\n createSummarization,\n getAllTopics,\n getPostsForTopic,\n getSummarizationsForTopic,\n} from \"./database/queries.ts\";\nimport type { Post, Topic } from \"./database/queries.ts\";\nimport { TwitterSentimentAnalyzer } from \"./twitter-sentiment.ts\";\n\n/**\n * OpenAI-powered summarization service for grouped tweets\n * Extends the existing sentiment analysis to provide topic-based summaries\n */\n\nexport interface TopicSummary {\n topicId: number;\n topicTitle: string;\n summary: string;\n keyPoints: string[];\n tweetCount: number;\n timeWindow: string;\n overallSentiment: {\n sentiment: \"positive\" | \"negative\" | \"neutral\";\n confidence: number;\n score: number;\n };\n trendingKeywords: string[];\n influentialAuthors: string[];\n}\n\nexport interface SummarizationRequest {\n topicId?: number;\n timeWindowHours?: number;\n maxTweets?: number;\n includeMedia?: boolean;\n includeSentiment?: boolean;\n}\n\nexport interface SummarizationResult {\n success: boolean;\n generated: number;\n stored: number;\n errors: string[];\n summaries: TopicSummary[];\n}\n\n/**\n * Service class for generating AI-powered summaries of tweets grouped by topics\n */\nexport class TwitterSummarizationService {\n private sentimentAnalyzer: TwitterSentimentAnalyzer;\n\n constructor() {\n // Initialize sentiment analyzer with default config\n this.sentimentAnalyzer = new TwitterSentimentAnalyzer();\n }\n\n /**\n * Generate a comprehensive summary for a specific topic\n */\n async generateTopicSummary(\n topicId: number,\n options: Omit = {},\n ): Promise {\n try {\n const {\n timeWindowHours = 24,\n maxTweets = 50,\n includeMedia = false,\n includeSentiment = true,\n } = options;\n\n // Get the topic details\n const topics = await getAllTopics();\n const topic = topics.find((t) => t.id === topicId);\n if (!topic) {\n throw new Error(`Topic with ID ${topicId} not found`);\n }\n\n // Get recent posts for this topic\n const posts = await getPostsForTopic(topicId, maxTweets);\n\n if (posts.length === 0) {\n console.log(`No posts found for topic ${topicId}`);\n return null;\n }\n\n // Filter posts by time window\n const cutoffTime = new Date(\n Date.now() - timeWindowHours * 60 * 60 * 1000,\n );\n const recentPosts = posts.filter((post) =>\n new Date(post.created_at) > cutoffTime\n );\n\n if (recentPosts.length === 0) {\n console.log(\n `No recent posts found for topic ${topicId} in the last ${timeWindowHours} hours`,\n );\n return null;\n }\n\n // Analyze sentiment if requested\n let sentimentData: {\n sentiment: \"positive\" | \"negative\" | \"neutral\";\n confidence: number;\n score: number;\n } | null = null;\n if (includeSentiment) {\n try {\n const sentimentResults = await this.sentimentAnalyzer.analyzeBatch(\n recentPosts,\n );\n sentimentData = this.aggregateSentiment(sentimentResults);\n } catch (error: unknown) {\n console.warn(\n \"Failed to analyze sentiment, continuing without it:\",\n error,\n );\n sentimentData = {\n sentiment: \"neutral\" as const,\n confidence: 0.5,\n score: 0,\n };\n }\n }\n\n // Group and format posts for summarization\n const groupedData = this.groupPostsForSummarization(recentPosts);\n\n // Generate summary using OpenAI\n const summary = await this.generateSummaryWithAI(\n topic,\n groupedData,\n sentimentData,\n );\n\n // Create the final topic summary\n const topicSummary: TopicSummary = {\n topicId,\n topicTitle: topic.title,\n summary: summary.mainSummary,\n keyPoints: summary.keyPoints,\n tweetCount: recentPosts.length,\n timeWindow: `${timeWindowHours} hours`,\n overallSentiment: sentimentData || {\n sentiment: \"neutral\",\n confidence: 0.5,\n score: 0,\n },\n trendingKeywords: summary.trendingKeywords,\n influentialAuthors: summary.influentialAuthors,\n };\n\n return topicSummary;\n } catch (error: unknown) {\n console.error(`Error generating summary for topic ${topicId}:`, error);\n throw error;\n }\n }\n\n /**\n * Generate summaries for all topics with recent activity\n */\n async generateAllTopicSummaries(\n options: Omit = {},\n ): Promise {\n try {\n const topics = await getAllTopics();\n const summaries: TopicSummary[] = [];\n\n console.log(`Generating summaries for ${topics.length} topics...`);\n\n for (const topic of topics) {\n try {\n console.log(`Processing topic: ${topic.title} (ID: ${topic.id})`);\n const summary = await this.generateTopicSummary(topic.id!, options);\n if (summary) {\n summaries.push(summary);\n console.log(`✓ Generated summary for topic: ${topic.title}`);\n } else {\n console.log(`- No recent activity for topic: ${topic.title}`);\n }\n } catch (error: unknown) {\n console.error(\n `Failed to generate summary for topic ${topic.id} (${topic.title}):`,\n error,\n );\n // Continue with other topics\n }\n }\n\n console.log(`Generated ${summaries.length} summaries successfully`);\n return summaries;\n } catch (error: unknown) {\n console.error(\"Error generating all topic summaries:\", error);\n throw error;\n }\n }\n\n /**\n * Store a topic summary in the database\n */\n async storeTopicSummary(topicSummary: TopicSummary): Promise {\n try {\n const summaryContent = JSON.stringify({\n summary: topicSummary.summary,\n keyPoints: topicSummary.keyPoints,\n sentiment: topicSummary.overallSentiment,\n tweetCount: topicSummary.tweetCount,\n timeWindow: topicSummary.timeWindow,\n trendingKeywords: topicSummary.trendingKeywords,\n influentialAuthors: topicSummary.influentialAuthors,\n });\n\n // Parse the time window to extract start and end dates\n const [startDate, endDate] = topicSummary.timeWindow.split(\" to \");\n\n const summarizationId = await createSummarization({\n topic_id: topicSummary.topicId,\n content: summaryContent,\n prompt: \"Topic summary generated automatically\",\n post_count: topicSummary.tweetCount,\n time_period_start: startDate.trim(),\n time_period_end: endDate.trim(),\n });\n\n return summarizationId;\n } catch (error: unknown) {\n console.error(\"Error storing topic summary:\", error);\n return null;\n }\n }\n\n /**\n * Generate and store summaries for all topics\n */\n async generateAndStoreSummaries(\n options: Omit = {},\n ): Promise {\n const errors: string[] = [];\n let generated = 0;\n let stored = 0;\n const summaries: TopicSummary[] = [];\n\n try {\n const generatedSummaries = await this.generateAllTopicSummaries(options);\n generated = generatedSummaries.length;\n summaries.push(...generatedSummaries);\n\n for (const summary of generatedSummaries) {\n try {\n const summaryId = await this.storeTopicSummary(summary);\n if (summaryId) {\n stored++;\n console.log(\n `✓ Stored summary for topic: ${summary.topicTitle} (ID: ${summaryId})`,\n );\n } else {\n const error =\n `Failed to store summary for topic ${summary.topicId} (${summary.topicTitle})`;\n errors.push(error);\n console.error(error);\n }\n } catch (error: unknown) {\n const errorMessage =\n `Error storing summary for topic ${summary.topicId} (${summary.topicTitle}): ${error}`;\n errors.push(errorMessage);\n console.error(errorMessage);\n }\n }\n\n return {\n success: errors.length === 0,\n generated,\n stored,\n errors,\n summaries,\n };\n } catch (error: unknown) {\n const errorMessage = `Error in generateAndStoreSummaries: ${error}`;\n errors.push(errorMessage);\n console.error(errorMessage);\n return {\n success: false,\n generated,\n stored,\n errors,\n summaries,\n };\n }\n }\n\n /**\n * Group posts for effective summarization\n */\n private groupPostsForSummarization(posts: Post[]): {\n authorCounts: Record;\n contentSample: string[];\n mediaCount: number;\n timeSpread: { earliest: string; latest: string };\n } {\n const authorCounts: Record = {};\n const contentSample: string[] = [];\n let mediaCount = 0;\n\n // Count authors and collect content\n for (const post of posts) {\n authorCounts[post.author] = (authorCounts[post.author] || 0) + 1;\n contentSample.push(`[@${post.author}]: ${post.content}`);\n\n // Count media (this would need to be enhanced based on actual media data structure)\n if (\n post.content.includes(\"http\") ||\n post.content.includes(\"pic.twitter.com\")\n ) {\n mediaCount++;\n }\n }\n\n // Calculate time spread\n const timestamps = posts.map((p) => new Date(p.created_at).getTime())\n .sort();\n const timeSpread = {\n earliest: new Date(timestamps[0]).toISOString(),\n latest: new Date(timestamps[timestamps.length - 1]).toISOString(),\n };\n\n return {\n authorCounts,\n contentSample: contentSample.slice(0, 20), // Limit for token efficiency\n mediaCount,\n timeSpread,\n };\n }\n\n /**\n * Generate summary using OpenAI\n */\n private async generateSummaryWithAI(\n topic: Topic,\n groupedData: ReturnType,\n sentimentData: any,\n ): Promise<{\n mainSummary: string;\n keyPoints: string[];\n trendingKeywords: string[];\n influentialAuthors: string[];\n }> {\n const prompt = this.buildSummarizationPrompt(\n topic,\n groupedData,\n sentimentData,\n );\n\n try {\n const openai = new OpenAI();\n\n const response = await openai.chat.completions.create({\n model: \"gpt-3.5-turbo\",\n messages: [\n {\n role: \"system\",\n content:\n `You are an expert social media analyst specializing in Twitter content summarization. \n Provide concise, insightful summaries that capture the key themes, sentiment, and notable patterns in the conversation.\n Always respond with valid JSON in the exact format requested.`,\n },\n {\n role: \"user\",\n content: prompt,\n },\n ],\n max_tokens: 800,\n temperature: 0.7,\n response_format: { type: \"json_object\" },\n });\n\n const result = response.choices[0].message.content?.trim();\n if (!result) {\n throw new Error(\"No response from OpenAI\");\n }\n\n const summaryData = JSON.parse(result);\n\n // Validate response structure\n if (!summaryData.mainSummary || !Array.isArray(summaryData.keyPoints)) {\n throw new Error(\"Invalid response format from OpenAI\");\n }\n\n return {\n mainSummary: summaryData.mainSummary,\n keyPoints: summaryData.keyPoints || [],\n trendingKeywords: summaryData.trendingKeywords || [],\n influentialAuthors: summaryData.influentialAuthors || [],\n };\n } catch (error: unknown) {\n console.error(\"Error generating summary with AI:\", error);\n\n // Fallback to simple summary\n return this.generateFallbackSummary(topic, groupedData);\n }\n }\n\n /**\n * Build the summarization prompt for OpenAI\n */\n private buildSummarizationPrompt(\n topic: Topic,\n groupedData: ReturnType,\n sentimentData: any,\n ): string {\n const { authorCounts, contentSample, mediaCount, timeSpread } = groupedData;\n\n const topAuthors = Object.entries(authorCounts)\n .sort(([, a], [, b]) => b - a)\n .slice(0, 5)\n .map(([author, count]) => `${author} (${count} posts)`);\n\n const sentimentInfo = sentimentData\n ? `Overall sentiment: ${sentimentData.sentiment} (confidence: ${\n sentimentData.confidence.toFixed(2)\n }, score: ${sentimentData.score.toFixed(2)})`\n : \"Sentiment analysis not available\";\n\n return `\nAnalyze and summarize the following Twitter conversation about \"${topic.title}\":\n\nTOPIC DESCRIPTION: ${topic.description || \"No description provided\"}\n\nCONVERSATION DATA:\n- Total posts: ${contentSample.length}\n- Time range: ${timeSpread.earliest} to ${timeSpread.latest}\n- Posts with media: ${mediaCount}\n- ${sentimentInfo}\n\nTOP CONTRIBUTORS: ${topAuthors.join(\", \")}\n\nSAMPLE POSTS:\n${contentSample.join(\"\\n\\n\")}\n\nPlease provide a comprehensive analysis in the following JSON format:\n{\n \"mainSummary\": \"A 2-3 sentence overview of the main themes and discussion points\",\n \"keyPoints\": [\"Point 1\", \"Point 2\", \"Point 3\", \"Point 4\", \"Point 5\"],\n \"trendingKeywords\": [\"keyword1\", \"keyword2\", \"keyword3\", \"keyword4\", \"keyword5\"],\n \"influentialAuthors\": [\"author1\", \"author2\", \"author3\"]\n}\n\nFocus on:\n1. The main themes and topics being discussed\n2. Any notable trends, events, or developments mentioned\n3. The overall tone and sentiment of the conversation\n4. Key players or influencers in the discussion\n5. Important keywords that capture the essence of the conversation\n\nProvide only valid JSON, no additional text.\n`;\n }\n\n /**\n * Generate a simple fallback summary when AI fails\n */\n private generateFallbackSummary(\n topic: Topic,\n groupedData: ReturnType,\n ): {\n mainSummary: string;\n keyPoints: string[];\n trendingKeywords: string[];\n influentialAuthors: string[];\n } {\n const { authorCounts, contentSample, mediaCount } = groupedData;\n\n const topAuthors = Object.entries(authorCounts)\n .sort(([, a], [, b]) => b - a)\n .slice(0, 3)\n .map(([author]) => author);\n\n return {\n mainSummary:\n `Discussion about ${topic.title} with ${contentSample.length} posts from ${\n Object.keys(authorCounts).length\n } contributors. ${mediaCount} posts included media content.`,\n keyPoints: [\n `${contentSample.length} total posts analyzed`,\n `${Object.keys(authorCounts).length} unique contributors`,\n `${mediaCount} posts with media content`,\n `Most active contributor: ${topAuthors[0] || \"Unknown\"}`,\n \"Detailed analysis unavailable due to processing error\",\n ],\n trendingKeywords: [topic.title.toLowerCase()],\n influentialAuthors: topAuthors,\n };\n }\n\n /**\n * Aggregate sentiment analysis results\n */\n private aggregateSentiment(sentimentResults: any[]): {\n sentiment: \"positive\" | \"negative\" | \"neutral\";\n confidence: number;\n score: number;\n } {\n if (sentimentResults.length === 0) {\n return { sentiment: \"neutral\", confidence: 0.5, score: 0 };\n }\n\n // Calculate average sentiment score and confidence\n const avgScore =\n sentimentResults.reduce((sum, result) => sum + result.score, 0) /\n sentimentResults.length;\n const avgConfidence =\n sentimentResults.reduce((sum, result) => sum + result.confidence, 0) /\n sentimentResults.length;\n\n // Determine overall sentiment based on average score\n let sentiment: \"positive\" | \"negative\" | \"neutral\" = \"neutral\";\n if (avgScore > 0.1) {\n sentiment = \"positive\";\n } else if (avgScore < -0.1) {\n sentiment = \"negative\";\n }\n\n return {\n sentiment,\n confidence: avgConfidence,\n score: avgScore,\n };\n }\n\n /**\n * Get recent summaries for a topic (for checking if summarization is needed)\n */\n async getRecentSummaries(\n topicId: number,\n hoursBack: number = 24,\n ): Promise {\n try {\n return await getSummarizationsForTopic(topicId, 10); // Get up to 10 recent summaries\n } catch (error: unknown) {\n console.error(\n `Error getting recent summaries for topic ${topicId}:`,\n error,\n );\n return [];\n }\n }\n}\n",[],[176,431],"83f47b4a-8a7c-11f0-a88e-0224a6c84d84","twitter-summarization-job.cron.ts","83f47c12-8a7c-11f0-a88e-0224a6c84d84",38,["D",1757092784206],"interval","/**\n * Scheduled job for generating AI-powered summaries of tweets grouped by topics\n * This cron job runs periodically to analyze recent tweets and generate summaries\n * \n * Configure this in Val Town's interface to run at your desired interval\n * Recommended: Every 6 hours or daily\n */\n\nimport { TwitterSummarizationService } from \"./twitter-summarization.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\nimport { email } from \"https://esm.town/v/std/email\";\n\nexport default async function (): Promise {\n const startTime = Date.now();\n console.log(\"🤖 Starting scheduled summarization job...\");\n\n try {\n // Initialize database\n await initializeDatabase();\n\n // Initialize the summarization service\n const summarizationService = new TwitterSummarizationService();\n\n // Configure summarization options\n const options = {\n timeWindowHours: parseInt(Deno.env.get(\"TWITTER_SUMMARIZATION_TIME_WINDOW\") || \"24\"),\n maxTweets: parseInt(Deno.env.get(\"TWITTER_SUMMARIZATION_MAX_TWEETS\") || \"50\"),\n includeMedia: Deno.env.get(\"TWITTER_SUMMARIZATION_INCLUDE_MEDIA\") !== \"false\",\n includeSentiment: Deno.env.get(\"TWITTER_SUMMARIZATION_INCLUDE_SENTIMENT\") !== \"false\",\n };\n\n console.log(\"📊 Summarization options:\", options);\n\n // Generate and store summaries\n const result = await summarizationService.generateAndStoreSummaries(options);\n\n // Calculate execution time\n const executionTime = Date.now() - startTime;\n\n // Log results\n console.log(\"📊 Summarization job completed:\");\n console.log(` - Generated: ${result.generated} summaries`);\n console.log(` - Stored: ${result.stored} summaries`);\n console.log(` - Errors: ${result.errors.length}`);\n console.log(` - Execution time: ${executionTime}ms`);\n\n if (result.errors.length > 0) {\n console.error(\"❌ Errors encountered:\");\n result.errors.forEach(error => console.error(` - ${error}`));\n }\n\n // Send email notification with results\n const shouldSendEmail = Deno.env.get(\"TWITTER_SUMMARIZATION_EMAIL_NOTIFICATIONS\") === \"true\";\n const userEmail = Deno.env.get(\"USER_EMAIL\");\n\n if (shouldSendEmail && userEmail) {\n try {\n await sendSummarizationEmail(result, executionTime, userEmail);\n console.log(\"📧 Email notification sent successfully\");\n } catch (emailError) {\n console.error(\"📧 Failed to send email notification:\", emailError);\n }\n }\n\n console.log(\"✅ Summarization job completed successfully\");\n\n } catch (error: unknown) {\n const executionTime = Date.now() - startTime;\n const errorMessage = `Summarization job failed: ${error}`;\n \n console.error(\"❌ Summarization job failed:\", error);\n\n // Try to send error notification email\n const shouldSendEmail = Deno.env.get(\"TWITTER_SUMMARIZATION_EMAIL_NOTIFICATIONS\") === \"true\";\n const userEmail = Deno.env.get(\"USER_EMAIL\");\n\n if (shouldSendEmail && userEmail) {\n try {\n await sendErrorEmail(error, executionTime, userEmail);\n console.log(\"📧 Error notification email sent\");\n } catch (emailError) {\n console.error(\"📧 Failed to send error notification email:\", emailError);\n }\n }\n\n throw error; // Re-throw to mark job as failed\n }\n}\n\n/**\n * Send email notification with summarization results\n */\nasync function sendSummarizationEmail(\n result: any,\n executionTime: number,\n recipientEmail: string,\n): Promise {\n const subject = `OffX: Summarization Report - ${result.generated} summaries generated`;\n \n const summariesList = result.summaries\n .map((s: any) => \n `• ${s.topicTitle}: ${s.tweetCount} tweets (${s.overallSentiment.sentiment} sentiment)`\n )\n .join('\\n');\n\n const html = `\n

🤖 Twitter Summarization Report

\n \n

📊 Summary

Generated: ${result.generated} summaries
Stored: ${result.stored} summaries
Execution time: ${(executionTime / 1000).toFixed(2)}s
Status: ${result.success ? '✅ Success' : '❌ Partial failure'}

\n\n ${result.summaries.length > 0 ? `\n

📝 Generated Summaries

${summariesList}

\n ` : ''}\n\n ${result.errors.length > 0 ? `\n

❌ Errors (${result.errors.length})

${error}

\n ` : ''}\n\n

Generated at ${new Date().toISOString()}

This is an automated report from your Twitter Summarization service.

To disable these notifications, set TWITTER_SUMMARIZATION_EMAIL_NOTIFICATIONS=false

\n `;\n\n const text = `\nTwitter Summarization Report\n\nGenerated: ${result.generated} summaries\nStored: ${result.stored} summaries\nExecution time: ${(executionTime / 1000).toFixed(2)}s\nStatus: ${result.success ? 'Success' : 'Partial failure'}\n\n${result.summaries.length > 0 ? `\nGenerated Summaries:\n${summariesList}\n` : ''}\n\n${result.errors.length > 0 ? `\nErrors (${result.errors.length}):\n${result.errors.map((error: string) => `- ${error}`).join('\\n')}\n` : ''}\n\nGenerated at ${new Date().toISOString()}\n `;\n\n await email({\n to: recipientEmail,\n subject,\n html,\n text,\n });\n}\n\n/**\n * Send error notification email\n */\nasync function sendErrorEmail(\n error: any,\n executionTime: number,\n recipientEmail: string,\n): Promise {\n const subject = \"OffX: Summarization Job Failed\";\n \n const html = `\n

❌ Summarization Job Failed

\n \n

Error Details

Error: ${error instanceof Error ? error.message : String(error) || error}

Execution time: ${(executionTime / 1000).toFixed(2)}s

Timestamp: ${new Date().toISOString()}

\n\n

Please check the logs for more details and consider investigating the issue.

You may want to check your OpenAI API key and quota limits.

\n `;\n\n const text = `\nSummarization Job Failed\n\nError: ${error instanceof Error ? error.message : String(error) || error}\nExecution time: ${(executionTime / 1000).toFixed(2)}s\nTimestamp: ${new Date().toISOString()}\n\nPlease check the logs for more details and consider investigating the issue.\n `;\n\n await email({\n to: recipientEmail,\n subject,\n html,\n text,\n });\n}",[],[176,439],"844d8262-8a7c-11f0-81bc-0224a6c84d84","twitter-summarization-api.http.ts","844d830c-8a7c-11f0-81bc-0224a6c84d84",52,["D",1757092784854],"/**\n * HTTP endpoint for testing and manually triggering Twitter summarization\n * This provides a way to test the summarization functionality and manually generate summaries\n */\n\nimport { TwitterSummarizationService } from \"./twitter-summarization.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\nimport { getAllTopics, getSummarizationsForTopic } from \"./database/queries.ts\";\n\n/**\n * HTTP endpoint for tweet summarization operations\n */\nexport default async function (req: Request): Promise {\n // Set CORS headers for all responses\n const headers = new Headers({\n \"Access-Control-Allow-Origin\": \"*\",\n \"Access-Control-Allow-Methods\": \"GET, POST, OPTIONS\",\n \"Access-Control-Allow-Headers\": \"Content-Type\",\n \"Content-Type\": \"application/json\",\n });\n\n // Handle preflight OPTIONS request\n if (req.method === \"OPTIONS\") {\n return new Response(null, { headers, status: 204 });\n }\n\n try {\n const url = new URL(req.url);\n const action = url.searchParams.get(\"action\") || \"status\";\n\n // Initialize database\n await initializeDatabase();\n\n let responseData: any = {\n status: \"ok\",\n timestamp: new Date().toISOString(),\n };\n\n const summarizationService = new TwitterSummarizationService();\n\n if (action === \"test\") {\n // Test the summarization service with sample data\n console.log(\"🧪 Testing summarization service...\");\n \n const topics = await getAllTopics();\n \n if (topics.length === 0) {\n responseData = {\n ...responseData,\n action: \"test\",\n status: \"warning\",\n message: \"No topics found. Please create some topics first.\",\n topics: [],\n };\n } else {\n // Test with the first topic only (to avoid hitting rate limits during testing)\n const testTopic = topics[0];\n console.log(`Testing with topic: ${testTopic.title} (ID: ${testTopic.id})`);\n\n const startTime = Date.now();\n const summary = await summarizationService.generateTopicSummary(testTopic.id!, {\n timeWindowHours: 48, // Look back further for testing\n maxTweets: 20, // Fewer tweets for testing\n includeMedia: true,\n includeSentiment: true,\n });\n\n const executionTime = Date.now() - startTime;\n\n if (!summary) {\n responseData = {\n ...responseData,\n action: \"test\",\n status: \"no_data\",\n message: `No recent posts found for topic: ${testTopic.title}`,\n topicTested: testTopic,\n availableTopics: topics.map(t => ({ id: t.id, title: t.title })),\n executionTimeMs: executionTime,\n };\n } else {\n responseData = {\n ...responseData,\n action: \"test\",\n status: \"success\",\n message: \"Test summarization completed successfully\",\n testSummary: summary,\n availableTopics: topics.map(t => ({ id: t.id, title: t.title })),\n executionTimeMs: executionTime,\n };\n }\n }\n\n } else if (action === \"generate\") {\n // Generate summaries for all topics or specific topic\n const topicId = url.searchParams.get(\"topicId\");\n const timeWindowHours = parseInt(url.searchParams.get(\"timeWindowHours\") || \"24\");\n const maxTweets = parseInt(url.searchParams.get(\"maxTweets\") || \"50\");\n const includeMedia = url.searchParams.get(\"includeMedia\") !== \"false\";\n const includeSentiment = url.searchParams.get(\"includeSentiment\") !== \"false\";\n\n const options = {\n timeWindowHours,\n maxTweets,\n includeMedia,\n includeSentiment,\n };\n\n const startTime = Date.now();\n\n if (topicId) {\n // Generate summary for specific topic\n const topicIdNum = parseInt(topicId);\n if (isNaN(topicIdNum)) {\n return new Response(JSON.stringify({\n status: \"error\",\n message: \"Invalid topic ID\",\n }), { headers, status: 400 });\n }\n\n console.log(`🤖 Generating summary for topic ${topicIdNum} with options:`, options);\n\n const summary = await summarizationService.generateTopicSummary(topicIdNum, options);\n \n if (!summary) {\n responseData = {\n ...responseData,\n action: \"generate\",\n status: \"no_data\",\n message: `No recent posts found for topic ${topicIdNum}`,\n topicId: topicIdNum,\n };\n } else {\n // Store the summary in the database\n const summaryId = await summarizationService.storeTopicSummary(summary);\n const executionTime = Date.now() - startTime;\n\n responseData = {\n ...responseData,\n action: \"generate\",\n status: \"success\",\n message: \"Summary generated and stored successfully\",\n summary,\n summaryId,\n executionTimeMs: executionTime,\n };\n }\n } else {\n // Generate summaries for all topics\n console.log(\"🤖 Generating summaries for all topics with options:\", options);\n\n const result = await summarizationService.generateAndStoreSummaries(options);\n const executionTime = Date.now() - startTime;\n\n responseData = {\n ...responseData,\n action: \"generate\",\n status: result.success ? \"success\" : \"partial_success\",\n message: `Generated ${result.generated} summaries, stored ${result.stored}`,\n results: {\n generated: result.generated,\n stored: result.stored,\n errors: result.errors.length,\n },\n summaries: result.summaries.map(s => ({\n topicId: s.topicId,\n topicTitle: s.topicTitle,\n tweetCount: s.tweetCount,\n sentiment: s.overallSentiment.sentiment,\n keyPointsCount: s.keyPoints.length,\n })),\n errors: result.errors,\n executionTimeMs: executionTime,\n };\n }\n\n } else if (action === \"summaries\") {\n // Get stored summaries for a topic\n const topicId = url.searchParams.get(\"topicId\");\n if (!topicId) {\n return new Response(JSON.stringify({\n status: \"error\",\n message: \"topicId parameter is required for summaries action\",\n }), { headers, status: 400 });\n }\n\n const topicIdNum = parseInt(topicId);\n if (isNaN(topicIdNum)) {\n return new Response(JSON.stringify({\n status: \"error\",\n message: \"Invalid topic ID\",\n }), { headers, status: 400 });\n }\n\n const limit = parseInt(url.searchParams.get(\"limit\") || \"10\");\n const summaries = await getSummarizationsForTopic(topicIdNum, limit);\n\n responseData = {\n ...responseData,\n action: \"summaries\",\n topicId: topicIdNum,\n summaries: summaries.map(s => ({\n id: s.id,\n content: JSON.parse(s.content),\n created_at: s.created_at,\n time_period: `${s.time_period_start} to ${s.time_period_end}`,\n })),\n count: summaries.length,\n };\n\n } else if (action === \"topics\") {\n // List all available topics\n const topics = await getAllTopics();\n \n responseData = {\n ...responseData,\n action: \"topics\",\n topics: topics.map(t => ({\n id: t.id,\n title: t.title,\n description: t.description,\n })),\n count: topics.length,\n };\n\n } else {\n // Default status action\n responseData = {\n ...responseData,\n action: \"status\",\n message: \"Summarization API is running\",\n availableActions: [\n \"test - Test summarization with first available topic\",\n \"generate - Generate summaries (add ?topicId=X for specific topic)\",\n \"summaries - Get stored summaries (requires ?topicId=X)\",\n \"topics - List all available topics\",\n ],\n config: {\n timeWindowHours: parseInt(Deno.env.get(\"TWITTER_SUMMARIZATION_TIME_WINDOW\") || \"24\"),\n maxTweets: parseInt(Deno.env.get(\"TWITTER_SUMMARIZATION_MAX_TWEETS\") || \"50\"),\n includeMedia: Deno.env.get(\"TWITTER_SUMMARIZATION_INCLUDE_MEDIA\") !== \"false\",\n includeSentiment: Deno.env.get(\"TWITTER_SUMMARIZATION_INCLUDE_SENTIMENT\") !== \"false\",\n },\n };\n }\n\n return new Response(JSON.stringify(responseData, null, 2), { headers });\n\n } catch (error: unknown) {\n console.error(\"Error processing summarization request:\", error);\n\n return new Response(\n JSON.stringify({\n status: \"error\",\n message: error instanceof Error ? error.message : String(error),\n timestamp: new Date().toISOString(),\n }, null, 2),\n { headers, status: 500 },\n );\n }\n}",[],[176,448],"848da7f2-8a7c-11f0-abb7-0224a6c84d84","twitter-sentiment.ts","848da89c-8a7c-11f0-abb7-0224a6c84d84",61,["D",1757092785079],"import { OpenAI } from \"https://esm.town/v/std/openai\";\nimport { Post } from \"./database/queries.ts\";\n\n/**\n * Sentiment analysis result interface\n */\nexport interface SentimentResult {\n post_id: number;\n sentiment: \"positive\" | \"negative\" | \"neutral\";\n score: number; // -1.0 to 1.0 where -1 is very negative, 0 is neutral, 1 is very positive\n confidence: number; // 0.0 to 1.0\n keywords: string[]; // Key emotional or opinion words that influenced the sentiment\n entities?: string[]; // Optional entities mentioned in the text\n}\n\n/**\n * Configuration options for sentiment analysis\n */\nexport interface SentimentAnalysisConfig {\n useAI: boolean; // Whether to use AI for sentiment analysis or rule-based approach\n confidenceThreshold: number; // Minimum confidence score to accept a sentiment result\n batchSize: number; // Number of tweets to analyze in a single batch\n includeEntities: boolean; // Whether to extract entities mentioned in the text\n requestsPerMinute?: number; // Maximum number of API requests per minute (rate limiting)\n}\n\n/**\n * Default sentiment analysis configuration\n */\nexport const DEFAULT_SENTIMENT_CONFIG: SentimentAnalysisConfig = {\n useAI: true,\n confidenceThreshold: 0.7,\n batchSize: 10,\n includeEntities: false,\n requestsPerMinute: 10, // Default to 10 requests per minute (Val Town limit)\n};\n\n/**\n * Twitter sentiment analysis class\n */\nexport class TwitterSentimentAnalyzer {\n private config: SentimentAnalysisConfig;\n private openai: OpenAI | null = null;\n private requestTimestamps: number[] = []; // Store timestamps of recent API requests\n\n /**\n * Create a new TwitterSentimentAnalyzer with the specified configuration\n * @param config Sentiment analysis configuration (optional, defaults to DEFAULT_SENTIMENT_CONFIG)\n */\n constructor(config: SentimentAnalysisConfig = DEFAULT_SENTIMENT_CONFIG) {\n this.config = {\n ...DEFAULT_SENTIMENT_CONFIG,\n ...config,\n };\n\n if (this.config.useAI) {\n this.initializeOpenAI();\n }\n }\n\n /**\n * Initialize OpenAI client\n */\n private initializeOpenAI(): void {\n try {\n this.openai = new OpenAI();\n } catch (error: unknown) {\n console.error(\"Failed to initialize OpenAI client:\", error);\n const errorMessage = error instanceof Error ? error instanceof Error ? error.message : String(error) : String(error);\n throw new Error(\n `OpenAI initialization failed: ${errorMessage}. Make sure OPENAI_API_KEY is set.`,\n );\n }\n }\n\n /**\n * Check if we can make another API request based on rate limiting\n * @returns boolean indicating if we can make another request\n */\n private canMakeRequest(): boolean {\n const now = Date.now();\n const oneMinuteAgo = now - 60000; // 1 minute in milliseconds\n\n // Remove timestamps older than 1 minute\n this.requestTimestamps = this.requestTimestamps.filter(\n (timestamp) => timestamp > oneMinuteAgo,\n );\n\n // Check if we've made fewer requests than our limit\n return this.requestTimestamps.length <\n (this.config.requestsPerMinute || 10);\n }\n\n /**\n * Wait until we can make another API request\n * @returns Promise that resolves when we can make another request\n */\n private async waitForRequestSlot(): Promise {\n if (this.canMakeRequest()) {\n return;\n }\n\n // Calculate time to wait based on the oldest request in our window\n const now = Date.now();\n const oldestTimestamp = this.requestTimestamps[0];\n const timeToWait = 60000 - (now - oldestTimestamp) + 50; // Add 50ms buffer\n\n console.log(\n `Rate limit reached. Waiting ${timeToWait}ms before next request.`,\n );\n\n return new Promise((resolve) => setTimeout(resolve, timeToWait));\n }\n\n /**\n * Record a new API request\n */\n private recordRequest(): void {\n this.requestTimestamps.push(Date.now());\n }\n\n /**\n * Analyze sentiment for a batch of tweets\n * @param tweets Array of tweets to analyze\n * @returns Array of sentiment analysis results\n */\n async analyzeBatch(tweets: Post[]): Promise {\n if (!tweets || tweets.length === 0) {\n return [];\n }\n\n const results: SentimentResult[] = [];\n\n // Process in batches according to configuration\n for (let i = 0; i < tweets.length; i += this.config.batchSize) {\n const batch = tweets.slice(i, i + this.config.batchSize);\n\n if (this.config.useAI) {\n const batchResults = await this.analyzeWithAI(batch);\n results.push(...batchResults);\n } else {\n const batchResults = this.analyzeWithRules(batch);\n results.push(...batchResults);\n }\n }\n\n return results;\n }\n\n /**\n * Analyze a single tweet's sentiment\n * @param tweet The tweet to analyze\n * @returns Sentiment analysis result\n */\n async analyzeTweet(tweet: Post): Promise {\n if (this.config.useAI) {\n const results = await this.analyzeWithAI([tweet]);\n return results[0];\n } else {\n return this.analyzeWithRules([tweet])[0];\n }\n }\n\n /**\n * Analyze tweets using OpenAI\n * @param tweets Array of tweets to analyze\n * @returns Array of sentiment analysis results\n */\n private async analyzeWithAI(tweets: Post[]): Promise {\n if (!this.openai) {\n throw new Error(\"OpenAI client not initialized\");\n }\n\n const results: SentimentResult[] = [];\n\n try {\n // Wait for a request slot to become available (throttling)\n await this.waitForRequestSlot();\n\n // Prepare the prompt for batch analysis\n const prompt = this.buildSentimentPrompt(tweets);\n\n // Record this request for rate limiting\n this.recordRequest();\n\n // Call OpenAI API\n const completion = await this.openai.chat.completions.create({\n messages: [\n {\n role: \"system\",\n content:\n \"You are a sentiment analysis expert. Analyze the sentiment of the provided tweets and return the results in the exact JSON format requested.\",\n },\n { role: \"user\", content: prompt },\n ],\n model: \"gpt-3.5-turbo\", // Using a smaller model for cost efficiency\n response_format: { type: \"json_object\" },\n max_tokens: 1000,\n });\n\n // Parse the response\n const responseContent = completion.choices[0]?.message?.content || \"\";\n if (!responseContent) {\n throw new Error(\"Empty response from OpenAI\");\n }\n\n try {\n const parsedResponse = JSON.parse(responseContent);\n\n if (!parsedResponse.results || !Array.isArray(parsedResponse.results)) {\n throw new Error(\"Invalid response format from OpenAI\");\n }\n\n // Map the AI results to our SentimentResult format\n for (let i = 0; i < parsedResponse.results.length; i++) {\n const result = parsedResponse.results[i];\n const tweet = tweets[i];\n\n if (!tweet || !tweet.id) {\n console.warn(\"Missing tweet ID for sentiment result\");\n continue;\n }\n\n results.push({\n post_id: tweet.id,\n sentiment: result.sentiment || \"neutral\",\n score: result.score || 0,\n confidence: result.confidence || 0.5,\n keywords: result.keywords || [],\n entities: this.config.includeEntities\n ? result.entities || []\n : undefined,\n });\n }\n } catch (parseError) {\n console.error(\"Failed to parse OpenAI response:\", parseError);\n const parseErrorMessage = parseError instanceof Error ? parseError.message : String(parseError);\n throw new Error(\n `Failed to parse OpenAI response: ${parseErrorMessage}`,\n );\n }\n } catch (error: unknown) {\n console.error(\"Error in AI sentiment analysis:\", error);\n\n // Fallback to rule-based analysis\n console.log(\"Falling back to rule-based sentiment analysis\");\n return this.analyzeWithRules(tweets);\n }\n\n return results;\n }\n\n /**\n * Build the prompt for sentiment analysis\n * @param tweets Array of tweets to analyze\n * @returns Formatted prompt for OpenAI\n */\n private buildSentimentPrompt(tweets: Post[]): string {\n const tweetTexts = tweets.map((tweet, index) =>\n `Tweet ${index + 1}: \"${tweet.content}\"`\n ).join(\"\\n\\n\");\n\n const includeEntities = this.config.includeEntities\n ? `Include an \"entities\" array with named entities mentioned in the tweet.`\n : \"\";\n\n return `\nAnalyze the sentiment of the following tweets:\n\n${tweetTexts}\n\nFor each tweet, determine:\n1. The overall sentiment (positive, negative, or neutral)\n2. A sentiment score from -1.0 (very negative) to 1.0 (very positive)\n3. A confidence score from 0.0 to 1.0 indicating how confident you are in this assessment\n4. Key emotional or opinion words that influenced the sentiment\n${includeEntities}\n\nReturn your analysis in the following JSON format:\n{\n \"results\": [\n {\n \"sentiment\": \"positive|negative|neutral\",\n \"score\": float,\n \"confidence\": float,\n \"keywords\": [\"word1\", \"word2\", ...],\n ${\n this.config.includeEntities\n ? `\"entities\": [\"entity1\", \"entity2\", ...],`\n : \"\"\n }\n },\n ...\n ]\n}\n\nOnly return valid JSON, no other text.\n`;\n }\n\n /**\n * Simple rule-based sentiment analysis as a fallback\n * @param tweets Array of tweets to analyze\n * @returns Array of sentiment analysis results\n */\n private analyzeWithRules(tweets: Post[]): SentimentResult[] {\n const positiveWords = [\n \"good\",\n \"great\",\n \"excellent\",\n \"amazing\",\n \"awesome\",\n \"wonderful\",\n \"fantastic\",\n \"happy\",\n \"love\",\n \"best\",\n \"beautiful\",\n \"perfect\",\n \"brilliant\",\n \"exciting\",\n \"joy\",\n ];\n\n const negativeWords = [\n \"bad\",\n \"terrible\",\n \"awful\",\n \"horrible\",\n \"poor\",\n \"worst\",\n \"hate\",\n \"sad\",\n \"disappointing\",\n \"disaster\",\n \"failed\",\n \"broken\",\n \"useless\",\n \"angry\",\n \"annoying\",\n ];\n\n return tweets.map((tweet) => {\n if (!tweet.id) {\n console.warn(\"Missing tweet ID for rule-based sentiment analysis\");\n return {\n post_id: 0,\n sentiment: \"neutral\" as const,\n score: 0,\n confidence: 0.5,\n keywords: [],\n };\n }\n\n const content = tweet.content.toLowerCase();\n const words = content.split(/\\s+/);\n\n let positiveCount = 0;\n let negativeCount = 0;\n const foundKeywords: string[] = [];\n\n for (const word of words) {\n const cleanWord = word.replace(/[^\\w]/g, \"\");\n\n if (positiveWords.includes(cleanWord)) {\n positiveCount++;\n foundKeywords.push(cleanWord);\n }\n\n if (negativeWords.includes(cleanWord)) {\n negativeCount++;\n foundKeywords.push(cleanWord);\n }\n }\n\n // Calculate sentiment score (-1.0 to 1.0)\n const totalSentimentWords = positiveCount + negativeCount;\n let score = 0;\n let confidence = 0.5; // Default moderate confidence\n\n if (totalSentimentWords > 0) {\n score = (positiveCount - negativeCount) / totalSentimentWords;\n confidence = Math.min(\n 0.7,\n 0.5 + (totalSentimentWords / words.length) * 0.2,\n );\n }\n\n // Determine sentiment category\n let sentiment: \"positive\" | \"negative\" | \"neutral\" = \"neutral\";\n if (score > 0.2) {\n sentiment = \"positive\";\n } else if (score < -0.2) {\n sentiment = \"negative\";\n }\n\n return {\n post_id: tweet.id,\n sentiment,\n score,\n confidence,\n keywords: foundKeywords,\n };\n });\n }\n\n /**\n * Load sentiment analysis configuration from environment variables\n * @returns SentimentAnalysisConfig object based on environment variables\n */\n static loadConfigFromEnv(): SentimentAnalysisConfig {\n const config: SentimentAnalysisConfig = { ...DEFAULT_SENTIMENT_CONFIG };\n\n // Use AI for sentiment analysis\n const useAIEnv = Deno.env.get(\"TWITTER_SENTIMENT_USE_AI\");\n if (useAIEnv) {\n config.useAI = useAIEnv.toLowerCase() === \"true\";\n }\n\n // Confidence threshold\n const confidenceThresholdEnv = Deno.env.get(\n \"TWITTER_SENTIMENT_CONFIDENCE_THRESHOLD\",\n );\n if (confidenceThresholdEnv) {\n const threshold = parseFloat(confidenceThresholdEnv);\n if (!isNaN(threshold) && threshold >= 0 && threshold <= 1) {\n config.confidenceThreshold = threshold;\n }\n }\n\n // Batch size\n const batchSizeEnv = Deno.env.get(\"TWITTER_SENTIMENT_BATCH_SIZE\");\n if (batchSizeEnv) {\n const batchSize = parseInt(batchSizeEnv, 10);\n if (!isNaN(batchSize) && batchSize > 0) {\n config.batchSize = batchSize;\n }\n }\n\n // Include entities\n const includeEntitiesEnv = Deno.env.get(\n \"TWITTER_SENTIMENT_INCLUDE_ENTITIES\",\n );\n if (includeEntitiesEnv) {\n config.includeEntities = includeEntitiesEnv.toLowerCase() === \"true\";\n }\n\n // Requests per minute (rate limiting)\n const requestsPerMinuteEnv = Deno.env.get(\n \"TWITTER_SENTIMENT_REQUESTS_PER_MINUTE\",\n );\n if (requestsPerMinuteEnv) {\n const rpm = parseInt(requestsPerMinuteEnv, 10);\n if (!isNaN(rpm) && rpm > 0) {\n config.requestsPerMinute = rpm;\n }\n }\n\n return config;\n }\n}\n",[],[176,456],"843f5bba-8a7c-11f0-b5e7-0224a6c84d84","twitter-sentiment-job.cron.ts","843f5c6e-8a7c-11f0-b5e7-0224a6c84d84",50,["D",1757092784738],"import { initializeDatabase } from \"./database/migrations.ts\";\nimport {\n createOrUpdatePostSentiment,\n getPostsWithoutSentimentAnalysis,\n} from \"./database/queries.ts\";\nimport { TwitterSentimentAnalyzer } from \"./twitter-sentiment.ts\";\nimport { email } from \"https://esm.town/v/std/email\";\n\n/**\n * Cron job that runs on a schedule to analyze tweets for sentiment\n * Default schedule: Every 4 hours\n */\nexport default async function () {\n console.log(\"Starting scheduled sentiment analysis job\");\n\n try {\n // Initialize database\n await initializeDatabase();\n\n // Create sentiment analyzer with configuration from environment variables\n const analyzer = new TwitterSentimentAnalyzer(\n TwitterSentimentAnalyzer.loadConfigFromEnv(),\n );\n\n // Get batch size from environment or use default\n const batchSize = parseInt(\n Deno.env.get(\"TWITTER_SENTIMENT_JOB_BATCH_SIZE\") || \"50\",\n 10,\n );\n\n // Get unanalyzed posts\n const posts = await getPostsWithoutSentimentAnalysis(batchSize);\n\n if (posts.length === 0) {\n console.log(\"No unanalyzed posts found\");\n return;\n }\n\n console.log(`Analyzing sentiment for ${posts.length} posts`);\n\n // Analyze sentiment\n const sentimentResults = await analyzer.analyzeBatch(posts);\n\n // Store results in database\n let storedCount = 0;\n for (const result of sentimentResults) {\n await createOrUpdatePostSentiment(result);\n storedCount++;\n }\n\n console.log(`Completed sentiment analysis: ${storedCount} results stored`);\n\n // Send notification email if enabled\n const sendNotification =\n Deno.env.get(\"TWITTER_SENTIMENT_JOB_NOTIFY\") === \"true\";\n if (sendNotification && storedCount > 0) {\n const sentimentCounts = {\n positive: sentimentResults.filter((r) =>\n r.sentiment === \"positive\"\n ).length,\n negative: sentimentResults.filter((r) =>\n r.sentiment === \"negative\"\n ).length,\n neutral: sentimentResults.filter((r) =>\n r.sentiment === \"neutral\"\n ).length,\n };\n\n await email({\n subject: `OffX: Sentiment Analysis Complete (${storedCount} posts)`,\n text: `\nSentiment analysis job completed successfully.\n\nPosts analyzed: ${storedCount}\n- Positive: ${sentimentCounts.positive}\n- Negative: ${sentimentCounts.negative}\n- Neutral: ${sentimentCounts.neutral}\n\nJob completed at: ${new Date().toISOString()}\n `,\n html: `\n

Sentiment Analysis Job Complete

The scheduled sentiment analysis job has completed successfully.

\n\n

Summary

Posts analyzed: ${storedCount}
Positive: ${sentimentCounts.positive}
Negative: ${sentimentCounts.negative}
Neutral: ${sentimentCounts.neutral}

\n\n

Job completed at: ${new Date().toISOString()}

\n `,\n });\n }\n } catch (error: unknown) {\n console.error(\"Error in sentiment analysis job:\", error);\n\n const errorMessage = error instanceof Error ? error instanceof Error ? error.message : String(error) : String(error);\n\n // Send error notification\n try {\n await email({\n subject: \"OffX: Sentiment Analysis Job Failed\",\n text: `\nSentiment analysis job failed with error:\n${errorMessage}\n\nError occurred at: ${new Date().toISOString()}\n `,\n html: `\n

Sentiment Analysis Job Failed

The scheduled sentiment analysis job encountered an error.

\n\n

Error Details

${errorMessage}

\n\n

Error occurred at: ${new Date().toISOString()}

\n `,\n });\n } catch (emailError) {\n console.error(\"Failed to send error notification email:\", emailError);\n }\n\n throw error; // Re-throw to ensure the job is marked as failed\n }\n}\n",[],[176,464],"8482f276-8a7c-11f0-b7f7-0224a6c84d84","twitter-sentiment-api.http.ts","8482f33e-8a7c-11f0-b7f7-0224a6c84d84",59,["D",1757092785218],"import { initializeDatabase } from \"./database/migrations.ts\";\nimport {\n createOrUpdatePostSentiment,\n getPostsWithoutSentimentAnalysis,\n getPostsWithSentiment,\n getRecentPosts,\n getSentimentStats,\n Post,\n} from \"./database/queries.ts\";\nimport { TwitterSentimentAnalyzer } from \"./twitter-sentiment.ts\";\n\n/**\n * HTTP endpoint for tweet sentiment analysis operations\n */\nexport default async function (req: Request): Promise {\n // Set CORS headers for all responses\n const headers = new Headers({\n \"Access-Control-Allow-Origin\": \"*\",\n \"Access-Control-Allow-Methods\": \"GET, POST, OPTIONS\",\n \"Access-Control-Allow-Headers\": \"Content-Type\",\n \"Content-Type\": \"application/json\",\n });\n\n // Handle preflight OPTIONS request\n if (req.method === \"OPTIONS\") {\n return new Response(null, { headers, status: 204 });\n }\n\n // Process the request\n try {\n const url = new URL(req.url);\n const action = url.searchParams.get(\"action\") || \"status\";\n\n // Initialize database\n await initializeDatabase();\n\n let responseData: any = {\n status: \"ok\",\n timestamp: new Date().toISOString(),\n };\n\n // Create sentiment analyzer with configuration from environment variables\n const analyzer = new TwitterSentimentAnalyzer(\n TwitterSentimentAnalyzer.loadConfigFromEnv(),\n );\n\n if (action === \"analyze\") {\n // Get posts to analyze - either unanalyzed posts or recent posts\n const useUnanalyzed = url.searchParams.get(\"unanalyzed\") === \"true\";\n const limit = parseInt(url.searchParams.get(\"limit\") || \"10\", 10);\n\n const posts = useUnanalyzed\n ? await getPostsWithoutSentimentAnalysis(limit)\n : await getRecentPosts(limit);\n\n if (posts.length === 0) {\n responseData = {\n ...responseData,\n action: \"analyze\",\n message: \"No posts found for analysis\",\n analyzedCount: 0,\n };\n } else {\n // Analyze sentiment\n const sentimentResults = await analyzer.analyzeBatch(posts);\n\n // Store results in database\n let storedCount = 0;\n for (const result of sentimentResults) {\n await createOrUpdatePostSentiment(result);\n storedCount++;\n }\n\n responseData = {\n ...responseData,\n action: \"analyze\",\n analyzedCount: sentimentResults.length,\n storedCount,\n results: sentimentResults.slice(0, 3), // Return first 3 results as preview\n };\n }\n } else if (action === \"query\") {\n // Query posts by sentiment\n const sentiment = url.searchParams.get(\"sentiment\") as\n | \"positive\"\n | \"negative\"\n | \"neutral\"\n | undefined;\n const minConfidence = parseFloat(\n url.searchParams.get(\"min_confidence\") || \"0.7\",\n );\n const limit = parseInt(url.searchParams.get(\"limit\") || \"10\", 10);\n\n const posts = await getPostsWithSentiment(\n sentiment,\n minConfidence,\n limit,\n );\n\n responseData = {\n ...responseData,\n action: \"query\",\n sentiment: sentiment || \"all\",\n minConfidence,\n count: posts.length,\n posts: posts.map((post) => ({\n id: post.id,\n content: post.content,\n author: post.author_username,\n created_at: post.created_at,\n })),\n };\n } else if (action === \"stats\") {\n // Get sentiment statistics\n const stats = await getSentimentStats();\n\n responseData = {\n ...responseData,\n action: \"stats\",\n stats,\n };\n } else {\n // Default status action\n responseData = {\n ...responseData,\n action: \"status\",\n message: \"Sentiment analysis API is running\",\n availableActions: [\"analyze\", \"query\", \"stats\"],\n config: {\n useAI: Deno.env.get(\"TWITTER_SENTIMENT_USE_AI\") === \"true\",\n batchSize: parseInt(\n Deno.env.get(\"TWITTER_SENTIMENT_BATCH_SIZE\") || \"10\",\n 10,\n ),\n },\n };\n }\n\n return new Response(JSON.stringify(responseData, null, 2), { headers });\n } catch (error: unknown) {\n console.error(\"Error processing sentiment analysis request:\", error);\n\n return new Response(\n JSON.stringify(\n {\n status: \"error\",\n message: error instanceof Error ? error.message : String(error),\n timestamp: new Date().toISOString(),\n },\n null,\n 2,\n ),\n { headers, status: 500 },\n );\n }\n}\n",[],[176,472],"844a4570-8a7c-11f0-a4f1-0224a6c84d84","twitter-processor.ts","844a4642-8a7c-11f0-a4f1-0224a6c84d84",51,["D",1757092784812],"import { Post, PostMedia } from \"./database/queries.ts\";\n\n/**\n * Interface for filter configuration\n */\nexport interface TwitterFilterConfig {\n keywords?: string[];\n languages?: string[];\n excludeKeywords?: string[];\n excludeUsers?: string[];\n includeUsers?: string[];\n minFavoriteCount?: number;\n minRetweetCount?: number;\n includeRetweets?: boolean;\n includeReplies?: boolean;\n}\n\n/**\n * Default filter configuration\n */\nexport const DEFAULT_FILTER_CONFIG: TwitterFilterConfig = {\n languages: [\"en\"],\n includeRetweets: false,\n includeReplies: false,\n};\n\n/**\n * Twitter data processing pipeline\n */\nexport class TwitterProcessor {\n private filterConfig: TwitterFilterConfig;\n\n /**\n * Create a new TwitterProcessor with the specified filter configuration\n * @param config Filter configuration (optional, defaults to DEFAULT_FILTER_CONFIG)\n */\n constructor(config: TwitterFilterConfig = DEFAULT_FILTER_CONFIG) {\n this.filterConfig = {\n ...DEFAULT_FILTER_CONFIG,\n ...config,\n };\n }\n\n /**\n * Process a batch of tweets through the pipeline\n * @param tweets Array of tweets to process\n * @returns Filtered and processed tweets\n */\n processTweets(tweets: Post[]): Post[] {\n if (!tweets || !Array.isArray(tweets) || tweets.length === 0) {\n return [];\n }\n\n return tweets\n .filter((tweet) => this.applyFilters(tweet))\n .map((tweet) => this.transformTweet(tweet));\n }\n\n /**\n * Apply all configured filters to a tweet\n * @param tweet The tweet to filter\n * @returns True if the tweet passes all filters, false otherwise\n */\n private applyFilters(tweet: Post): boolean {\n // Filter by language\n if (this.filterConfig.languages && this.filterConfig.languages.length > 0) {\n // Note: Twitter API doesn't always provide language info in the response\n // This would need to be extended if language detection is critical\n // For now, we'll assume English if not specified\n const tweetLanguage = \"en\";\n if (!this.filterConfig.languages.includes(tweetLanguage)) {\n return false;\n }\n }\n\n // Filter retweets if configured\n if (\n !this.filterConfig.includeRetweets && tweet.content.startsWith(\"RT @\")\n ) {\n return false;\n }\n\n // Filter replies if configured\n if (!this.filterConfig.includeReplies && tweet.content.startsWith(\"@\")) {\n return false;\n }\n\n // Filter by keywords\n if (this.filterConfig.keywords && this.filterConfig.keywords.length > 0) {\n const tweetText = tweet.content.toLowerCase();\n const hasKeyword = this.filterConfig.keywords.some((keyword) =>\n tweetText.includes(keyword.toLowerCase())\n );\n\n if (!hasKeyword) {\n return false;\n }\n }\n\n // Filter by excluded keywords\n if (\n this.filterConfig.excludeKeywords &&\n this.filterConfig.excludeKeywords.length > 0\n ) {\n const tweetText = tweet.content.toLowerCase();\n const hasExcludedKeyword = this.filterConfig.excludeKeywords.some(\n (keyword) => tweetText.includes(keyword.toLowerCase()),\n );\n\n if (hasExcludedKeyword) {\n return false;\n }\n }\n\n // Filter by included users\n if (\n this.filterConfig.includeUsers &&\n this.filterConfig.includeUsers.length > 0\n ) {\n const username = tweet.author_username.toLowerCase();\n const isIncludedUser = this.filterConfig.includeUsers.some((user) =>\n username === user.toLowerCase()\n );\n\n if (!isIncludedUser) {\n return false;\n }\n }\n\n // Filter by excluded users\n if (\n this.filterConfig.excludeUsers &&\n this.filterConfig.excludeUsers.length > 0\n ) {\n const username = tweet.author_username.toLowerCase();\n const isExcludedUser = this.filterConfig.excludeUsers.some((user) =>\n username === user.toLowerCase()\n );\n\n if (isExcludedUser) {\n return false;\n }\n }\n\n return true;\n }\n\n /**\n * Transform a tweet for storage and processing\n * @param tweet The tweet to transform\n * @returns Transformed tweet\n */\n private transformTweet(tweet: Post): Post {\n // Create a copy of the tweet to avoid modifying the original\n const transformedTweet: Post = { ...tweet };\n\n // Clean up the content (e.g., remove excessive whitespace, normalize URLs)\n transformedTweet.content = this.cleanContent(transformedTweet.content);\n\n // Transform media if present\n if (transformedTweet.media && transformedTweet.media.length > 0) {\n transformedTweet.media = transformedTweet.media.map((media) =>\n this.transformMedia(media)\n );\n }\n\n return transformedTweet;\n }\n\n /**\n * Clean up tweet content\n * @param content The raw tweet content\n * @returns Cleaned content\n */\n private cleanContent(content: string): string {\n // Remove excessive whitespace\n let cleaned = content.replace(/\\s+/g, \" \").trim();\n\n // Normalize URLs (optional)\n // This is a simple example - could be expanded for more sophisticated URL normalization\n cleaned = cleaned.replace(/https:\\/\\/t\\.co\\/\\w+/g, (url) => url.trim());\n\n return cleaned;\n }\n\n /**\n * Transform media attachments\n * @param media The media to transform\n * @returns Transformed media\n */\n private transformMedia(media: PostMedia): PostMedia {\n // Create a copy to avoid modifying the original\n const transformedMedia: PostMedia = { ...media };\n\n // Ensure media URLs are properly formatted\n if (\n transformedMedia.media_url &&\n !transformedMedia.media_url.startsWith(\"http\")\n ) {\n transformedMedia.media_url = `https://${transformedMedia.media_url}`;\n }\n\n return transformedMedia;\n }\n\n /**\n * Load filter configuration from environment variables\n * @returns TwitterFilterConfig object based on environment variables\n */\n static loadConfigFromEnv(): TwitterFilterConfig {\n const config: TwitterFilterConfig = { ...DEFAULT_FILTER_CONFIG };\n\n // Keywords filter\n const keywordsEnv = Deno.env.get(\"TWITTER_FILTER_KEYWORDS\");\n if (keywordsEnv) {\n config.keywords = keywordsEnv.split(\",\").map((k) => k.trim());\n }\n\n // Excluded keywords\n const excludeKeywordsEnv = Deno.env.get(\"TWITTER_FILTER_EXCLUDE_KEYWORDS\");\n if (excludeKeywordsEnv) {\n config.excludeKeywords = excludeKeywordsEnv.split(\",\").map((k) =>\n k.trim()\n );\n }\n\n // Included users\n const includeUsersEnv = Deno.env.get(\"TWITTER_FILTER_INCLUDE_USERS\");\n if (includeUsersEnv) {\n config.includeUsers = includeUsersEnv.split(\",\").map((u) => u.trim());\n }\n\n // Excluded users\n const excludeUsersEnv = Deno.env.get(\"TWITTER_FILTER_EXCLUDE_USERS\");\n if (excludeUsersEnv) {\n config.excludeUsers = excludeUsersEnv.split(\",\").map((u) => u.trim());\n }\n\n // Languages\n const languagesEnv = Deno.env.get(\"TWITTER_FILTER_LANGUAGES\");\n if (languagesEnv) {\n config.languages = languagesEnv.split(\",\").map((l) => l.trim());\n }\n\n // Boolean flags\n const includeRetweetsEnv = Deno.env.get(\"TWITTER_FILTER_INCLUDE_RETWEETS\");\n if (includeRetweetsEnv) {\n config.includeRetweets = includeRetweetsEnv.toLowerCase() === \"true\";\n }\n\n const includeRepliesEnv = Deno.env.get(\"TWITTER_FILTER_INCLUDE_REPLIES\");\n if (includeRepliesEnv) {\n config.includeReplies = includeRepliesEnv.toLowerCase() === \"true\";\n }\n\n return config;\n }\n}\n",[],[176,480],"84540024-8a7c-11f0-931f-0224a6c84d84","twitter-media.ts","845400c4-8a7c-11f0-931f-0224a6c84d84",53,["D",1757092784793],"import {\n createPostMedia,\n getPostMediaByPostId,\n Post,\n PostMedia,\n} from \"./database/queries.ts\";\n\n/**\n * Media type enumeration\n */\nexport enum MediaType {\n PHOTO = \"photo\",\n VIDEO = \"video\",\n GIF = \"animated_gif\",\n UNKNOWN = \"unknown\",\n}\n\n/**\n * Extended media metadata interface\n */\nexport interface MediaMetadata {\n width?: number;\n height?: number;\n duration_ms?: number;\n preview_url?: string;\n alt_text?: string;\n variants?: MediaVariant[];\n}\n\n/**\n * Media variant interface (for videos with multiple formats/bitrates)\n */\nexport interface MediaVariant {\n content_type: string;\n url: string;\n bitrate?: number;\n}\n\n/**\n * Extended post media interface with metadata\n */\nexport interface ExtendedPostMedia extends PostMedia {\n metadata?: MediaMetadata;\n}\n\n/**\n * Twitter media extraction and processing class\n */\nexport class TwitterMediaExtractor {\n /**\n * Extract media from Twitter API response\n * @param timelineData Twitter API response data\n * @param tweetId ID of the tweet to extract media from\n * @returns Array of extracted media items\n */\n static extractMediaFromResponse(\n timelineData: any,\n tweetId: string,\n ): ExtendedPostMedia[] {\n const media: ExtendedPostMedia[] = [];\n\n // Find the tweet in the response\n const tweet = timelineData.data?.find((t: any) => t.id === tweetId);\n\n if (\n !tweet || !tweet.attachments?.media_keys || !timelineData.includes?.media\n ) {\n return media;\n }\n\n // Extract media items\n for (const mediaKey of tweet.attachments.media_keys) {\n const mediaItem = timelineData.includes.media.find(\n (m: any) => m.media_key === mediaKey,\n );\n\n if (mediaItem) {\n const mediaType = this.normalizeMediaType(mediaItem.type);\n\n // Extract metadata\n const metadata: MediaMetadata = {\n width: mediaItem.width,\n height: mediaItem.height,\n duration_ms: mediaItem.duration_ms,\n preview_url: mediaItem.preview_image_url,\n alt_text: mediaItem.alt_text,\n };\n\n // Extract variants for videos\n if (mediaItem.variants && Array.isArray(mediaItem.variants)) {\n metadata.variants = mediaItem.variants.map((v: any) => ({\n content_type: v.content_type,\n url: v.url,\n bitrate: v.bit_rate,\n }));\n }\n\n // Create media object\n const postMedia: ExtendedPostMedia = {\n post_id: 0, // This will be set after post creation\n media_url: this.getBestMediaUrl(mediaItem),\n media_type: mediaType,\n alt_text: mediaItem.alt_text,\n metadata,\n };\n\n media.push(postMedia);\n }\n }\n\n return media;\n }\n\n /**\n * Get the best media URL based on media type\n * @param mediaItem Twitter API media item\n * @returns Best URL for the media\n */\n private static getBestMediaUrl(mediaItem: any): string {\n // For photos, use the URL directly\n if (mediaItem.url) {\n return mediaItem.url;\n }\n\n // For videos and GIFs, try to get the best quality variant\n if (mediaItem.variants && Array.isArray(mediaItem.variants)) {\n // Sort variants by bitrate (highest first)\n const sortedVariants = [...mediaItem.variants].sort((a, b) =>\n (b.bit_rate || 0) - (a.bit_rate || 0)\n );\n\n // Get the highest quality variant\n if (sortedVariants.length > 0) {\n return sortedVariants[0].url;\n }\n }\n\n // Fallback to preview image\n if (mediaItem.preview_image_url) {\n return mediaItem.preview_image_url;\n }\n\n return \"\";\n }\n\n /**\n * Normalize media type from Twitter API\n * @param type Media type from Twitter API\n * @returns Normalized media type\n */\n private static normalizeMediaType(type: string): string {\n switch (type) {\n case \"photo\":\n return MediaType.PHOTO;\n case \"video\":\n return MediaType.VIDEO;\n case \"animated_gif\":\n return MediaType.GIF;\n default:\n return MediaType.UNKNOWN;\n }\n }\n\n /**\n * Store media items for a post\n * @param postId Database ID of the post\n * @param media Array of media items to store\n * @returns Number of media items stored\n */\n static async storePostMedia(\n postId: number,\n media: ExtendedPostMedia[],\n ): Promise {\n if (!postId || !media || media.length === 0) {\n return 0;\n }\n\n let storedCount = 0;\n\n // Check for existing media\n const existingMedia = await getPostMediaByPostId(postId);\n\n // If media already exists for this post, skip\n if (existingMedia && existingMedia.length > 0) {\n return 0;\n }\n\n // Store each media item\n for (const mediaItem of media) {\n // Set the post ID\n mediaItem.post_id = postId;\n\n // Store in database\n await createPostMedia(mediaItem);\n storedCount++;\n }\n\n return storedCount;\n }\n\n /**\n * Process and enhance media items\n * @param media Array of media items to process\n * @returns Processed media items\n */\n static processMedia(media: ExtendedPostMedia[]): ExtendedPostMedia[] {\n return media.map((item) => {\n // Create a copy to avoid modifying the original\n const processedItem = { ...item };\n\n // Ensure media URLs are properly formatted\n if (\n processedItem.media_url && !processedItem.media_url.startsWith(\"http\")\n ) {\n processedItem.media_url = `https://${processedItem.media_url}`;\n }\n\n // Process preview URLs in metadata\n if (\n processedItem.metadata?.preview_url &&\n !processedItem.metadata.preview_url.startsWith(\"http\")\n ) {\n processedItem.metadata.preview_url =\n `https://${processedItem.metadata.preview_url}`;\n }\n\n // Process variant URLs\n if (processedItem.metadata?.variants) {\n processedItem.metadata.variants = processedItem.metadata.variants.map(\n (variant) => {\n if (variant.url && !variant.url.startsWith(\"http\")) {\n return { ...variant, url: `https://${variant.url}` };\n }\n return variant;\n },\n );\n }\n\n return processedItem;\n });\n }\n}\n",[],[176,488],"84232e40-8a7c-11f0-9716-0224a6c84d84","twitter-media-test.http.ts","84232f08-8a7c-11f0-9716-0224a6c84d84",45,["D",1757092784576],"import { initializeTwitterClient } from \"./twitter-api.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\nimport { getPostById, getRecentPosts } from \"./database/queries.ts\";\nimport { TwitterMediaExtractor } from \"./twitter-media.ts\";\n\n/**\n * HTTP endpoint to test Twitter media extraction functionality\n * This endpoint allows testing and validation of the media extraction features\n */\nexport default async function (req: Request): Promise {\n // Check for basic auth\n const authHeader = req.headers.get(\"Authorization\");\n if (!authHeader || !authHeader.startsWith(\"Basic \")) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n const base64Credentials = authHeader.split(\" \")[1];\n const credentials = atob(base64Credentials);\n const [username, password] = credentials.split(\":\");\n\n const expectedUsername = Deno.env.get(\"AUTH_USERNAME\");\n const expectedPassword = Deno.env.get(\"AUTH_PASSWORD\");\n\n if (username !== expectedUsername || password !== expectedPassword) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n // Process the request\n try {\n const url = new URL(req.url);\n const action = url.searchParams.get(\"action\") || \"status\";\n\n // Initialize database\n await initializeDatabase();\n\n let responseData: any = {\n status: \"ok\",\n timestamp: new Date().toISOString(),\n };\n\n if (action === \"fetch-media\") {\n // Fetch tweets with media from Twitter API\n const twitterClient = initializeTwitterClient();\n const rawTweets = await twitterClient.fetchHomeTimeline(20);\n\n // Filter to only tweets with media\n const tweetsWithMedia = rawTweets.filter((tweet) =>\n tweet.media && tweet.media.length > 0\n );\n\n responseData = {\n ...responseData,\n action: \"fetch-media\",\n totalFetched: rawTweets.length,\n withMediaCount: tweetsWithMedia.length,\n mediaItems: tweetsWithMedia.map((tweet) => ({\n tweetId: tweet.external_id,\n author: tweet.author_username,\n mediaCount: tweet.media?.length || 0,\n media: tweet.media,\n })),\n };\n } else if (action === \"media-stats\") {\n // Get media statistics from database\n const limit = parseInt(url.searchParams.get(\"limit\") || \"50\", 10);\n const recentPosts = await getRecentPosts(limit);\n\n // Count media types\n const mediaStats = {\n totalPosts: recentPosts.length,\n postsWithMedia: 0,\n mediaTypes: {} as Record,\n totalMediaItems: 0,\n };\n\n for (const post of recentPosts) {\n if (post.media && post.media.length > 0) {\n mediaStats.postsWithMedia++;\n mediaStats.totalMediaItems += post.media.length;\n\n for (const media of post.media) {\n const type = media.media_type || \"unknown\";\n mediaStats.mediaTypes[type] = (mediaStats.mediaTypes[type] || 0) +\n 1;\n }\n }\n }\n\n responseData = {\n ...responseData,\n action: \"media-stats\",\n stats: mediaStats,\n };\n } else if (action === \"post-media\") {\n // Get media for a specific post\n const postId = parseInt(url.searchParams.get(\"id\") || \"0\", 10);\n\n if (!postId) {\n return new Response(\n JSON.stringify(\n {\n status: \"error\",\n message: \"Missing post ID parameter\",\n },\n null,\n 2,\n ),\n {\n status: 400,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n const post = await getPostById(postId);\n\n if (!post) {\n return new Response(\n JSON.stringify(\n {\n status: \"error\",\n message: \"Post not found\",\n },\n null,\n 2,\n ),\n {\n status: 404,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n responseData = {\n ...responseData,\n action: \"post-media\",\n post: {\n id: post.id,\n external_id: post.external_id,\n author: post.author_username,\n content: post.content.substring(0, 50) +\n (post.content.length > 50 ? \"...\" : \"\"),\n url: post.url,\n },\n mediaCount: post.media?.length || 0,\n media: post.media,\n };\n }\n\n return new Response(JSON.stringify(responseData, null, 2), {\n status: 200,\n headers: { \"Content-Type\": \"application/json\" },\n });\n } catch (error: unknown) {\n console.error(\"Error in Twitter media test endpoint:\", error);\n\n return new Response(\n JSON.stringify(\n {\n status: \"error\",\n message: error instanceof Error ? error instanceof Error ? error.message : String(error) : String(error),\n timestamp: new Date().toISOString(),\n },\n null,\n 2,\n ),\n {\n status: 500,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n}\n",[],[176,496],"845a41a0-8a7c-11f0-be41-0224a6c84d84","twitter-fetch.cron.ts","845a4236-8a7c-11f0-be41-0224a6c84d84",54,["D",1757092784950],"import { initializeTwitterClient, storeTweets } from \"./twitter-api.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\nimport { email } from \"https://esm.town/v/std/email\";\nimport { TwitterProcessor } from \"./twitter-processor.ts\";\nimport {\n TwitterError,\n TwitterErrorType,\n withRetry,\n} from \"./twitter-error-handler.ts\";\n\n/**\n * Cron job to fetch tweets from Twitter API and store them in the database\n * This function runs on a schedule defined in Val Town\n */\nexport default async function fetchTwitterData() {\n console.log(\"Starting Twitter data fetch...\");\n\n try {\n // Initialize the database\n await initializeDatabase();\n\n // Initialize Twitter client\n const twitterClient = initializeTwitterClient();\n\n // Fetch tweets with retry mechanism already built into the client\n console.log(\"Fetching tweets from Twitter API...\");\n let rawTweets;\n try {\n rawTweets = await twitterClient.fetchHomeTimeline(100);\n console.log(`Fetched ${rawTweets.length} raw tweets from Twitter API`);\n } catch (error: unknown) {\n // Convert to TwitterError for better error handling\n const errorInstance = error instanceof Error ? error : new Error(String(error));\n const twitterError = error instanceof TwitterError\n ? error\n : new TwitterError(\n `Failed to fetch tweets: ${errorInstance.message}`,\n TwitterErrorType.Unknown,\n undefined,\n errorInstance,\n );\n\n // Handle different error types appropriately\n if (twitterError.type === TwitterErrorType.RateLimit) {\n const resetTime = twitterError.rateLimitReset;\n const resetMessage = resetTime\n ? `Rate limit will reset at ${resetTime.toLocaleString()}`\n : \"Rate limit exceeded\";\n\n throw new Error(`Twitter API rate limit exceeded. ${resetMessage}`);\n } else if (twitterError.type === TwitterErrorType.Authentication) {\n throw new Error(\n \"Twitter API authentication failed. Please check your API credentials.\",\n );\n } else {\n throw twitterError;\n }\n }\n\n // Initialize the processor with config from environment variables\n const processor = new TwitterProcessor(\n TwitterProcessor.loadConfigFromEnv(),\n );\n\n // Process tweets through the pipeline\n const processedTweets = processor.processTweets(rawTweets);\n console.log(`Processed ${processedTweets.length} tweets after filtering`);\n\n // Store tweets in the database with deduplication and built-in retry\n const newTweetsCount = await storeTweets(processedTweets);\n console.log(`Stored ${newTweetsCount} new tweets in the database`);\n\n // Send notification email if configured\n const emailTo = Deno.env.get(\"EMAIL_TO\");\n if (emailTo && newTweetsCount > 0) {\n await withRetry(async () => {\n await email({\n to: emailTo,\n subject: `OffX: Fetched ${newTweetsCount} new tweets`,\n text:\n `The OffX Twitter ingestion service has fetched ${rawTweets.length} tweets, processed ${processedTweets.length} after filtering, and stored ${newTweetsCount} new tweets.`,\n html: `

OffX Twitter Ingestion

The OffX Twitter ingestion service has:

Fetched ${rawTweets.length} tweets
Processed ${processedTweets.length} after filtering
Stored ${newTweetsCount} new tweets

`,\n });\n });\n }\n\n return {\n success: true,\n fetchedCount: rawTweets.length,\n processedCount: processedTweets.length,\n newCount: newTweetsCount,\n timestamp: new Date().toISOString(),\n };\n } catch (error: unknown) {\n console.error(\"Error in Twitter fetch cron job:\", error);\n\n // Extract useful error information\n const errorInstance = error instanceof Error ? error : new Error(String(error));\n const errorType = error instanceof TwitterError ? error.type : \"unknown\";\n const errorMessage = errorInstance.message || \"Unknown error\";\n const statusCode = error instanceof TwitterError\n ? error.statusCode\n : undefined;\n\n // Construct detailed error message\n const detailedError = `\n Type: ${errorType}\n Message: ${errorMessage}\n ${statusCode ? `Status Code: ${statusCode}` : \"\"}\n Time: ${new Date().toISOString()}\n `;\n\n // Send error notification with retry mechanism\n try {\n await withRetry(async () => {\n await email({\n subject: \"OffX: Twitter Fetch Error\",\n text: `Error in Twitter fetch cron job: ${detailedError}`,\n html: `

OffX Twitter Fetch Error

The following error occurred during the Twitter fetch cron job:

${detailedError}

Please check the logs for more details.

`,\n });\n });\n } catch (emailError) {\n console.error(\"Failed to send error notification email:\", emailError);\n }\n\n return {\n success: false,\n error: errorMessage,\n errorType,\n statusCode,\n timestamp: new Date().toISOString(),\n };\n }\n}\n",[],[176,504],"842891a0-8a7c-11f0-af15-0224a6c84d84","twitter-error-test.http.ts","84289240-8a7c-11f0-af15-0224a6c84d84",46,["D",1757092784592],"import { initializeTwitterClient } from \"./twitter-api.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\nimport {\n DEFAULT_RETRY_CONFIG,\n RetryConfig,\n TwitterError,\n TwitterErrorType,\n withRetry,\n} from \"./twitter-error-handler.ts\";\n\n/**\n * HTTP endpoint to test Twitter error handling and retry mechanisms\n * This endpoint allows testing and validation of the error handling features\n */\nexport default async function (req: Request): Promise {\n // Check for basic auth\n const authHeader = req.headers.get(\"Authorization\");\n if (!authHeader || !authHeader.startsWith(\"Basic \")) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n const base64Credentials = authHeader.split(\" \")[1];\n const credentials = atob(base64Credentials);\n const [username, password] = credentials.split(\":\");\n\n const expectedUsername = Deno.env.get(\"AUTH_USERNAME\");\n const expectedPassword = Deno.env.get(\"AUTH_PASSWORD\");\n\n if (username !== expectedUsername || password !== expectedPassword) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n // Process the request\n try {\n const url = new URL(req.url);\n const action = url.searchParams.get(\"action\") || \"status\";\n\n // Initialize database\n await initializeDatabase();\n\n let responseData: any = {\n status: \"ok\",\n timestamp: new Date().toISOString(),\n };\n\n if (action === \"test-retry\") {\n // Get retry configuration from query params or use defaults\n const maxRetries = parseInt(\n url.searchParams.get(\"max_retries\") || \"3\",\n 10,\n );\n const initialDelayMs = parseInt(\n url.searchParams.get(\"initial_delay\") || \"1000\",\n 10,\n );\n const maxDelayMs = parseInt(\n url.searchParams.get(\"max_delay\") || \"60000\",\n 10,\n );\n const backoffMultiplier = parseFloat(\n url.searchParams.get(\"backoff\") || \"2\",\n );\n const jitterFactor = parseFloat(url.searchParams.get(\"jitter\") || \"0.2\");\n\n // Create custom retry config for testing\n const retryConfig: RetryConfig = {\n ...DEFAULT_RETRY_CONFIG,\n maxRetries,\n initialDelayMs,\n maxDelayMs,\n backoffMultiplier,\n jitterFactor,\n };\n\n // Simulate a function that fails a certain number of times before succeeding\n const failCount = parseInt(url.searchParams.get(\"fail_count\") || \"2\", 10);\n let attemptCount = 0;\n\n const retryResult = await withRetry(async () => {\n attemptCount++;\n console.log(\n `Retry test attempt ${attemptCount} of ${failCount + 1} needed`,\n );\n\n if (attemptCount <= failCount) {\n throw new Error(`Simulated failure on attempt ${attemptCount}`);\n }\n\n return {\n message: \"Operation succeeded after retries\",\n attemptCount,\n };\n }, retryConfig);\n\n responseData = {\n ...responseData,\n action: \"test-retry\",\n retryConfig,\n retryResult,\n };\n } else if (action === \"test-error-types\") {\n // Test different error types\n const errorType = url.searchParams.get(\"type\") || \"rate_limit\";\n let error: TwitterError;\n\n switch (errorType) {\n case \"rate_limit\":\n error = new TwitterError(\n \"Twitter API rate limit exceeded\",\n TwitterErrorType.RateLimit,\n 429,\n );\n // Set rate limit reset time to 30 seconds from now\n error.rateLimitReset = new Date(Date.now() + 30000);\n break;\n\n case \"authentication\":\n error = new TwitterError(\n \"Twitter API authentication failed\",\n TwitterErrorType.Authentication,\n 401,\n );\n break;\n\n case \"server_error\":\n error = new TwitterError(\n \"Twitter server error\",\n TwitterErrorType.ServerError,\n 500,\n );\n break;\n\n case \"network_error\":\n error = new TwitterError(\n \"Network connection failed\",\n TwitterErrorType.NetworkError,\n );\n break;\n\n default:\n error = new TwitterError(\n \"Unknown error type\",\n TwitterErrorType.Unknown,\n );\n }\n\n responseData = {\n ...responseData,\n action: \"test-error-types\",\n errorType,\n error: {\n message: error instanceof Error ? error.message : String(error),\n type: error.type,\n statusCode: error.statusCode,\n isRetryable: error.isRetryable,\n rateLimitReset: error.rateLimitReset,\n },\n };\n } else if (action === \"test-live-retry\") {\n // Test retry mechanism with actual Twitter API\n // Get custom retry config\n const maxRetries = parseInt(\n url.searchParams.get(\"max_retries\") || \"3\",\n 10,\n );\n const initialDelayMs = parseInt(\n url.searchParams.get(\"initial_delay\") || \"1000\",\n 10,\n );\n\n const retryConfig: RetryConfig = {\n ...DEFAULT_RETRY_CONFIG,\n maxRetries,\n initialDelayMs,\n };\n\n // Initialize Twitter client with custom retry config\n const twitterClient = initializeTwitterClient();\n\n // Fetch tweets with retry mechanism\n const startTime = Date.now();\n const tweets = await twitterClient.fetchHomeTimeline(10);\n const endTime = Date.now();\n\n responseData = {\n ...responseData,\n action: \"test-live-retry\",\n retryConfig,\n tweetCount: tweets.length,\n executionTimeMs: endTime - startTime,\n tweetSample: tweets.slice(0, 2),\n };\n } else if (action === \"retry-config\") {\n // Show current retry configuration\n const config = DEFAULT_RETRY_CONFIG;\n\n // Get environment overrides\n const maxRetries = Deno.env.get(\"TWITTER_MAX_RETRIES\");\n const initialDelay = Deno.env.get(\"TWITTER_INITIAL_DELAY_MS\");\n const maxDelay = Deno.env.get(\"TWITTER_MAX_DELAY_MS\");\n const backoffMultiplier = Deno.env.get(\"TWITTER_BACKOFF_MULTIPLIER\");\n const jitterFactor = Deno.env.get(\"TWITTER_JITTER_FACTOR\");\n\n responseData = {\n ...responseData,\n action: \"retry-config\",\n defaultConfig: config,\n environmentOverrides: {\n maxRetries: maxRetries ? parseInt(maxRetries, 10) : undefined,\n initialDelayMs: initialDelay ? parseInt(initialDelay, 10) : undefined,\n maxDelayMs: maxDelay ? parseInt(maxDelay, 10) : undefined,\n backoffMultiplier: backoffMultiplier\n ? parseFloat(backoffMultiplier)\n : undefined,\n jitterFactor: jitterFactor ? parseFloat(jitterFactor) : undefined,\n },\n };\n }\n\n return new Response(JSON.stringify(responseData, null, 2), {\n status: 200,\n headers: { \"Content-Type\": \"application/json\" },\n });\n } catch (error: unknown) {\n console.error(\"Error in Twitter error test endpoint:\", error);\n\n // Convert to TwitterError if needed\n const twitterError = error instanceof TwitterError\n ? error\n : TwitterError.fromError(error instanceof Error ? error : new Error(String(error)));\n\n return new Response(\n JSON.stringify(\n {\n status: \"error\",\n message: error instanceof Error ? error instanceof Error ? error.message : String(error) : String(error),\n errorType: twitterError.type,\n isRetryable: twitterError.isRetryable,\n statusCode: twitterError.statusCode,\n timestamp: new Date().toISOString(),\n },\n null,\n 2,\n ),\n {\n status: 500,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n}\n",[],[176,512],"8472e6a6-8a7c-11f0-9dec-0224a6c84d84","twitter-error-handler.ts","8472e76e-8a7c-11f0-9dec-0224a6c84d84",57,["D",1757092785078],"/**\n * Error handling and retry mechanisms for Twitter API requests\n * Provides utilities for handling rate limits, network failures, and transient errors\n */\n\n/**\n * Configuration options for retry mechanisms\n */\nexport interface RetryConfig {\n /** Maximum number of retry attempts */\n maxRetries: number;\n /** Initial delay in milliseconds before the first retry */\n initialDelayMs: number;\n /** Maximum delay in milliseconds between retries */\n maxDelayMs: number;\n /** Multiplier for exponential backoff */\n backoffMultiplier: number;\n /** Jitter factor to add randomness to retry delays (0-1) */\n jitterFactor: number;\n /** HTTP status codes that should trigger a retry */\n retryableStatusCodes: number[];\n /** Error types that should trigger a retry */\n retryableErrors: string[];\n}\n\n/**\n * Default retry configuration\n */\nexport const DEFAULT_RETRY_CONFIG: RetryConfig = {\n maxRetries: 3,\n initialDelayMs: 1000, // 1 second\n maxDelayMs: 60000, // 60 seconds\n backoffMultiplier: 2,\n jitterFactor: 0.2,\n retryableStatusCodes: [\n 408, // Request Timeout\n 429, // Too Many Requests (rate limit)\n 500, // Internal Server Error\n 502, // Bad Gateway\n 503, // Service Unavailable\n 504, // Gateway Timeout\n ],\n retryableErrors: [\n \"ECONNRESET\",\n \"ETIMEDOUT\",\n \"ECONNREFUSED\",\n \"EPIPE\",\n \"EHOSTUNREACH\",\n \"ENETUNREACH\",\n \"ENOTFOUND\",\n \"socket hang up\",\n \"network error\",\n ],\n};\n\n/**\n * Result of a retry operation\n */\nexport interface RetryResult {\n /** Whether the operation was successful */\n success: boolean;\n /** The result of the operation if successful */\n result?: T;\n /** Error that occurred if the operation failed */\n error?: Error;\n /** Number of retry attempts made */\n attempts: number;\n /** Total time spent in retries (ms) */\n totalRetryTimeMs: number;\n /** Whether the operation reached the maximum number of retries */\n maxRetriesReached: boolean;\n}\n\n/**\n * Twitter API specific error types\n */\nexport enum TwitterErrorType {\n RateLimit = \"rate_limit\",\n Authentication = \"authentication\",\n Authorization = \"authorization\",\n NotFound = \"not_found\",\n InvalidRequest = \"invalid_request\",\n ServerError = \"server_error\",\n NetworkError = \"network_error\",\n Unknown = \"unknown\",\n}\n\n/**\n * Twitter API error with additional context\n */\nexport class TwitterError extends Error {\n /** Type of Twitter error */\n type: TwitterErrorType;\n /** HTTP status code if available */\n statusCode?: number;\n /** Rate limit reset time if applicable */\n rateLimitReset?: Date;\n /** Whether this error is retryable */\n isRetryable: boolean;\n /** Original error that caused this error */\n originalError?: Error;\n\n constructor(\n message: string,\n type: TwitterErrorType = TwitterErrorType.Unknown,\n statusCode?: number,\n originalError?: Error,\n ) {\n super(message);\n this.name = \"TwitterError\";\n this.type = type;\n this.statusCode = statusCode;\n this.originalError = originalError;\n\n // Determine if error is retryable\n this.isRetryable = this.determineIfRetryable();\n }\n\n /**\n * Determine if this error should be retried\n */\n private determineIfRetryable(): boolean {\n // Rate limits are always retryable\n if (this.type === TwitterErrorType.RateLimit) {\n return true;\n }\n\n // Server errors are retryable\n if (this.type === TwitterErrorType.ServerError) {\n return true;\n }\n\n // Network errors are retryable\n if (this.type === TwitterErrorType.NetworkError) {\n return true;\n }\n\n // Authentication and authorization errors are not retryable\n if (\n this.type === TwitterErrorType.Authentication ||\n this.type === TwitterErrorType.Authorization\n ) {\n return false;\n }\n\n // Check status code if available\n if (this.statusCode) {\n return DEFAULT_RETRY_CONFIG.retryableStatusCodes.includes(\n this.statusCode,\n );\n }\n\n return false;\n }\n\n /**\n * Create a TwitterError from an HTTP response\n */\n static fromResponse(response: Response, originalError?: Error): TwitterError {\n const statusCode = response.status;\n let type = TwitterErrorType.Unknown;\n let message = `Twitter API error: ${response.statusText}`;\n\n // Determine error type based on status code\n if (statusCode === 401) {\n type = TwitterErrorType.Authentication;\n message = \"Twitter API authentication failed\";\n } else if (statusCode === 403) {\n type = TwitterErrorType.Authorization;\n message = \"Not authorized to perform this action\";\n } else if (statusCode === 404) {\n type = TwitterErrorType.NotFound;\n message = \"Requested resource not found\";\n } else if (statusCode === 429) {\n type = TwitterErrorType.RateLimit;\n message = \"Twitter API rate limit exceeded\";\n\n // Parse rate limit reset time if available\n const resetHeader = response.headers.get(\"x-rate-limit-reset\");\n if (resetHeader) {\n const resetTime = parseInt(resetHeader, 10) * 1000; // Convert to milliseconds\n const error = new TwitterError(\n message,\n type,\n statusCode,\n originalError,\n );\n error.rateLimitReset = new Date(resetTime);\n return error;\n }\n } else if (statusCode >= 400 && statusCode < 500) {\n type = TwitterErrorType.InvalidRequest;\n } else if (statusCode >= 500) {\n type = TwitterErrorType.ServerError;\n }\n\n return new TwitterError(message, type, statusCode, originalError);\n }\n\n /**\n * Create a TwitterError from any error\n */\n static fromError(error: Error): TwitterError {\n // Check if it's already a TwitterError\n if (error instanceof TwitterError) {\n return error;\n }\n\n // Check for network errors\n const networkErrorPatterns = DEFAULT_RETRY_CONFIG.retryableErrors;\n const isNetworkError = networkErrorPatterns.some((pattern) => {\n const errorString = error instanceof Error\n ? error.message\n : String(error);\n const errorName = error instanceof Error ? error.name : \"\";\n return errorString.includes(pattern) || errorName.includes(pattern);\n });\n\n if (isNetworkError) {\n return new TwitterError(\n `Network error: ${\n error instanceof Error ? error.message : String(error)\n }`,\n TwitterErrorType.NetworkError,\n undefined,\n error,\n );\n }\n\n // Default to unknown error\n return new TwitterError(\n `Twitter API error: ${\n error instanceof Error ? error.message : String(error)\n }`,\n TwitterErrorType.Unknown,\n undefined,\n error,\n );\n }\n}\n\n/**\n * Calculate delay for the next retry attempt using exponential backoff with jitter\n */\nexport function calculateRetryDelay(\n attempt: number,\n config: RetryConfig = DEFAULT_RETRY_CONFIG,\n): number {\n // Base delay with exponential backoff\n const baseDelay = Math.min(\n config.initialDelayMs * Math.pow(config.backoffMultiplier, attempt),\n config.maxDelayMs,\n );\n\n // Add jitter to prevent thundering herd problem\n const jitterRange = baseDelay * config.jitterFactor;\n const jitter = Math.random() * jitterRange - (jitterRange / 2);\n\n return Math.max(0, Math.floor(baseDelay + jitter));\n}\n\n/**\n * Execute a function with retry logic\n * @param operation Function to execute with retry logic\n * @param config Retry configuration\n * @returns Result of the operation with retry metadata\n */\nexport async function withRetry(\n operation: () => Promise,\n config: RetryConfig = DEFAULT_RETRY_CONFIG,\n): Promise> {\n let attempts = 0;\n let lastError: Error | undefined;\n const startTime = Date.now();\n\n while (attempts <= config.maxRetries) {\n try {\n // Attempt the operation\n const result = await operation();\n\n // Operation succeeded\n return {\n success: true,\n result,\n attempts,\n totalRetryTimeMs: Date.now() - startTime,\n maxRetriesReached: false,\n };\n } catch (error: unknown) {\n attempts++;\n lastError = error instanceof Error ? error : new Error(String(error));\n\n // Convert to TwitterError for better handling\n const twitterError = TwitterError.fromError(lastError);\n\n // Log the error\n console.error(\n `Twitter API error (attempt ${attempts}/${config.maxRetries + 1}):`,\n twitterError,\n );\n\n // Check if we should retry\n const shouldRetry = attempts <= config.maxRetries &&\n twitterError.isRetryable;\n\n if (!shouldRetry) {\n break;\n }\n\n // Calculate delay for next retry\n const delay = calculateRetryDelay(attempts - 1, config);\n\n // Special handling for rate limit errors with reset time\n if (\n twitterError.type === TwitterErrorType.RateLimit &&\n twitterError.rateLimitReset\n ) {\n const resetTime = twitterError.rateLimitReset.getTime();\n const now = Date.now();\n const timeToReset = Math.max(0, resetTime - now);\n\n // Use the rate limit reset time if it's sooner than our calculated delay\n if (timeToReset > 0 && timeToReset < delay) {\n console.log(\n `Rate limit exceeded. Waiting for reset in ${timeToReset}ms`,\n );\n await new Promise((resolve) =>\n setTimeout(resolve, timeToReset + 1000)\n ); // Add 1 second buffer\n continue;\n }\n }\n\n // Wait before retrying\n console.log(`Retrying in ${delay}ms...`);\n await new Promise((resolve) => setTimeout(resolve, delay));\n }\n }\n\n // All retries failed\n return {\n success: false,\n error: lastError,\n attempts,\n totalRetryTimeMs: Date.now() - startTime,\n maxRetriesReached: attempts > config.maxRetries,\n };\n}\n\n/**\n * Load retry configuration from environment variables\n */\nexport function loadRetryConfigFromEnv(): RetryConfig {\n const config = { ...DEFAULT_RETRY_CONFIG };\n\n const maxRetries = Deno.env.get(\"TWITTER_MAX_RETRIES\");\n if (maxRetries) {\n config.maxRetries = parseInt(maxRetries, 10);\n }\n\n const initialDelay = Deno.env.get(\"TWITTER_INITIAL_DELAY_MS\");\n if (initialDelay) {\n config.initialDelayMs = parseInt(initialDelay, 10);\n }\n\n const maxDelay = Deno.env.get(\"TWITTER_MAX_DELAY_MS\");\n if (maxDelay) {\n config.maxDelayMs = parseInt(maxDelay, 10);\n }\n\n const backoffMultiplier = Deno.env.get(\"TWITTER_BACKOFF_MULTIPLIER\");\n if (backoffMultiplier) {\n config.backoffMultiplier = parseFloat(backoffMultiplier);\n }\n\n const jitterFactor = Deno.env.get(\"TWITTER_JITTER_FACTOR\");\n if (jitterFactor) {\n config.jitterFactor = parseFloat(jitterFactor);\n }\n\n return config;\n}\n",[],[176,520],"9e1ae140-8ad0-11f0-8132-0224a6c84d84","twitter-api.ts","84a0b14e-8a7c-11f0-a12e-0224a6c84d84",97,["D",1757128905876],"import { TwitterApi } from \"https://esm.sh/twitter-api-v2\";\nimport {\n createPost,\n getPostByExternalId,\n Post,\n PostMedia,\n} from \"./database/queries.ts\";\nimport { ExtendedPostMedia, TwitterMediaExtractor } from \"./twitter-media.ts\";\nimport {\n loadRetryConfigFromEnv,\n RetryConfig,\n TwitterError,\n TwitterErrorType,\n withRetry,\n} from \"./twitter-error-handler.ts\";\n\ninterface TwitterClientConfig {\n appKey: string;\n appSecret: string;\n accessToken: string;\n accessSecret: string;\n}\n\n/**\n * Twitter API client configuration and initialization\n */\nexport class TwitterClient {\n private client: TwitterApi;\n private retryConfig: RetryConfig;\n\n constructor(config: TwitterClientConfig, retryConfig?: RetryConfig) {\n if (\n !config.appKey || !config.appSecret || !config.accessToken ||\n !config.accessSecret\n ) {\n throw new Error(\n \"Twitter app key, app secret, access token, and access token secret are required\",\n );\n }\n this.client = new TwitterApi(config).readOnly;\n this.retryConfig = retryConfig || loadRetryConfigFromEnv();\n }\n\n /**\n * Fetches the latest tweets from the user's home timeline\n * @param maxResults Maximum number of tweets to fetch (default: 100)\n * @returns Array of processed tweets\n */\n async fetchHomeTimeline(maxResults: number = 100): Promise {\n const retryResult = await withRetry(\n async () => {\n const homeTimeline = await this.client.v2.homeTimeline({\n max_results: maxResults,\n \"tweet.fields\": [\n \"created_at\",\n \"author_id\",\n \"text\",\n \"attachments\",\n \"entities\",\n ],\n \"user.fields\": [\"name\", \"username\"],\n \"media.fields\": [\n \"url\",\n \"preview_image_url\",\n \"type\",\n \"alt_text\",\n \"width\",\n \"height\",\n \"duration_ms\",\n \"variants\",\n ],\n expansions: [\n \"author_id\",\n \"attachments.media_keys\",\n \"entities.mentions.username\",\n ],\n });\n\n console.log(homeTimeline);\n\n // Process and return the tweets\n return this.processTweets(homeTimeline.data);\n },\n this.retryConfig,\n );\n\n if (!retryResult.success) {\n // Log detailed information about the failure\n console.error(\n \"Failed to fetch Twitter timeline after multiple attempts:\",\n {\n attempts: retryResult.attempts,\n totalRetryTimeMs: retryResult.totalRetryTimeMs,\n maxRetriesReached: retryResult.maxRetriesReached,\n error: retryResult.error,\n },\n );\n\n // Convert to TwitterError for consistent error handling\n const twitterError = TwitterError.fromError(\n retryResult.error || new Error(\"Unknown error\"),\n );\n\n // Throw the error with additional context\n throw new TwitterError(\n `Failed to fetch Twitter timeline after ${retryResult.attempts} attempts: ${twitterError.message}`,\n twitterError.type,\n twitterError.statusCode,\n twitterError,\n );\n }\n\n return retryResult.result || [];\n }\n\n /**\n * Process raw Twitter API response into our database format\n * @param timelineData Twitter API response data\n * @returns Array of processed Post objects ready for database storage\n */\n private processTweets(timelineData: any): Post[] {\n const tweets: Post[] = [];\n\n if (\n !timelineData || !timelineData.data || !Array.isArray(timelineData.data)\n ) {\n console.warn(\"No tweets found in the Twitter API response\");\n return [];\n }\n\n // Process each tweet\n for (const tweet of timelineData.data) {\n try {\n // Find the author in the includes.users array\n const author = timelineData.includes?.users?.find(\n (user: any) => user.id === tweet.author_id,\n );\n\n if (!author) {\n console.warn(`Author not found for tweet ${tweet.id}`);\n continue;\n }\n\n // Extract media using the specialized extractor\n const media: ExtendedPostMedia[] = TwitterMediaExtractor\n .extractMediaFromResponse(\n timelineData,\n tweet.id,\n );\n\n // Process the extracted media\n const processedMedia = TwitterMediaExtractor.processMedia(media);\n\n // Create post object\n const post: Post = {\n external_id: tweet.id,\n content: tweet.text,\n author: author.name,\n author_username: author.username,\n platform: \"twitter\",\n created_at: tweet.created_at || new Date().toISOString(),\n url: `https://twitter.com/${author.username}/status/${tweet.id}`,\n media: processedMedia,\n };\n\n tweets.push(post);\n } catch (error: unknown) {\n // Log error but continue processing other tweets\n console.error(`Error processing tweet ${tweet.id}:`, error);\n }\n }\n\n return tweets;\n }\n}\n\n/**\n * Initialize Twitter API client with bearer token from environment\n * @returns Configured TwitterClient instance\n */\nexport function initializeTwitterClient(): TwitterClient {\n const appKey = Deno.env.get(\"TWITTER_APP_KEY\");\n const appSecret = Deno.env.get(\"TWITTER_APP_SECRET\");\n const accessToken = Deno.env.get(\"TWITTER_ACCESS_TOKEN\");\n const accessSecret = Deno.env.get(\"TWITTER_ACCESS_TOKEN_SECRET\");\n\n if (!appKey || !appSecret || !accessToken || !accessSecret) {\n throw new Error(\n \"TWITTER_APP_KEY, TWITTER_APP_SECRET, TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_TOKEN_SECRET environment variables are not set\",\n );\n }\n\n return new TwitterClient({\n appKey,\n appSecret,\n accessToken,\n accessSecret,\n });\n}\n\n/**\n * Store tweets in the database with deduplication\n * @param tweets Array of processed tweets\n * @returns Number of new tweets stored\n */\nexport async function storeTweets(tweets: Post[]): Promise {\n let newTweetsCount = 0;\n let failedTweets = 0;\n\n for (const tweet of tweets) {\n try {\n // Use retry mechanism for database operations as well\n const retryResult = await withRetry(async () => {\n // Check if the tweet already exists in the database\n const existingTweet = await getPostByExternalId(tweet.external_id);\n\n if (!existingTweet) {\n // Store the new tweet\n const postId = await createPost(tweet);\n\n // If the tweet has media, ensure it's properly stored with the enhanced metadata\n if (tweet.media && tweet.media.length > 0) {\n await TwitterMediaExtractor.storePostMedia(\n postId,\n tweet.media as ExtendedPostMedia[],\n );\n }\n\n return true; // New tweet stored\n }\n\n return false; // Tweet already exists\n });\n\n if (retryResult.success && retryResult.result) {\n newTweetsCount++;\n }\n } catch (error: unknown) {\n failedTweets++;\n console.error(`Failed to store tweet ${tweet.external_id}:`, error);\n }\n }\n\n // Log summary\n if (failedTweets > 0) {\n console.warn(\n `Failed to store ${failedTweets} tweets out of ${tweets.length}`,\n );\n }\n\n return newTweetsCount;\n}\n",[],[176,528],"841d63de-8a7c-11f0-abb7-0224a6c84d84","topic-api-test.http.ts","841d64b0-8a7c-11f0-abb7-0224a6c84d84",44,["D",1757092784549],"/**\n * Topic Management API Test Endpoints\n * This file provides HTTP endpoints to test the Topic Management API functionality\n */\nexport default async function (req: Request): Promise {\n // Check for basic auth\n const authHeader = req.headers.get(\"Authorization\");\n if (!authHeader || !authHeader.startsWith(\"Basic \")) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n const base64Credentials = authHeader.split(\" \")[1];\n const credentials = atob(base64Credentials);\n const [username, password] = credentials.split(\":\");\n\n // Simple hard-coded auth for testing\n const validUsername = Deno.env.get(\"AUTH_USERNAME\") || \"admin\";\n const validPassword = Deno.env.get(\"AUTH_PASSWORD\") || \"password\";\n\n if (username !== validUsername || password !== validPassword) {\n return new Response(\"Unauthorized\", {\n status: 401,\n headers: { \"WWW-Authenticate\": \"Basic\" },\n });\n }\n\n // Parse URL to get action\n const url = new URL(req.url);\n const action = url.searchParams.get(\"action\") || \"help\";\n\n // Import necessary modules\n const { initializeDatabase } = await import(\"./database/migrations.ts\");\n const {\n createTopic,\n getAllTopics,\n getTopicById,\n updateTopic,\n deleteTopic,\n } = await import(\"./database/queries.ts\");\n\n // Initialize database\n await initializeDatabase();\n\n // Process actions\n switch (action) {\n case \"list\":\n // List all topics\n const topics = await getAllTopics();\n return new Response(\n JSON.stringify({ action: \"list\", topics }, null, 2),\n {\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n\n case \"get\":\n // Get a specific topic\n const id = parseInt(url.searchParams.get(\"id\") || \"0\");\n if (isNaN(id) || id <= 0) {\n return new Response(\n JSON.stringify({ error: \"Invalid topic ID\" }, null, 2),\n {\n status: 400,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n const topic = await getTopicById(id);\n if (!topic) {\n return new Response(\n JSON.stringify({ error: \"Topic not found\" }, null, 2),\n {\n status: 404,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n return new Response(\n JSON.stringify({ action: \"get\", topic }, null, 2),\n {\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n\n case \"create\":\n // Create a new topic\n const title = url.searchParams.get(\"title\");\n const description = url.searchParams.get(\"description\");\n\n if (!title) {\n return new Response(\n JSON.stringify({ error: \"Title is required\" }, null, 2),\n {\n status: 400,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n const newTopicId = await createTopic({\n title,\n description: description || undefined,\n });\n\n const newTopic = await getTopicById(newTopicId);\n return new Response(\n JSON.stringify({ action: \"create\", topic: newTopic }, null, 2),\n {\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n\n case \"update\":\n // Update a topic\n const updateId = parseInt(url.searchParams.get(\"id\") || \"0\");\n const updateTitle = url.searchParams.get(\"title\");\n const updateDescription = url.searchParams.get(\"description\");\n\n if (isNaN(updateId) || updateId <= 0) {\n return new Response(\n JSON.stringify({ error: \"Invalid topic ID\" }, null, 2),\n {\n status: 400,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n if (!updateTitle && updateDescription === null) {\n return new Response(\n JSON.stringify(\n { error: \"At least one field must be provided for update\" },\n null,\n 2,\n ),\n {\n status: 400,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n // Check if topic exists\n const existingTopic = await getTopicById(updateId);\n if (!existingTopic) {\n return new Response(\n JSON.stringify({ error: \"Topic not found\" }, null, 2),\n {\n status: 404,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n // Update the topic\n await updateTopic({\n id: updateId,\n title: updateTitle || existingTopic.title,\n description: updateDescription !== null\n ? updateDescription\n : existingTopic.description,\n });\n\n const updatedTopic = await getTopicById(updateId);\n return new Response(\n JSON.stringify({ action: \"update\", topic: updatedTopic }, null, 2),\n {\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n\n case \"delete\":\n // Delete a topic\n const deleteId = parseInt(url.searchParams.get(\"id\") || \"0\");\n\n if (isNaN(deleteId) || deleteId <= 0) {\n return new Response(\n JSON.stringify({ error: \"Invalid topic ID\" }, null, 2),\n {\n status: 400,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n // Check if topic exists\n const topicToDelete = await getTopicById(deleteId);\n if (!topicToDelete) {\n return new Response(\n JSON.stringify({ error: \"Topic not found\" }, null, 2),\n {\n status: 404,\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n\n // Delete the topic\n await deleteTopic(deleteId);\n return new Response(\n JSON.stringify(\n {\n action: \"delete\",\n message: \"Topic deleted successfully\",\n topicId: deleteId,\n },\n null,\n 2,\n ),\n {\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n\n case \"help\":\n default:\n // Show help information\n return new Response(\n JSON.stringify(\n {\n message: \"Topic Management API Test\",\n availableActions: {\n list: \"List all topics - ?action=list\",\n get: \"Get a specific topic - ?action=get&id=1\",\n create:\n \"Create a new topic - ?action=create&title=NewTopic&description=Description\",\n update:\n \"Update a topic - ?action=update&id=1&title=UpdatedTitle&description=UpdatedDescription\",\n delete: \"Delete a topic - ?action=delete&id=1\",\n },\n examples: {\n list: \"/api/topic-test?action=list\",\n get: \"/api/topic-test?action=get&id=1\",\n create:\n \"/api/topic-test?action=create&title=Tech News&description=Latest tech updates\",\n update:\n \"/api/topic-test?action=update&id=1&title=Updated Tech News\",\n delete: \"/api/topic-test?action=delete&id=1\",\n },\n },\n null,\n 2,\n ),\n {\n headers: { \"Content-Type\": \"application/json\" },\n },\n );\n }\n}\n",[],[176,536],"fb0432d8-8a8c-11f0-a4af-0224a6c84d84","index.ts",89,["D",1757099855979],13,"import { Hono } from \"npm:hono@4.8.3\";\nimport { basicAuth } from \"npm:hono@4.8.3/basic-auth\";\nimport {\n readFile,\n serveFile,\n} from \"https://esm.town/v/std/utils@85-main/index.ts\";\nimport { initializeDatabase } from \"./database/index.ts\";\nimport topicRoutes from \"./routes/topics.ts\";\nimport configRoutes from \"./routes/config.ts\";\nimport summarizationRoutes from \"./routes/summarizations.ts\";\nimport { sanitizeMiddleware } from \"./security/input-sanitizer.ts\";\n\n// Initialize the database\ntry {\n await initializeDatabase();\n console.log(\"Database initialized successfully\");\n} catch (error: unknown) {\n console.error(\"Error initializing database:\", error);\n}\n\nconst app = new Hono();\n\n// Apply input sanitization middleware globally\napp.use(\"*\", sanitizeMiddleware);\n\n// Unwrap Hono errors to see original error details\napp.onError((err, c) => {\n throw err;\n});\n\n// Basic authentication middleware\nconst auth = basicAuth({\n username: Deno.env.get(\"AUTH_USERNAME\")!,\n password: Deno.env.get(\"AUTH_PASSWORD\")!,\n // This is just a placeholder - in production, always use environment variables\n});\n\n// Public routes\napp.get(\"/api/health\", (c) => {\n return c.json({ status: \"ok\", timestamp: new Date().toISOString() });\n});\n\n// Login route that returns success message on correct credentials\napp.get(\"/api/login\", auth, (c) => {\n return c.json({\n success: true,\n message: \"Authentication successful\",\n user: c.req.header(\"Authorization\")?.split(\" \")[1]?.split(\":\")[0] ||\n \"user\",\n });\n});\n\n// Protected API routes\napp.get(\"/api/dashboard\", auth, (c) => {\n return c.json({\n message: \"Protected dashboard data\",\n data: {\n summaries: [],\n topics: [],\n },\n });\n});\n\n// Apply authentication middleware to protected routes\napp.use(\"/api/topics/*\", auth);\napp.use(\"/api/config/*\", auth);\napp.use(\"/api/summarizations/*\", auth);\n\n// Mount routes\napp.route(\"/api/topics\", topicRoutes);\napp.route(\"/api/config\", configRoutes);\napp.route(\"/api/summarizations\", summarizationRoutes);\n\n// Serve static files\napp.get(\"/frontend/*\", (c) => serveFile(c.req.path, import.meta.url));\napp.get(\"/shared/*\", (c) => serveFile(c.req.path, import.meta.url));\n\n// For SPA routing, serve index.html for all other routes\napp.get(\"*\", async (c) => {\n try {\n // Try to serve the requested file first\n return await serveFile(c.req.path, import.meta.url);\n } catch {\n // If file not found, serve the index.html for client-side routing\n const html = await readFile(\"/frontend/index.html\", import.meta.url);\n return c.html(html);\n }\n});\n\n// Export the fetch handler for HTTP vals\nexport default app.fetch;\n",[],[176,544],"84350570-8a7c-11f0-b5d4-0224a6c84d84","email-delivery.ts","8435062e-8a7c-11f0-b5d4-0224a6c84d84",48,["D",1757092784657],"email","/**\n * Email delivery service for sending AI-generated summaries to users\n * Provides user-friendly summary emails with rich formatting\n */\n\nimport { email } from \"https://esm.town/v/std/email\";\nimport { getRecentSummarizations } from \"./database/queries.ts\";\nimport type { TopicSummary } from \"./twitter-summarization.ts\";\n\nexport interface EmailDeliveryOptions {\n recipientEmail: string;\n subject?: string;\n includeLastHours?: number;\n maxSummaries?: number;\n includeMetrics?: boolean;\n}\n\nexport interface EmailSummaryData {\n id: number;\n topicTitle: string;\n content: {\n summary: string;\n keyPoints: string[];\n sentiment: {\n sentiment: \"positive\" | \"negative\" | \"neutral\";\n confidence: number;\n score: number;\n };\n tweetCount: number;\n timeWindow: string;\n trendingKeywords: string[];\n influentialAuthors: string[];\n };\n created_at: string;\n time_period: string;\n}\n\n/**\n * Main email delivery service class\n */\nexport class EmailDeliveryService {\n \n /**\n * Send latest summaries via email\n */\n async sendLatestSummaries(options: EmailDeliveryOptions): Promise<{\n success: boolean;\n sentCount: number;\n error?: string;\n }> {\n try {\n const {\n recipientEmail,\n subject = \"Your Twitter Feed Summary\",\n maxSummaries = 10,\n includeMetrics = true,\n } = options;\n\n console.log(`📧 Preparing to send summary email to ${recipientEmail}`);\n\n // Fetch recent summaries from database\n const summaries = await this.fetchRecentSummaries(maxSummaries);\n \n if (summaries.length === 0) {\n console.log(\"📧 No recent summaries found, skipping email\");\n return { success: true, sentCount: 0 };\n }\n\n // Generate email content\n const emailContent = this.generateEmailTemplate(summaries, includeMetrics);\n const finalSubject = this.generateSubject(subject, summaries.length);\n\n // Send email with retry logic\n await this.sendEmailWithRetry({\n to: recipientEmail,\n subject: finalSubject,\n html: emailContent.html,\n text: emailContent.text,\n });\n\n console.log(`📧 Successfully sent summary email with ${summaries.length} summaries`);\n return { success: true, sentCount: summaries.length };\n\n } catch (error: unknown) {\n console.error(\"📧 Failed to send summary email:\", error);\n return { \n success: false, \n sentCount: 0, \n error: error instanceof Error ? error instanceof Error ? error.message : String(error) : String(error)\n };\n }\n }\n\n /**\n * Fetch recent summaries from database and format them\n */\n private async fetchRecentSummaries(limit: number): Promise {\n try {\n const rawSummaries = await getRecentSummarizations(limit);\n \n return rawSummaries.map(summary => ({\n id: summary.id!,\n topicTitle: (summary as any).topic_title || \"Unknown Topic\",\n content: JSON.parse(summary.content),\n created_at: summary.created_at!,\n time_period: `${summary.time_period_start} to ${summary.time_period_end}`,\n }));\n } catch (error: unknown) {\n console.error(\"Error fetching summaries:\", error);\n return [];\n }\n }\n\n /**\n * Generate email subject line\n */\n private generateSubject(baseSubject: string, summaryCount: number): string {\n const timeStr = new Date().toLocaleDateString();\n return `${baseSubject} - ${summaryCount} topics (${timeStr})`;\n }\n\n /**\n * Generate complete email template with HTML and text versions\n */\n private generateEmailTemplate(summaries: EmailSummaryData[], includeMetrics: boolean): {\n html: string;\n text: string;\n } {\n const html = this.generateHtmlTemplate(summaries, includeMetrics);\n const text = this.generateTextTemplate(summaries, includeMetrics);\n \n return { html, text };\n }\n\n /**\n * Generate HTML email template\n */\n private generateHtmlTemplate(summaries: EmailSummaryData[], includeMetrics: boolean): string {\n const summaryCards = summaries.map(summary => this.generateSummaryCard(summary)).join('\\n');\n \n const metricsSection = includeMetrics ? `\n

📊 Summary Metrics

Topics covered: ${summaries.length}
Total tweets analyzed: ${summaries.reduce((sum, s) => sum + s.content.tweetCount, 0)}
Sentiment breakdown: ${this.getSentimentBreakdown(summaries)}
Generated: ${new Date().toLocaleString()}

\n ` : '';\n\n return `\n \n \n \n \n \n Your Twitter Feed Summary\n \n \n \n

🐦 Your Twitter Feed Summary

AI-powered insights from your social media

\n\n ${metricsSection}\n\n

📝 Topic Summaries

\n ${summaryCards}\n

\n\n

\n Generated by OffX AI Summary Service
\n This email contains AI-generated content based on your Twitter feed\n

\n \n \n \n `;\n }\n\n /**\n * Generate individual summary card HTML\n */\n private generateSummaryCard(summary: EmailSummaryData): string {\n const sentimentColor = {\n positive: '#28a745',\n negative: '#dc3545',\n neutral: '#6c757d'\n }[summary.content.sentiment.sentiment];\n\n const keyPointsList = summary.content.keyPoints\n .map(point => `

${point}

`)\n .join('');\n\n const keywords = summary.content.trendingKeywords.length > 0 \n ? summary.content.trendingKeywords.slice(0, 5).map(kw => \n `${kw}`\n ).join('')\n : 'No keywords identified';\n\n return `\n

\n \n

📌 ${summary.topicTitle}

🗣️ ${summary.content.tweetCount} tweets

\n ${summary.content.sentiment.sentiment.toUpperCase()}\n

\n\n

\n ${summary.content.summary}\n

\n\n ${summary.content.keyPoints.length > 0 ? `\n

🔑 Key Points:

\n ${keyPointsList}\n \n

\n ` : ''}\n\n

🏷️ Trending Keywords:

${keywords}

\n\n ${summary.content.influentialAuthors.length > 0 ? `\n

\n Notable voices: ${summary.content.influentialAuthors.slice(0, 3).join(', ')}\n

\n ` : ''}\n\n

\n Generated ${new Date(summary.created_at).toLocaleString()} • Time window: ${summary.time_period}\n

\n \n

\n `;\n }\n\n /**\n * Generate plain text email template\n */\n private generateTextTemplate(summaries: EmailSummaryData[], includeMetrics: boolean): string {\n const header = `\n🐦 YOUR TWITTER FEED SUMMARY\nAI-powered insights from your social media\n\n${new Date().toLocaleDateString()}\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n`;\n\n const metricsSection = includeMetrics ? `\n📊 SUMMARY METRICS\n• Topics covered: ${summaries.length}\n• Total tweets analyzed: ${summaries.reduce((sum, s) => sum + s.content.tweetCount, 0)}\n• Sentiment breakdown: ${this.getSentimentBreakdown(summaries)}\n• Generated: ${new Date().toLocaleString()}\n\n` : '';\n\n const summaryTexts = summaries.map(summary => `\n📌 ${summary.topicTitle.toUpperCase()}\n🗣️ ${summary.content.tweetCount} tweets • Sentiment: ${summary.content.sentiment.sentiment.toUpperCase()}\n\n${summary.content.summary}\n\n🔑 Key Points:\n${summary.content.keyPoints.map(point => `• ${point}`).join('\\n')}\n\n🏷️ Keywords: ${summary.content.trendingKeywords.slice(0, 5).join(', ') || 'None'}\n\n${summary.content.influentialAuthors.length > 0 ? `Notable voices: ${summary.content.influentialAuthors.slice(0, 3).join(', ')}` : ''}\n\nGenerated: ${new Date(summary.created_at).toLocaleString()} • Time window: ${summary.time_period}\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n`).join('\\n');\n\n const footer = `\nGenerated by OffX AI Summary Service\nThis email contains AI-generated content based on your Twitter feed\n`;\n\n return header + metricsSection + summaryTexts + footer;\n }\n\n /**\n * Get sentiment breakdown statistics\n */\n private getSentimentBreakdown(summaries: EmailSummaryData[]): string {\n const counts = summaries.reduce((acc, s) => {\n acc[s.content.sentiment.sentiment] = (acc[s.content.sentiment.sentiment] || 0) + 1;\n return acc;\n }, {} as Record);\n\n return Object.entries(counts)\n .map(([sentiment, count]) => `${count} ${sentiment}`)\n .join(', ');\n }\n\n /**\n * Send email with retry logic\n */\n private async sendEmailWithRetry(emailData: {\n to: string;\n subject: string;\n html: string;\n text: string;\n }, maxRetries: number = 3): Promise {\n let lastError: Error | null = null;\n\n for (let attempt = 1; attempt <= maxRetries; attempt++) {\n try {\n console.log(`📧 Sending email attempt ${attempt}/${maxRetries} to ${emailData.to}`);\n \n await email({\n to: emailData.to,\n subject: emailData.subject,\n html: emailData.html,\n text: emailData.text,\n });\n\n console.log(`📧 Email sent successfully on attempt ${attempt}`);\n return; // Success!\n\n } catch (error: unknown) {\n lastError = error instanceof Error ? error : new Error(String(error));\n console.warn(`📧 Email attempt ${attempt} failed:`, lastError.message);\n \n if (attempt < maxRetries) {\n // Wait before retrying (exponential backoff)\n const delay = Math.pow(2, attempt) * 1000; // 2s, 4s, 8s\n console.log(`📧 Retrying in ${delay}ms...`);\n await new Promise(resolve => setTimeout(resolve, delay));\n }\n }\n }\n\n // All attempts failed\n throw new Error(`Failed to send email after ${maxRetries} attempts. Last error: ${lastError?.message}`);\n }\n}",[],[176,552],"312a22e6-8ace-11f0-b451-0224a6c84d84","email-delivery-job.cron.ts","8439388e-8a7c-11f0-a88e-0224a6c84d84",94,["D",1757127864107],"/**\n * Scheduled job for sending AI-generated summaries via email\n * This cron job sends periodic email digests with the latest summaries\n * \n * Configure this in Val Town's interface to run at your desired interval\n * Recommended: Daily at 8 AM or weekly on Sundays\n */\n\nimport { EmailDeliveryService } from \"./email-delivery.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\nimport { email } from \"https://esm.town/v/std/email\";\n\nexport default async function (): Promise {\n const startTime = Date.now();\n console.log(\"📧 Starting scheduled email delivery job...\");\n\n try {\n // Initialize database\n await initializeDatabase();\n\n // Get configuration from environment variables\n const recipientEmail = Deno.env.get(\"USER_EMAIL\");\n const emailFrequency = Deno.env.get(\"EMAIL_DELIVERY_FREQUENCY\") || \"daily\";\n const maxSummaries = parseInt(Deno.env.get(\"EMAIL_MAX_SUMMARIES\") || \"10\");\n const emailEnabled = Deno.env.get(\"EMAIL_DELIVERY_ENABLED\") !== \"false\";\n\n if (!emailEnabled) {\n console.log(\"📧 Email delivery is disabled via EMAIL_DELIVERY_ENABLED=false\");\n return;\n }\n\n if (!recipientEmail) {\n const errorMsg = \"📧 No recipient email configured. Set USER_EMAIL environment variable.\";\n console.error(errorMsg);\n throw new Error(errorMsg);\n }\n\n console.log(\"📧 Email delivery configuration:\", {\n recipient: recipientEmail,\n frequency: emailFrequency,\n maxSummaries,\n });\n\n // Initialize email service\n const emailService = new EmailDeliveryService();\n\n // Generate appropriate subject based on frequency\n const subjects = {\n daily: \"Your Daily Twitter Summary\",\n weekly: \"Your Weekly Twitter Digest\", \n hourly: \"Your Twitter Update\",\n };\n const subject = subjects[emailFrequency as keyof typeof subjects] || \"Your Twitter Feed Summary\";\n\n // Send email with summaries\n const result = await emailService.sendLatestSummaries({\n recipientEmail,\n subject,\n maxSummaries,\n includeMetrics: true,\n });\n\n // Calculate execution time\n const executionTime = Date.now() - startTime;\n\n // Log results\n console.log(\"📧 Email delivery job completed:\");\n console.log(` - Success: ${result.success}`);\n console.log(` - Summaries sent: ${result.sentCount}`);\n console.log(` - Execution time: ${executionTime}ms`);\n\n if (!result.success) {\n console.error(` - Error: ${result.error}`);\n }\n\n // Send notification email about the job status if configured\n const sendJobNotifications = Deno.env.get(\"EMAIL_JOB_NOTIFICATIONS\") === \"true\";\n const adminEmail = Deno.env.get(\"ADMIN_EMAIL\") || recipientEmail;\n\n if (sendJobNotifications && adminEmail && (!result.success || result.sentCount === 0)) {\n try {\n await sendJobNotificationEmail(result, executionTime, adminEmail);\n console.log(\"📧 Job notification email sent successfully\");\n } catch (emailError) {\n console.error(\"📧 Failed to send job notification email:\", emailError);\n }\n }\n\n if (result.success && result.sentCount > 0) {\n console.log(\"✅ Email delivery job completed successfully\");\n } else if (result.sentCount === 0) {\n console.log(\"ℹ️ Email delivery job completed - no new summaries to send\");\n } else {\n throw new Error(result.error || \"Unknown email delivery error\");\n }\n\n } catch (error: unknown) {\n const executionTime = Date.now() - startTime;\n const errorMessage = `Email delivery job failed: ${error}`;\n \n console.error(\"❌ Email delivery job failed:\", error);\n\n // Try to send error notification email\n const sendJobNotifications = Deno.env.get(\"EMAIL_JOB_NOTIFICATIONS\") === \"true\";\n const adminEmail = Deno.env.get(\"ADMIN_EMAIL\") || Deno.env.get(\"USER_EMAIL\");\n\n if (sendJobNotifications && adminEmail) {\n try {\n await sendJobErrorEmail(error, executionTime, adminEmail);\n console.log(\"📧 Error notification email sent\");\n } catch (emailError) {\n console.error(\"📧 Failed to send error notification email:\", emailError);\n }\n }\n\n throw error; // Re-throw to mark job as failed\n }\n}\n\n/**\n * Send job notification email for issues\n */\nasync function sendJobNotificationEmail(\n result: any,\n executionTime: number,\n recipientEmail: string,\n): Promise {\n const subject = result.success \n ? \"OffX: Email Delivery - No New Summaries\"\n : \"OffX: Email Delivery Failed\";\n \n const html = `\n

\n 📧 Email Delivery Job Report\n

\n \n

📊 Job Status

Success: ${result.success ? '✅ Yes' : '❌ No'}
Summaries sent: ${result.sentCount}
Execution time: ${(executionTime / 1000).toFixed(2)}s
Error: ${result.error}

\n\n ${result.sentCount === 0 ? `\n

ℹ️ No New Summaries

There were no new summaries to send since the last email delivery.

This might be normal if:

No new Twitter data has been processed recently
No new summarizations have been generated
The email frequency is set too high

\n ` : ''}\n\n

Generated at ${new Date().toISOString()}

This is an automated report from your Email Delivery service.

To disable these notifications, set EMAIL_JOB_NOTIFICATIONS=false

\n `;\n\n const text = `\nEmail Delivery Job Report\n\nSuccess: ${result.success ? 'Yes' : 'No'}\nSummaries sent: ${result.sentCount}\nExecution time: ${(executionTime / 1000).toFixed(2)}s\n${result.error ? `Error: ${result.error}` : ''}\n\n${result.sentCount === 0 ? `\nNo new summaries were available to send.\nThis might be normal if no new content has been processed recently.\n` : ''}\n\nGenerated at ${new Date().toISOString()}\n `;\n\n await email({\n to: recipientEmail,\n subject,\n html,\n text,\n });\n}\n\n/**\n * Send error notification email\n */\nasync function sendJobErrorEmail(\n error: any,\n executionTime: number,\n recipientEmail: string,\n): Promise {\n const subject = \"OffX: Email Delivery Job Failed\";\n \n const html = `\n

❌ Email Delivery Job Failed

\n \n

Error Details

Error: ${error instanceof Error ? error.message : String(error) || error}

Execution time: ${(executionTime / 1000).toFixed(2)}s

Timestamp: ${new Date().toISOString()}

\n\n

Please check the logs for more details and consider investigating the issue.

Common causes might include:

Database connectivity issues
Email service problems
Configuration errors
Network failures

\n `;\n\n const text = `\nEmail Delivery Job Failed\n\nError: ${error instanceof Error ? error.message : String(error) || error}\nExecution time: ${(executionTime / 1000).toFixed(2)}s\nTimestamp: ${new Date().toISOString()}\n\nPlease check the logs for more details and consider investigating the issue.\n `;\n\n await email({\n to: recipientEmail,\n subject,\n html,\n text,\n });\n}",[],[176,561],"847d6d38-8a7c-11f0-a12e-0224a6c84d84","email-delivery-api.http.ts","847d6e1e-8a7c-11f0-a12e-0224a6c84d84",58,["D",1757092785181],"/**\n * HTTP endpoint for manual email delivery testing and triggering\n * Provides a way to test email delivery and manually send summaries\n */\n\nimport { EmailDeliveryService } from \"./email-delivery.ts\";\nimport { initializeDatabase } from \"./database/migrations.ts\";\n\nexport default async function (req: Request): Promise {\n // Set CORS headers for all responses\n const headers = new Headers({\n \"Access-Control-Allow-Origin\": \"*\",\n \"Access-Control-Allow-Methods\": \"GET, POST, OPTIONS\",\n \"Access-Control-Allow-Headers\": \"Content-Type\",\n \"Content-Type\": \"application/json\",\n });\n\n // Handle preflight OPTIONS request\n if (req.method === \"OPTIONS\") {\n return new Response(null, { headers, status: 204 });\n }\n\n try {\n const url = new URL(req.url);\n const action = url.searchParams.get(\"action\") || \"status\";\n\n // Initialize database\n await initializeDatabase();\n\n let responseData: any = {\n status: \"ok\",\n timestamp: new Date().toISOString(),\n };\n\n const emailService = new EmailDeliveryService();\n\n if (action === \"send\") {\n // Send summary email\n console.log(\"📧 Manual email send requested...\");\n \n const recipientEmail = url.searchParams.get(\"email\") || Deno.env.get(\"USER_EMAIL\");\n const subject = url.searchParams.get(\"subject\") || undefined;\n const maxSummaries = parseInt(url.searchParams.get(\"maxSummaries\") || \"10\");\n const includeMetrics = url.searchParams.get(\"includeMetrics\") !== \"false\";\n\n if (!recipientEmail) {\n return new Response(JSON.stringify({\n status: \"error\",\n message: \"No recipient email provided. Set ?email=your@email.com or USER_EMAIL environment variable.\",\n }), { headers, status: 400 });\n }\n\n const startTime = Date.now();\n const result = await emailService.sendLatestSummaries({\n recipientEmail,\n subject,\n maxSummaries,\n includeMetrics,\n });\n\n const executionTime = Date.now() - startTime;\n\n responseData = {\n ...responseData,\n action: \"send\",\n success: result.success,\n message: result.success \n ? `Email sent successfully with ${result.sentCount} summaries`\n : `Email sending failed: ${result.error}`,\n details: {\n recipientEmail,\n sentCount: result.sentCount,\n executionTimeMs: executionTime,\n error: result.error,\n },\n };\n\n } else if (action === \"test\") {\n // Test email service without actually sending\n console.log(\"🧪 Testing email service...\");\n \n const maxSummaries = parseInt(url.searchParams.get(\"maxSummaries\") || \"5\");\n\n // This is a dry run - we prepare the email but don't send it\n const testEmail = \"test@example.com\";\n \n try {\n // We'll simulate the email preparation process\n const summaries = await (emailService as any).fetchRecentSummaries(maxSummaries);\n \n if (summaries.length === 0) {\n responseData = {\n ...responseData,\n action: \"test\",\n status: \"no_data\",\n message: \"No recent summaries found for email testing\",\n summaryCount: 0,\n };\n } else {\n const emailContent = (emailService as any).generateEmailTemplate(summaries, true);\n \n responseData = {\n ...responseData,\n action: \"test\",\n status: \"success\",\n message: \"Email content generated successfully (not sent)\",\n details: {\n summaryCount: summaries.length,\n emailHtmlLength: emailContent.html.length,\n emailTextLength: emailContent.text.length,\n summaryTitles: summaries.map((s: any) => s.topicTitle),\n },\n };\n }\n } catch (error: unknown) {\n responseData = {\n ...responseData,\n action: \"test\",\n status: \"error\",\n message: `Email test failed: ${error}`,\n };\n }\n\n } else if (action === \"config\") {\n // Show email configuration\n responseData = {\n ...responseData,\n action: \"config\",\n configuration: {\n defaultRecipient: Deno.env.get(\"USER_EMAIL\") || \"Not configured\",\n maxRetries: 3,\n retryDelays: \"2s, 4s, 8s (exponential backoff)\",\n emailProvider: \"Val Town email service\",\n featuresEnabled: {\n htmlEmail: true,\n textFallback: true,\n sentimentAnalysis: true,\n keywordExtraction: true,\n metrics: true,\n },\n },\n };\n\n } else {\n // Default status action\n responseData = {\n ...responseData,\n action: \"status\",\n message: \"Email delivery service is running\",\n availableActions: [\n \"send - Send summary email (add ?email=recipient@example.com)\",\n \"test - Test email generation without sending (add ?maxSummaries=N)\",\n \"config - View email service configuration\",\n ],\n environment: {\n userEmailConfigured: !!Deno.env.get(\"USER_EMAIL\"),\n emailServiceAvailable: true,\n },\n };\n }\n\n return new Response(JSON.stringify(responseData, null, 2), { headers });\n\n } catch (error: unknown) {\n console.error(\"Error processing email delivery request:\", error);\n\n return new Response(\n JSON.stringify({\n status: \"error\",\n message: error instanceof Error ? error instanceof Error ? error.message : String(error) : String(error),\n timestamp: new Date().toISOString(),\n }, null, 2),\n { headers, status: 500 },\n );\n }\n}",[],[176,569],"846d3cec-8a7c-11f0-93cc-0224a6c84d84","SCHEDULER_SETUP.md","846d3db4-8a7c-11f0-93cc-0224a6c84d84",56,["D",1757092785074],"# Twitter Data Ingestion Scheduler Setup\n\nThis document explains how to set up the scheduler for the Twitter data ingestion process in Val Town.\n\n## Overview\n\nThe Twitter data ingestion process is implemented as a cron job in Val Town. This job fetches tweets from the Twitter API, processes them through our filtering pipeline, and stores them in the database. The scheduler runs this job at regular intervals to ensure timely and consistent data updates.\n\n## Val Town Cron Configuration\n\nAs per the [Val Town documentation](https://docs.val.town/quickstarts/first-cron/), cron schedules cannot be configured programmatically via code. Instead, they must be set up through the Val Town web GUI.\n\n### Steps to Configure the Scheduler\n\n1. **Upload the Cron File**:\n - Ensure the `twitter-fetch.cron.ts` file is uploaded to your Val Town account\n - The file should have the `.cron.ts` extension to indicate it's a cron job\n\n2. **Configure Schedule in Val Town Web GUI**:\n - Log in to your Val Town account\n - Navigate to the `twitter-fetch.cron.ts` val\n - Look for the \"Schedule\" or \"Cron\" settings in the val's interface\n - Set the desired schedule using cron syntax (e.g., `0 */3 * * *` for every 3 hours)\n\n3. **Recommended Schedule**:\n - For production use, we recommend running the job every 1-3 hours (`0 */1 * * *` to `0 */3 * * *`)\n - For testing or development, you might want to run it more frequently\n - Consider Twitter API rate limits when setting the schedule\n\n## Environment Variables\n\nThe cron job requires the following environment variables to be set in your Val Town environment:\n\n- `TWITTER_BEARER_TOKEN`: Your Twitter API bearer token\n- `EMAIL_TO` (optional): Email address to send notifications to\n- `AUTH_USERNAME` and `AUTH_PASSWORD`: Credentials for accessing the test endpoints\n- `TWITTER_FILTER_*` (optional): Various filter configuration options\n\n## Monitoring and Logs\n\n- The cron job logs its activity to the Val Town console\n- If configured, it sends email notifications about successful runs and errors\n- You can view the logs in the Val Town interface for the val\n\n## Troubleshooting\n\n- If the job fails due to rate limiting, it will automatically retry with exponential backoff\n- Error emails include detailed information about the failure\n- Check the Val Town logs for more detailed error messages\n\n## Manual Triggering\n\nFor testing or one-off runs, you can manually trigger the cron job using the test endpoint:\n\n```\nhttps://your-val-town-username.web.val.run/twitter-test\n```\n\nThis endpoint requires basic authentication using the `AUTH_USERNAME` and `AUTH_PASSWORD` environment variables. ",[],[176,577],"842d73a0-8a7c-11f0-81bc-0224a6c84d84","input-sanitizer.ts","842d7472-8a7c-11f0-81bc-0224a6c84d84","83c567a6-8a7c-11f0-85b0-0224a6c84d84",47,["D",1757092784657],"/**\n * Basic input sanitization utility for single-user application\n * Provides minimal sanitization to prevent common XSS attacks\n */\n\n/**\n * Simple input sanitization function that removes dangerous characters\n * @param input - The string to sanitize\n * @returns Sanitized string with dangerous characters removed\n */\nexport function sanitizeInput(input: string): string {\n if (typeof input !== 'string') {\n return '';\n }\n \n // Remove the most dangerous characters that could be used for XSS\n // Keep it simple - just remove < and > characters\n return input.replace(/[<>]/g, '');\n}\n\n/**\n * Sanitize an object's string properties\n * @param obj - Object with string properties to sanitize\n * @returns New object with sanitized string values\n */\nexport function sanitizeObject>(obj: T): T {\n const sanitized = { ...obj };\n \n for (const key in sanitized) {\n if (typeof sanitized[key] === 'string') {\n sanitized[key] = sanitizeInput(sanitized[key]) as T[Extract];\n }\n }\n \n return sanitized;\n}\n\n/**\n * Middleware for Hono to automatically sanitize request body\n */\nexport const sanitizeMiddleware = async (c: any, next: any) => {\n // Only sanitize POST/PUT requests with JSON body\n if (c.req.method === 'POST' || c.req.method === 'PUT') {\n try {\n const contentType = c.req.header('content-type');\n if (contentType && contentType.includes('application/json')) {\n const originalJson = c.req.json;\n c.req.json = async () => {\n const body = await originalJson.call(c.req);\n return sanitizeObject(body);\n };\n }\n } catch (error: unknown) {\n // If there's an error, continue without sanitization\n console.warn('Input sanitization failed:', error);\n }\n }\n \n await next();\n};",[],[176,593,585],"security","83c566ac-8a7c-11f0-85b0-0224a6c84d84",33,["D",1757092783975],[216],[176,593],["D",1757092785287],[],[176,93,66],["D",1757092785379],[],[176,93,81],["D",1757092785321],[],[176,93,91],["D",1751317131604],[212,213,214],[176,93],"83f066f4-8a7c-11f0-bfd7-0224a6c84d84","run-benchmark.ts","83f067c6-8a7c-11f0-bfd7-0224a6c84d84","06a4dc18-55f5-11f0-9d32-f69ea79377d9",37,["D",1757092784200],"import { runBenchmarks } from \"./benchmark.ts\";\n\n/**\n * This script runs the database benchmarks and outputs the results\n * Run with: deno run --allow-net backend/database/run-benchmark.ts\n */\n\nconsole.log(\"Starting database performance benchmarking...\");\nconsole.log(\n \"This will create test data in the database to measure performance.\",\n);\nconsole.log(\"Press Ctrl+C to cancel or wait for benchmarks to complete.\");\n\n// Small delay to allow user to cancel if needed\nsetTimeout(async () => {\n try {\n await runBenchmarks();\n } catch (error: unknown) {\n console.error(\"Benchmark failed:\", error);\n }\n}, 2000);\n",[],[176,620,612],"database","854fe786-8a7c-11f0-85b0-0224a6c84d84","queries.ts","854fe858-8a7c-11f0-85b0-0224a6c84d84",75,["D",1757092784273],"import { sqlite } from \"https://esm.town/v/stevekrouse/sqlite\";\nimport { TABLES } from \"./migrations.ts\";\n\n// Type definitions\nexport interface Topic {\n id?: number;\n title: string;\n description?: string;\n created_at?: string;\n updated_at?: string;\n}\n\nexport interface Post {\n id?: number;\n external_id: string;\n content: string;\n author: string;\n author_username: string;\n platform: string;\n created_at: string;\n fetched_at?: string;\n url?: string;\n media?: PostMedia[];\n}\n\nexport interface PostMedia {\n id?: number;\n post_id: number;\n media_url: string;\n media_type: string;\n alt_text?: string;\n created_at?: string;\n}\n\nexport interface Summarization {\n id?: number;\n topic_id: number;\n content: string;\n prompt: string;\n post_count: number;\n created_at?: string;\n time_period_start: string;\n time_period_end: string;\n time_period?: string; // For compatibility with existing code\n}\n\nexport interface TopicPost {\n topic_id: number;\n post_id: number;\n added_at?: string;\n}\n\nexport interface PostSentiment {\n id?: number;\n post_id: number;\n sentiment: \"positive\" | \"negative\" | \"neutral\";\n score: number; // -1.0 to 1.0\n confidence: number; // 0.0 to 1.0\n keywords: string[];\n entities?: string[];\n analyzed_at?: string;\n}\n\n// Topic operations\nexport async function createTopic(topic: Topic): Promise {\n const result = await sqlite.execute(\n `INSERT INTO ${TABLES.TOPICS} (title, description) VALUES (?, ?)`,\n [topic.title, topic.description || null],\n );\n return Number(result.lastInsertRowid) || 0;\n}\n\nexport async function getTopicById(id: number): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.TOPICS} WHERE id = ?`,\n [id],\n );\n return result.rows.length > 0 ? result.rows[0] as unknown as Topic : null;\n}\n\nexport async function getAllTopics(): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.TOPICS} ORDER BY title`,\n );\n return result.rows as unknown as Topic[];\n}\n\nexport async function updateTopic(topic: Topic): Promise {\n if (!topic.id) return false;\n\n await sqlite.execute(\n `UPDATE ${TABLES.TOPICS} SET title = ?, description = ?, updated_at = CURRENT_TIMESTAMP WHERE id = ?`,\n [topic.title, topic.description || null, topic.id],\n );\n return true;\n}\n\nexport async function deleteTopic(id: number): Promise {\n await sqlite.execute(`DELETE FROM ${TABLES.TOPICS} WHERE id = ?`, [id]);\n return true;\n}\n\n// Post operations\nexport async function createPost(post: Post): Promise {\n const result = await sqlite.execute(\n `INSERT INTO ${TABLES.POSTS} \n (external_id, content, author, author_username, platform, created_at, url) \n VALUES (?, ?, ?, ?, ?, ?, ?)`,\n [\n post.external_id,\n post.content,\n post.author,\n post.author_username,\n post.platform,\n post.created_at,\n post.url || null,\n ],\n );\n\n const postId = Number(result.lastInsertRowid) || 0;\n\n // Insert media if provided\n if (post.media && post.media.length > 0) {\n for (const media of post.media) {\n await createPostMedia({\n ...media,\n post_id: postId,\n });\n }\n }\n\n return postId;\n}\n\nexport async function getPostById(id: number): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.POSTS} WHERE id = ?`,\n [id],\n );\n\n if (result.rows.length === 0) return null;\n\n const post = result.rows[0] as unknown as Post;\n post.media = await getMediaForPost(post.id!);\n\n return post;\n}\n\nexport async function getPostByExternalId(\n externalId: string,\n): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.POSTS} WHERE external_id = ?`,\n [externalId],\n );\n\n if (result.rows.length === 0) return null;\n\n const post = result.rows[0] as unknown as Post;\n post.media = await getMediaForPost(post.id!);\n\n return post;\n}\n\nexport async function getRecentPosts(limit: number = 100): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.POSTS} ORDER BY created_at DESC LIMIT ?`,\n [limit],\n );\n\n const posts = result.rows as unknown as Post[];\n\n // Get media for each post\n for (const post of posts) {\n post.media = await getMediaForPost(post.id!);\n }\n\n return posts;\n}\n\nexport async function deletePost(id: number): Promise {\n await sqlite.execute(`DELETE FROM ${TABLES.POSTS} WHERE id = ?`, [id]);\n return true;\n}\n\n// Post Media operations\nexport async function createPostMedia(media: PostMedia): Promise {\n const result = await sqlite.execute(\n `INSERT INTO ${TABLES.POST_MEDIA} \n (post_id, media_url, media_type, alt_text) \n VALUES (?, ?, ?, ?)`,\n [\n media.post_id,\n media.media_url,\n media.media_type,\n media.alt_text || null,\n ],\n );\n\n return Number(result.lastInsertRowid) || 0;\n}\n\nexport async function getMediaForPost(postId: number): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.POST_MEDIA} WHERE post_id = ?`,\n [postId],\n );\n\n return result.rows as unknown as PostMedia[];\n}\n\n/**\n * Get media items for a post by post ID (alias for getMediaForPost)\n * @param postId ID of the post\n * @returns Array of media items for the post\n */\nexport async function getPostMediaByPostId(\n postId: number,\n): Promise {\n return getMediaForPost(postId);\n}\n\n// Topic-Post relationship operations\nexport async function addPostToTopic(\n topicId: number,\n postId: number,\n): Promise {\n try {\n await sqlite.execute(\n `INSERT INTO ${TABLES.TOPIC_POSTS} (topic_id, post_id) VALUES (?, ?)`,\n [topicId, postId],\n );\n return true;\n } catch (error: unknown) {\n // Likely a duplicate entry, which is fine\n return false;\n }\n}\n\nexport async function removePostFromTopic(\n topicId: number,\n postId: number,\n): Promise {\n await sqlite.execute(\n `DELETE FROM ${TABLES.TOPIC_POSTS} WHERE topic_id = ? AND post_id = ?`,\n [topicId, postId],\n );\n return true;\n}\n\nexport async function getPostsForTopic(\n topicId: number,\n limit: number = 100,\n): Promise {\n const result = await sqlite.execute(\n `SELECT p.* FROM ${TABLES.POSTS} p\n JOIN ${TABLES.TOPIC_POSTS} tp ON p.id = tp.post_id\n WHERE tp.topic_id = ?\n ORDER BY p.created_at DESC\n LIMIT ?`,\n [topicId, limit],\n );\n\n const posts = result.rows as unknown as Post[];\n\n // Get media for each post\n for (const post of posts) {\n post.media = await getMediaForPost(post.id!);\n }\n\n return posts;\n}\n\n// Summarization operations\nexport async function createSummarization(\n summarization: Summarization,\n): Promise {\n const result = await sqlite.execute(\n `INSERT INTO ${TABLES.SUMMARIZATIONS} \n (topic_id, content, prompt, post_count, time_period_start, time_period_end) \n VALUES (?, ?, ?, ?, ?, ?)`,\n [\n summarization.topic_id,\n summarization.content,\n summarization.prompt,\n summarization.post_count,\n summarization.time_period_start,\n summarization.time_period_end,\n ],\n );\n\n return Number(result.lastInsertRowid) || 0;\n}\n\nexport async function getSummarizationById(\n id: number,\n): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.SUMMARIZATIONS} WHERE id = ?`,\n [id],\n );\n\n return result.rows.length > 0 ? result.rows[0] as unknown as Summarization : null;\n}\n\nexport async function getSummarizationsForTopic(\n topicId: number,\n limit: number = 10,\n): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.SUMMARIZATIONS} \n WHERE topic_id = ? \n ORDER BY created_at DESC \n LIMIT ?`,\n [topicId, limit],\n );\n\n return result.rows as unknown as Summarization[];\n}\n\nexport async function getRecentSummarizations(\n limit: number = 10,\n): Promise {\n const result = await sqlite.execute(\n `SELECT s.*, t.title as topic_title \n FROM ${TABLES.SUMMARIZATIONS} s\n JOIN ${TABLES.TOPICS} t ON s.topic_id = t.id\n ORDER BY s.created_at DESC \n LIMIT ?`,\n [limit],\n );\n\n return result.rows as unknown as (Summarization & { topic_title: string })[];\n}\n\nexport async function deleteSummarization(id: number): Promise {\n await sqlite.execute(\n `DELETE FROM ${TABLES.SUMMARIZATIONS} WHERE id = ?`,\n [id],\n );\n return true;\n}\n\n// Sentiment analysis operations\nexport async function createPostSentiment(\n sentiment: PostSentiment,\n): Promise {\n const result = await sqlite.execute(\n `INSERT INTO ${TABLES.POST_SENTIMENT} \n (post_id, sentiment, score, confidence, keywords, entities) \n VALUES (?, ?, ?, ?, ?, ?)`,\n [\n sentiment.post_id,\n sentiment.sentiment,\n sentiment.score,\n sentiment.confidence,\n JSON.stringify(sentiment.keywords),\n sentiment.entities ? JSON.stringify(sentiment.entities) : null,\n ],\n );\n\n return Number(result.lastInsertRowid) || 0;\n}\n\nexport async function getPostSentiment(\n postId: number,\n): Promise {\n const result = await sqlite.execute(\n `SELECT * FROM ${TABLES.POST_SENTIMENT} WHERE post_id = ?`,\n [postId],\n );\n\n if (result.rows.length === 0) return null;\n\n const sentiment = result.rows[0] as any;\n\n // Parse JSON strings back to arrays\n return {\n ...sentiment,\n keywords: JSON.parse(sentiment.keywords || \"[]\"),\n entities: sentiment.entities ? JSON.parse(sentiment.entities) : undefined,\n };\n}\n\nexport async function updatePostSentiment(\n sentiment: PostSentiment,\n): Promise {\n await sqlite.execute(\n `UPDATE ${TABLES.POST_SENTIMENT} \n SET sentiment = ?, score = ?, confidence = ?, keywords = ?, entities = ?, analyzed_at = CURRENT_TIMESTAMP\n WHERE post_id = ?`,\n [\n sentiment.sentiment,\n sentiment.score,\n sentiment.confidence,\n JSON.stringify(sentiment.keywords),\n sentiment.entities ? JSON.stringify(sentiment.entities) : null,\n sentiment.post_id,\n ],\n );\n\n return true;\n}\n\nexport async function createOrUpdatePostSentiment(\n sentiment: PostSentiment,\n): Promise {\n // Check if sentiment analysis already exists for this post\n const existingSentiment = await getPostSentiment(sentiment.post_id);\n\n if (existingSentiment) {\n await updatePostSentiment(sentiment);\n return existingSentiment.id!;\n } else {\n return await createPostSentiment(sentiment);\n }\n}\n\nexport async function getPostsWithSentiment(\n sentiment?: \"positive\" | \"negative\" | \"neutral\",\n minConfidence: number = 0.7,\n limit: number = 100,\n): Promise {\n let query = `\n SELECT p.* FROM ${TABLES.POSTS} p\n JOIN ${TABLES.POST_SENTIMENT} ps ON p.id = ps.post_id\n WHERE ps.confidence >= ?\n `;\n\n const params: any[] = [minConfidence];\n\n if (sentiment) {\n query += ` AND ps.sentiment = ?`;\n params.push(sentiment);\n }\n\n query += ` ORDER BY p.created_at DESC LIMIT ?`;\n params.push(limit);\n\n const result = await sqlite.execute(query, params);\n\n const posts = result.rows as unknown as Post[];\n\n // Get media for each post\n for (const post of posts) {\n post.media = await getMediaForPost(post.id!);\n }\n\n return posts;\n}\n\nexport async function getPostsWithoutSentimentAnalysis(\n limit: number = 100,\n): Promise {\n const result = await sqlite.execute(\n `SELECT p.* FROM ${TABLES.POSTS} p\n LEFT JOIN ${TABLES.POST_SENTIMENT} ps ON p.id = ps.post_id\n WHERE ps.id IS NULL\n ORDER BY p.created_at DESC\n LIMIT ?`,\n [limit],\n );\n\n const posts = result.rows as unknown as Post[];\n\n // Get media for each post\n for (const post of posts) {\n post.media = await getMediaForPost(post.id!);\n }\n\n return posts;\n}\n\nexport async function getSentimentStats(): Promise<{\n positive: number;\n negative: number;\n neutral: number;\n total: number;\n avgScore: number;\n}> {\n const result = await sqlite.execute(`\n SELECT \n COUNT(*) as total,\n SUM(CASE WHEN sentiment = 'positive' THEN 1 ELSE 0 END) as positive,\n SUM(CASE WHEN sentiment = 'negative' THEN 1 ELSE 0 END) as negative,\n SUM(CASE WHEN sentiment = 'neutral' THEN 1 ELSE 0 END) as neutral,\n AVG(score) as avgScore\n FROM ${TABLES.POST_SENTIMENT}\n `);\n\n if (result.rows.length === 0) {\n return {\n positive: 0,\n negative: 0,\n neutral: 0,\n total: 0,\n avgScore: 0,\n };\n }\n\n return result.rows[0] as unknown as any;\n}\n",[],[176,620,622],"840547ea-8a7c-11f0-81bc-0224a6c84d84","migrations.ts","8405489e-8a7c-11f0-81bc-0224a6c84d84",40,["D",1757092784357],"import { sqlite } from \"https://esm.town/v/stevekrouse/sqlite\";\n\n// Table names - use versioning to handle schema changes\nexport const TABLES = {\n TOPICS: \"topics_v1\",\n POSTS: \"posts_v1\",\n POST_MEDIA: \"post_media_v1\",\n SUMMARIZATIONS: \"summarizations_v1\",\n TOPIC_POSTS: \"topic_posts_v1\", // Junction table for many-to-many relationship\n POST_SENTIMENT: \"post_sentiment_v1\", // New table for sentiment analysis\n};\n\n/**\n * Creates all database tables if they don't exist\n */\nexport async function createTables() {\n // Create topics table\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLES.TOPICS} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n title TEXT NOT NULL,\n description TEXT,\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP\n )\n `);\n\n // Create posts table (for tweets or other social media content)\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLES.POSTS} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n external_id TEXT UNIQUE, -- Twitter/X tweet ID or other platform ID\n content TEXT NOT NULL,\n author TEXT NOT NULL,\n author_username TEXT NOT NULL,\n platform TEXT NOT NULL, -- 'twitter', 'mastodon', etc.\n created_at TIMESTAMP NOT NULL,\n fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n url TEXT\n )\n `);\n\n // Create post_media table for media attached to posts\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLES.POST_MEDIA} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n post_id INTEGER NOT NULL,\n media_url TEXT NOT NULL,\n media_type TEXT NOT NULL, -- 'image', 'video', etc.\n alt_text TEXT,\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n FOREIGN KEY (post_id) REFERENCES ${TABLES.POSTS}(id) ON DELETE CASCADE\n )\n `);\n\n // Create junction table for many-to-many relationship between topics and posts\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLES.TOPIC_POSTS} (\n topic_id INTEGER NOT NULL,\n post_id INTEGER NOT NULL,\n added_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n PRIMARY KEY (topic_id, post_id),\n FOREIGN KEY (topic_id) REFERENCES ${TABLES.TOPICS}(id) ON DELETE CASCADE,\n FOREIGN KEY (post_id) REFERENCES ${TABLES.POSTS}(id) ON DELETE CASCADE\n )\n `);\n\n // Create summarizations table\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLES.SUMMARIZATIONS} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n topic_id INTEGER NOT NULL,\n content TEXT NOT NULL,\n prompt TEXT NOT NULL, -- Store the prompt used to generate the summary\n post_count INTEGER NOT NULL, -- Number of posts included in this summary\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n time_period_start TIMESTAMP NOT NULL, -- Start of the time period covered\n time_period_end TIMESTAMP NOT NULL, -- End of the time period covered\n FOREIGN KEY (topic_id) REFERENCES ${TABLES.TOPICS}(id) ON DELETE CASCADE\n )\n `);\n\n // Create post sentiment table\n await sqlite.execute(`\n CREATE TABLE IF NOT EXISTS ${TABLES.POST_SENTIMENT} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n post_id INTEGER NOT NULL,\n sentiment TEXT NOT NULL, -- 'positive', 'negative', 'neutral'\n score REAL NOT NULL, -- -1.0 to 1.0\n confidence REAL NOT NULL, -- 0.0 to 1.0\n keywords TEXT, -- JSON array of keywords\n entities TEXT, -- JSON array of entities (optional)\n analyzed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n FOREIGN KEY (post_id) REFERENCES ${TABLES.POSTS}(id) ON DELETE CASCADE,\n UNIQUE(post_id) -- Only one sentiment analysis per post\n )\n `);\n\n // Create indexes for performance\n await createIndexes();\n}\n\n/**\n * Creates indexes for optimizing query performance\n */\nasync function createIndexes() {\n // Indexes for posts table\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_external_id ON ${TABLES.POSTS}(external_id)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_author ON ${TABLES.POSTS}(author)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_created_at ON ${TABLES.POSTS}(created_at)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_platform ON ${TABLES.POSTS}(platform)`,\n );\n\n // Indexes for topic_posts junction table\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_topic_posts_topic_id ON ${TABLES.TOPIC_POSTS}(topic_id)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_topic_posts_post_id ON ${TABLES.TOPIC_POSTS}(post_id)`,\n );\n\n // Indexes for summarizations table\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_summarizations_topic_id ON ${TABLES.SUMMARIZATIONS}(topic_id)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_summarizations_created_at ON ${TABLES.SUMMARIZATIONS}(created_at)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_summarizations_time_period ON ${TABLES.SUMMARIZATIONS}(time_period_start, time_period_end)`,\n );\n\n // Indexes for post sentiment table\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_post_sentiment_post_id ON ${TABLES.POST_SENTIMENT}(post_id)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_post_sentiment_sentiment ON ${TABLES.POST_SENTIMENT}(sentiment)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_post_sentiment_score ON ${TABLES.POST_SENTIMENT}(score)`,\n );\n}\n\n/**\n * Initialize the database with all required tables and indexes\n */\nexport async function initializeDatabase() {\n try {\n await createTables();\n console.log(\"Database initialized successfully\");\n return true;\n } catch (error: unknown) {\n console.error(\"Error initializing database:\", error);\n throw error;\n }\n}\n\n/**\n * Schema Change Utilities\n *\n * Val Town's SQLite implementation has limited support for ALTER TABLE operations.\n * Instead, we follow the recommended approach of creating new tables with updated\n * schemas and migrating data from old tables.\n */\n\n/**\n * Check if a table exists in the database\n * @param tableName The name of the table to check\n * @returns True if the table exists, false otherwise\n */\nexport async function tableExists(tableName: string): Promise {\n const result = await sqlite.execute(\n `SELECT name FROM sqlite_master WHERE type='table' AND name=?`,\n [tableName],\n );\n return result.rows.length > 0;\n}\n\n/**\n * Get the column information for a table\n * @param tableName The name of the table to get column info for\n * @returns Array of column information objects\n */\nexport async function getTableColumns(\n tableName: string,\n): Promise<{ name: string; type: string }[]> {\n const result = await sqlite.execute(`PRAGMA table_info(${tableName})`);\n return result.rows.map((row: any) => ({\n name: row.name,\n type: row.type,\n }));\n}\n\n/**\n * Create a new table version with an updated schema\n * @param oldTableName The name of the old table\n * @param newTableName The name of the new table\n * @param createTableSql The SQL statement to create the new table\n * @param columnMappings Mapping of old column names to new column names\n */\nexport async function migrateTable(\n oldTableName: string,\n newTableName: string,\n createTableSql: string,\n columnMappings: { [oldCol: string]: string } = {},\n): Promise {\n // Check if old table exists\n const oldTableExists = await tableExists(oldTableName);\n if (!oldTableExists) {\n console.log(\n `Old table ${oldTableName} does not exist. Creating new table ${newTableName} without migration.`,\n );\n await sqlite.execute(createTableSql);\n return;\n }\n\n // Create the new table\n await sqlite.execute(createTableSql);\n\n // Get columns from old table\n const oldColumns = await getTableColumns(oldTableName);\n const oldColumnNames = oldColumns.map((col) => col.name);\n\n // Get columns from new table\n const newColumns = await getTableColumns(newTableName);\n const newColumnNames = newColumns.map((col) => col.name);\n\n // Build column mapping for the migration\n const columnsToCopy: string[] = [];\n\n oldColumnNames.forEach((oldCol) => {\n const mappedCol = columnMappings[oldCol] || oldCol;\n if (newColumnNames.includes(mappedCol)) {\n columnsToCopy.push(`${oldCol} AS ${mappedCol}`);\n }\n });\n\n if (columnsToCopy.length === 0) {\n console.log(\n `No matching columns between ${oldTableName} and ${newTableName}. Skipping data migration.`,\n );\n return;\n }\n\n // Copy data from old table to new table\n try {\n await sqlite.execute(\n `INSERT INTO ${newTableName} SELECT ${\n columnsToCopy.join(\", \")\n } FROM ${oldTableName}`,\n );\n console.log(\n `Successfully migrated data from ${oldTableName} to ${newTableName}`,\n );\n } catch (error: unknown) {\n console.error(\n `Error migrating data from ${oldTableName} to ${newTableName}:`,\n error,\n );\n throw error;\n }\n}\n\n/**\n * Example schema change function for topics table\n * This shows how to migrate from topics_v1 to topics_v2 with a new column\n */\nexport async function upgradeTopicsToV2(): Promise {\n const oldTable = \"topics_v1\";\n const newTable = \"topics_v2\";\n\n // Define the new table schema\n const createTableSql = `\n CREATE TABLE IF NOT EXISTS ${newTable} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n title TEXT NOT NULL,\n description TEXT,\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n is_active BOOLEAN DEFAULT 1, -- New column added in v2\n color TEXT DEFAULT '#3498db' -- New column added in v2\n )\n `;\n\n // Define column mappings (only needed if column names change)\n const columnMappings = {\n // No renamed columns in this example\n };\n\n // Migrate the table\n await migrateTable(oldTable, newTable, createTableSql, columnMappings);\n\n // Update the TABLES constant to point to the new version\n // Note: In a real application, you would update the constant in the code\n // and redeploy. This is just for demonstration.\n console.log(\"Migration complete. Update the TABLES constant to:\");\n console.log(`TOPICS: \"${newTable}\"`);\n}\n\n/**\n * Example schema change function for posts table\n * This shows how to migrate from posts_v1 to posts_v2 with renamed columns\n */\nexport async function upgradePostsToV2(): Promise {\n const oldTable = \"posts_v1\";\n const newTable = \"posts_v2\";\n\n // Define the new table schema\n const createTableSql = `\n CREATE TABLE IF NOT EXISTS ${newTable} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n external_id TEXT UNIQUE,\n content TEXT NOT NULL,\n author_name TEXT NOT NULL, -- Renamed from 'author'\n author_username TEXT NOT NULL,\n source TEXT NOT NULL, -- Renamed from 'platform'\n published_at TIMESTAMP NOT NULL, -- Renamed from 'created_at'\n fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n url TEXT,\n engagement_count INTEGER DEFAULT 0 -- New column\n )\n `;\n\n // Define column mappings for renamed columns\n const columnMappings = {\n \"author\": \"author_name\",\n \"platform\": \"source\",\n \"created_at\": \"published_at\",\n };\n\n // Migrate the table\n await migrateTable(oldTable, newTable, createTableSql, columnMappings);\n\n // Update the TABLES constant to point to the new version\n console.log(\"Migration complete. Update the TABLES constant to:\");\n console.log(`POSTS: \"${newTable}\"`);\n\n // Create indexes for the new table\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_external_id ON ${newTable}(external_id)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_author_name ON ${newTable}(author_name)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_published_at ON ${newTable}(published_at)`,\n );\n await sqlite.execute(\n `CREATE INDEX IF NOT EXISTS idx_posts_source ON ${newTable}(source)`,\n );\n}\n",[],[176,620,630],"84146bda-8a7c-11f0-a4f1-0224a6c84d84","84146c84-8a7c-11f0-a4f1-0224a6c84d84",42,["D",1757092784417],"import { initializeDatabase, TABLES } from \"./migrations.ts\";\nimport * as queries from \"./queries.ts\";\n\nexport { initializeDatabase, queries, TABLES };\n\n// Re-export interfaces for convenience\nexport type {\n Post,\n PostMedia,\n Summarization,\n Topic,\n TopicPost,\n} from \"./queries.ts\";\n\n// Initialize the database when this module is imported\n// Comment this out in production if you want to initialize manually\n// for better performance in serverless environments\ninitializeDatabase().catch(console.error);\n",[],[176,620,544],"8410d740-8a7c-11f0-af15-0224a6c84d84","benchmark.ts","8410d7fe-8a7c-11f0-af15-0224a6c84d84",41,["D",1757092784458],"import { initializeDatabase } from \"./migrations.ts\";\nimport * as queries from \"./queries.ts\";\nimport { Post, PostMedia } from \"./queries.ts\";\n\n/**\n * Simple benchmarking utility for database operations\n * This helps identify performance bottlenecks in the data access layer\n */\n\n// Helper function to measure execution time\nasync function measureTime(\n name: string,\n fn: () => Promise,\n iterations: number = 1,\n): Promise<{ result: T; avgTime: number }> {\n console.log(`Running benchmark: ${name} (${iterations} iterations)`);\n\n const times: number[] = [];\n let result: T = null as unknown as T;\n\n for (let i = 0; i < iterations; i++) {\n const start = performance.now();\n result = await fn();\n const end = performance.now();\n const time = end - start;\n times.push(time);\n\n if (iterations > 1) {\n console.log(` Iteration ${i + 1}: ${time.toFixed(2)}ms`);\n }\n }\n\n const avgTime = times.reduce((sum, time) => sum + time, 0) / times.length;\n console.log(` Average time: ${avgTime.toFixed(2)}ms`);\n\n return { result, avgTime };\n}\n\n// Sample data generation\nfunction generateSampleTopic(index: number) {\n return {\n title: `Test Topic ${index}`,\n description: `This is a test topic ${index} for benchmarking purposes.`,\n };\n}\n\nfunction generateSamplePost(index: number): Omit {\n return {\n external_id: `test-${index}`,\n content:\n `This is test post content ${index} with some additional text to make it more realistic. #testing #benchmark`,\n author: `Test Author ${index % 10}`,\n author_username: `testuser${index % 10}`,\n platform: index % 2 === 0 ? \"twitter\" : \"mastodon\",\n created_at: new Date(Date.now() - index * 3600000).toISOString(),\n url: `https://example.com/post/${index}`,\n };\n}\n\nfunction generateSampleMedia(\n postId: number,\n index: number,\n): Omit {\n return {\n post_id: postId,\n media_url: `https://example.com/media/${postId}-${index}.jpg`,\n media_type: \"image\",\n alt_text: `Sample image ${index} for post ${postId}`,\n };\n}\n\nfunction generateSampleSummarization(topicId: number, index: number) {\n const endDate = new Date();\n const startDate = new Date(endDate.getTime() - (24 * 3600000)); // 24 hours before\n\n return {\n topic_id: topicId,\n content:\n `This is a sample summarization ${index} for topic ${topicId}. It contains multiple sentences to simulate a realistic summary. The key points discussed were testing, benchmarking, and performance optimization.`,\n prompt: \"Summarize the latest posts about this topic\",\n post_count: 25,\n time_period_start: startDate.toISOString(),\n time_period_end: endDate.toISOString(),\n };\n}\n\n// Benchmark scenarios\nasync function benchmarkTopicOperations(count: number = 10) {\n console.log(\"\\n=== Benchmarking Topic Operations ===\");\n\n // Create topics\n const topicIds: number[] = [];\n await measureTime(\"Create topics\", async () => {\n for (let i = 0; i < count; i++) {\n const topic = generateSampleTopic(i);\n const id = await queries.createTopic(topic);\n topicIds.push(id);\n }\n });\n\n // Get all topics\n await measureTime(\"Get all topics\", async () => {\n return await queries.getAllTopics();\n });\n\n // Get individual topics\n await measureTime(\"Get topic by ID\", async () => {\n for (const id of topicIds.slice(0, Math.min(count, 5))) {\n await queries.getTopicById(id);\n }\n });\n\n // Update topics\n await measureTime(\"Update topics\", async () => {\n for (const id of topicIds.slice(0, Math.min(count, 5))) {\n await queries.updateTopic({\n id,\n title: `Updated Topic ${id}`,\n description: `This topic was updated during benchmarking.`,\n });\n }\n });\n\n return topicIds;\n}\n\nasync function benchmarkPostOperations(\n count: number = 20,\n topicIds: number[] = [],\n) {\n console.log(\"\\n=== Benchmarking Post Operations ===\");\n\n // Create posts\n const postIds: number[] = [];\n await measureTime(\"Create posts\", async () => {\n for (let i = 0; i < count; i++) {\n const post = generateSamplePost(i) as Post;\n // Add media to some posts\n if (i % 3 === 0) {\n post.media = [\n generateSampleMedia(0, 1),\n generateSampleMedia(0, 2),\n ];\n }\n const id = await queries.createPost(post);\n postIds.push(id);\n }\n });\n\n // Get posts by ID\n await measureTime(\"Get posts by ID\", async () => {\n for (const id of postIds.slice(0, Math.min(count, 5))) {\n await queries.getPostById(id);\n }\n });\n\n // Get posts by external ID\n await measureTime(\"Get posts by external ID\", async () => {\n for (let i = 0; i < Math.min(count, 5); i++) {\n await queries.getPostByExternalId(`test-${i}`);\n }\n });\n\n // Get recent posts\n await measureTime(\"Get recent posts\", async () => {\n return await queries.getRecentPosts(10);\n });\n\n // Add posts to topics\n if (topicIds.length > 0) {\n await measureTime(\"Add posts to topics\", async () => {\n for (const postId of postIds) {\n // Add each post to 1-3 random topics\n const numTopics = Math.floor(Math.random() * 3) + 1;\n for (let i = 0; i < numTopics; i++) {\n const randomTopicIndex = Math.floor(Math.random() * topicIds.length);\n await queries.addPostToTopic(topicIds[randomTopicIndex], postId);\n }\n }\n });\n\n // Get posts for topics\n await measureTime(\"Get posts for topic\", async () => {\n for (const topicId of topicIds.slice(0, Math.min(topicIds.length, 3))) {\n await queries.getPostsForTopic(topicId, 10);\n }\n });\n }\n\n return postIds;\n}\n\nasync function benchmarkSummarizationOperations(\n count: number = 5,\n topicIds: number[] = [],\n) {\n console.log(\"\\n=== Benchmarking Summarization Operations ===\");\n\n if (topicIds.length === 0) {\n console.log(\"No topics available for summarization benchmarking\");\n return [];\n }\n\n // Create summarizations\n const summaryIds: number[] = [];\n await measureTime(\"Create summarizations\", async () => {\n for (let i = 0; i < count; i++) {\n // Distribute summarizations across available topics\n const topicId = topicIds[i % topicIds.length];\n const summarization = generateSampleSummarization(topicId, i);\n const id = await queries.createSummarization(summarization);\n summaryIds.push(id);\n }\n });\n\n // Get summarizations by ID\n await measureTime(\"Get summarization by ID\", async () => {\n for (const id of summaryIds.slice(0, Math.min(count, 3))) {\n await queries.getSummarizationById(id);\n }\n });\n\n // Get summarizations for topic\n await measureTime(\"Get summarizations for topic\", async () => {\n for (const topicId of topicIds.slice(0, Math.min(topicIds.length, 3))) {\n await queries.getSummarizationsForTopic(topicId, 5);\n }\n });\n\n // Get recent summarizations\n await measureTime(\"Get recent summarizations\", async () => {\n return await queries.getRecentSummarizations(5);\n });\n\n return summaryIds;\n}\n\n// Main benchmark function\nexport async function runBenchmarks() {\n console.log(\"Starting database benchmarks...\");\n\n try {\n // Initialize the database\n await initializeDatabase();\n\n // Run benchmarks\n const topicIds = await benchmarkTopicOperations(10);\n const postIds = await benchmarkPostOperations(20, topicIds);\n await benchmarkSummarizationOperations(5, topicIds);\n\n // Clean up test data if needed\n // Uncomment this section to delete test data after benchmarking\n /*\n console.log(\"\\n=== Cleaning up test data ===\");\n for (const id of postIds) {\n await queries.deletePost(id);\n }\n for (const id of topicIds) {\n await queries.deleteTopic(id);\n }\n */\n\n console.log(\"\\nBenchmarks completed successfully!\");\n } catch (error: unknown) {\n console.error(\"Error during benchmarking:\", error);\n }\n}\n\n// Run benchmarks if this file is executed directly\n// Comment out this check for now as we'll run it manually\n// if (import.meta.url === Deno.mainModule) {\n// runBenchmarks();\n// }\n",[],[176,620,645],"8418f8b2-8a7c-11f0-be41-0224a6c84d84","8418f948-8a7c-11f0-be41-0224a6c84d84",43,["D",1757092784365],"# Database Module\n\nThis module handles all database operations for the OffX application using Val Town's SQLite module.\n\n## Schema Design\n\nThe database schema follows a normalized design (up to 3NF) with the following tables:\n\n### Tables\n\n1. **topics_v1**\n - Stores user-defined topics for categorizing and summarizing posts\n - Fields: id, title, description, created_at, updated_at\n\n2. **posts_v1**\n - Stores posts from various platforms (Twitter/X, etc.)\n - Fields: id, external_id, content, author, author_username, platform, created_at, fetched_at, url\n\n3. **post_media_v1**\n - Stores media attachments for posts\n - Fields: id, post_id, media_url, media_type, alt_text, created_at\n - Foreign key: post_id references posts_v1(id)\n\n4. **topic_posts_v1**\n - Junction table for many-to-many relationship between topics and posts\n - Fields: topic_id, post_id, added_at\n - Foreign keys: topic_id references topics_v1(id), post_id references posts_v1(id)\n\n5. **summarizations_v1**\n - Stores AI-generated summaries of posts for a topic\n - Fields: id, topic_id, content, prompt, post_count, created_at, time_period_start, time_period_end\n - Foreign key: topic_id references topics_v1(id)\n\n## Indexes\n\nThe following indexes are created to optimize query performance:\n\n1. **Posts table indexes**\n - idx_posts_external_id: For fast lookups by external platform ID\n - idx_posts_author: For filtering/searching posts by author\n - idx_posts_created_at: For time-based queries and sorting\n - idx_posts_platform: For filtering posts by platform\n\n2. **Topic-Posts junction table indexes**\n - idx_topic_posts_topic_id: For finding all posts related to a specific topic\n - idx_topic_posts_post_id: For finding all topics a post belongs to\n\n3. **Summarizations table indexes**\n - idx_summarizations_topic_id: For retrieving summaries for a specific topic\n - idx_summarizations_created_at: For time-based sorting of summaries\n - idx_summarizations_time_period: Compound index for efficient time range queries\n\n## Schema Change Strategy\n\nVal Town's SQLite implementation has limited support for ALTER TABLE operations. Instead of traditional migration frameworks, we use a table versioning approach for schema changes.\n\n### Table Versioning Approach\n\n1. **Version Suffix in Table Names**\n - All tables include a version suffix (e.g., `topics_v1`)\n - When schema changes are needed, create a new table with an incremented version (e.g., `topics_v2`)\n\n2. **Data Migration Process**\n - Create the new table with the updated schema\n - Copy data from the old table to the new table, mapping columns as needed\n - Update the `TABLES` constant in `migrations.ts` to point to the new table versions\n - Deploy the updated code\n\n3. **Utility Functions**\n - `tableExists(tableName)`: Check if a table exists in the database\n - `getTableColumns(tableName)`: Get column information for a table\n - `migrateTable(oldTableName, newTableName, createTableSql, columnMappings)`: Migrate data between tables\n\n### Example Schema Change\n\nTo add new columns to the topics table:\n\n```typescript\n// 1. Create a migration function\nasync function upgradeTopicsToV2() {\n const oldTable = \"topics_v1\";\n const newTable = \"topics_v2\";\n \n // Define the new table schema with additional columns\n const createTableSql = `\n CREATE TABLE IF NOT EXISTS ${newTable} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n title TEXT NOT NULL,\n description TEXT,\n created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,\n is_active BOOLEAN DEFAULT 1, -- New column added in v2\n color TEXT DEFAULT '#3498db' -- New column added in v2\n )\n `;\n \n // Migrate the table\n await migrateTable(oldTable, newTable, createTableSql);\n}\n\n// 2. Update the TABLES constant after migration\nexport const TABLES = {\n TOPICS: \"topics_v2\", // Updated to v2\n POSTS: \"posts_v1\",\n // ...other tables\n};\n```\n\n### Handling Column Renames\n\nWhen renaming columns, provide a mapping in the migration function:\n\n```typescript\n// Column mappings for renamed columns\nconst columnMappings = {\n \"author\": \"author_name\",\n \"platform\": \"source\",\n \"created_at\": \"published_at\"\n};\n\n// Use the mappings in the migration function\nawait migrateTable(oldTable, newTable, createTableSql, columnMappings);\n```\n\n### Best Practices\n\n1. **Always increment version numbers** - Never reuse a table name for a different schema\n2. **Test migrations thoroughly** - Verify data integrity after migration\n3. **Deploy schema changes separately** from application logic changes when possible\n4. **Document schema changes** in code comments and commit messages\n5. **Update all affected queries** when changing table or column names\n6. **Create appropriate indexes** for new tables and columns\n\n## Usage\n\nThe database module is automatically initialized when imported. For production deployments, you may want to comment out the auto-initialization for better performance in serverless environments:\n\n```typescript\n// Comment this out in production if you want to initialize manually\n// for better performance in serverless environments\ninitializeDatabase().catch(console.error);\n```",[],[176,620,269],"06a4db3c-55f5-11f0-9d32-f69ea79377d9",11,["D",1751317131524],[205,206,207,208,209,210],[176,620],"266ad7e8-55f3-11f0-9719-f69ea79377d9",["D",1751316325838],[204,211,215,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236],[176],["D",1751316115780],{"_11":12,"_145":72},{"_11":12,"_141":-5,"_71":72,"_31":70,"_142":143,"_144":30},{"_41":132,"_13":16,"_35":38,"_133":96,"_134":-5,"_135":-5,"_136":-5,"_137":-5,"_138":-5,"_139":14,"_140":12},"ctx",{"_54":-5,"_164":705,"_706":707,"_708":709,"_710":30,"_711":30,"_712":30,"_713":24,"_714":-5,"_715":-7,"_716":-7,"_717":26,"_718":30,"_719":30,"_720":-5,"_721":30,"_722":30,"_723":30,"_724":-5},"ENV",{"_676":677,"_678":679,"_680":16,"_681":682,"_683":684,"_685":686,"_687":688,"_689":690,"_691":692,"_693":694,"_695":696,"_697":698,"_699":700,"_701":702,"_703":704},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","e8c4b7030cb7600e4eb0736b15f422c8b8bd130f","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]

History · routes