SCAFFOLD.md

This file provides step-by-step instructions for scaffolding a new Val Town application from scratch.

Note: This is a one-time setup guide. After completing these steps, see CLAUDE.md for ongoing development guidelines and architectural patterns.

This is a Notion webhook integration application that:

• Receives webhooks from Notion when pages/databases change
• Processes Notion data and integrates with external services (Firecrawler, etc.)
• Saves processed results back to Notion
• Provides a React frontend for viewing and managing Notion data

Follow these steps to scaffold the complete val structure:

mkdir -p backend/controllers backend/routes/api backend/routes/tasks backend/routes/views backend/services backend/utils backend/crons backend/email
mkdir -p frontend/components frontend/hooks
mkdir -p shared

Directory Purpose:

• backend/controllers/ - Business logic
• backend/routes/ - HTTP endpoints (webhook handlers, APIs, views)
• backend/services/ - External API integrations
• backend/utils/ - Backend-only utilities (pure functions, no external deps)
• backend/crons/ - Val Town cron trigger handlers (scheduled tasks)
• backend/email/ - Val Town email triggers & utilities (incoming/outgoing emails)
• frontend/components/ - React components
• frontend/hooks/ - Custom React hooks
• shared/ - Cross-platform code (types, utilities)

Create main.http.tsx with:

• IMPORTANT: Use .http.tsx extension (see AGENTS.md for Val Town trigger types)
• Hono app initialization
• Error unwrapper: app.onError((err, c) => { throw err; })
• Static file serving using Val Town utils (see AGENTS.md for serveFile pattern)
• Root route serving frontend/index.html
• Authentication middleware (array-based protection pattern)
• Commented route mounting examples
• Export: export default app.fetch

Organize services by external API. For Notion, create a directory structure:

backend/services/notion/index.ts:

• Import Notion client: import { Client } from "npm:@notionhq/client@2"
• Initialize and export client: export const notion = new Client({ auth: Deno.env.get("NOTION_API_KEY") })
• Re-export all functions from submodules:
  • export * from "./pages.ts"
  • export * from "./databases.ts"

backend/services/notion/pages.ts:

• Import shared client: import { notion } from "./index.ts"
• Export page-related functions:
  • getNotionPage(pageId) - Fetch page by ID
  • updateNotionPage(pageId, properties) - Update page properties
  • createNotionPage(databaseId, properties) - Create new page
  • getPageBlocks(pageId) - Get page content blocks

backend/services/notion/databases.ts:

• Import shared client: import { notion } from "./index.ts"
• Export database-related functions:
  • queryNotionDatabase(databaseId, filter?, sorts?) - Query database
  • getNotionDatabase(databaseId) - Get database info

Service function rules:

• All functions return {success, data, error} format
• Catch errors and return error objects (don't throw)
• Use the shared notion client from index.ts

IMPORTANT: Organize controllers by purpose or context. Each controller handles a specific domain or concern.

backend/controllers/healthController.ts:

• Export getHealth() function for system health checks
• Returns system information in data:
  • timestamp - Current ISO timestamp
  • project - Val Town project info (from backend/utils/valtown.ts)
  • configuration - Environment variable status (NOTION_API_KEY, WEBHOOK_SECRET)
• Uses backend/utils/valtown.ts for project information
• No service calls needed (checks environment variables and parses import.meta.url)
• Returns standard {success, data, error, details?} format
• Route returns just data (success indicated by HTTP 200)
• PUBLIC - No authentication required
• Default baseline view: Displayed on root / by App.tsx
  • Shows project name in H1: username/projectname
  • Shows endpoint path in H2: /api/health
  • Displays full JSON response
  • As you add more info to health endpoint, it automatically appears in UI

backend/controllers/pageController.ts:

• Import Notion service: import * as notionService from '../services/notion/index.ts'
• Export controller functions:
  • getPage(pageId) - Get page with validation
  • updatePage(pageId, properties) - Update with validation
  • createPage(databaseId, properties) - Create with validation
  • getPageContent(pageId) - Get blocks with validation
  • getRecentPages(hours?) - Get recently edited pages
• All functions return {success, data, error, details?} format
• Include filterSensitiveProperties() helper to remove button properties
• Validate inputs before calling services

Controller Organization Pattern:

• Group related operations in one controller (e.g., all page operations in pageController.ts)
• Separate concerns by domain (health, pages, webhooks, etc.)
• Controllers can grow to include more functions as features are added
• May call services (Notion, external APIs) or just check system state (health)

backend/routes/authCheck.ts:

• Import Hono types: import { Context, Next } from "npm:hono@4"
• Export webhookAuth(c, next) function:
  • Check for X-API-KEY header (case-insensitive)
  • Compare with Deno.env.get('WEBHOOK_SECRET')
  • If WEBHOOK_SECRET not configured, log warning and bypass auth (development mode)
  • Return 401 if invalid or missing
  • Call await next() if valid
  • Log failed authentication attempts

shared/types.ts:

• Define ControllerResponse<T> interface with {success, data, error, details?}
• Define Notion types: NotionPage, NotionDatabase, NotionWebhookPayload
• Define API types: UpdatePageRequest, CreatePageRequest, etc.
• Use generic types that work in browser and Deno

shared/utils.ts:

• Export utility functions:
  • isValidPageId(pageId) - Validate Notion ID format
  • normalizePageId(pageId) - Remove hyphens
  • formatPageId(pageId) - Add hyphens in 8-4-4-4-12 format
  • getPlainText(richText) - Extract text from Notion rich text
  • getPropertyValue(properties, name) - Get value by property type
• ONLY use standard JavaScript (no Deno or browser-specific APIs)

backend/utils/ directory:

• IMPORTANT: Only for pure utility functions with NO external dependencies and NO business logic
• Can use Deno APIs (unlike shared/utils.ts which must be browser-compatible)

backend/utils/valtown.ts (default utility):

• Import Val Town standard library: import { parseProject } from "https://esm.town/v/std/utils@85-main/index.ts"
• Export functions:
  • getProjectInfo() - Get Val Town project info (username, name, branch, links)
  • getProjectName() - Get just the project/val name
  • getProjectUsername() - Get the owner's username
  • getBlobKeyPrefix() - Get namespaced blob key prefix (username_projectname)
  • getProjectUrl() - Get the Val Town project URL
• Used by healthController.ts to populate project info
• Used by services for blob key namespacing

Additional utils (create as needed):

• cryptoUtils.ts - JWT signing, hashing with Web Crypto API
• validationUtils.ts - Advanced validation using Deno features
• envUtils.ts - Environment variable parsing helpers

Rules:

• Export pure functions only (no side effects)
• Keep functions small and focused on single responsibilities
• Do NOT add:
  • External API calls (those go in services/)
  • Business logic (that goes in controllers/)
  • Browser-compatible code (that goes in shared/utils.ts)

IMPORTANT: These are Val Town-specific features for scheduled tasks and email handling.

Val Town Trigger Documentation:

• Cron triggers: https://docs.val.town/vals/cron/
• Email triggers: https://docs.val.town/vals/email/
• Email utilities: https://docs.val.town/reference/std/email/

Create cron handlers for scheduled tasks:

When to create:

• ✅ Periodic data synchronization (hourly, daily, weekly)
• ✅ Scheduled cleanup or maintenance tasks
• ✅ Regular status checks or health monitoring
• ✅ Batch processing at intervals
• ✅ Scheduled reports or summaries

File naming: [taskName].cron.tsx

Pattern:

// backend/crons/syncNotionData.cron.tsx
import * as pageController from '../controllers/pageController.ts';

export default async function syncNotionData(interval: Interval) {
  console.log(`[Cron] Starting sync at ${new Date().toISOString()}`);

  try {
    const result = await pageController.syncAllPages();

    if (!result.success) {
      console.error(`[Cron] Sync failed: ${result.error}`);
      return;
    }

    console.log(`[Cron] Sync completed`);
  } catch (err) {
    console.error(`[Cron] Error:`, err);
  }
}

Rules:

• Use .cron.tsx extension
• Export default async function accepting interval: Interval
• Call controllers (same architecture as routes)
• Execute and complete (no return value needed)
• Add logging for monitoring
• Configure schedule in Val Town UI

Create email handlers for two purposes:

1. Email Triggers - Handle incoming emails

When to create:

• ✅ Process incoming support/contact emails
• ✅ Create Notion pages from email content
• ✅ Auto-reply to specific patterns
• ✅ Parse and route inbound messages

File naming: [handlerName].email.tsx

Pattern:

// backend/email/processTicket.email.tsx
import { email } from "https://esm.town/v/std/email";
import * as ticketController from '../controllers/ticketController.ts';

export default async function processTicket(email: Email) {
  console.log(`[Email] From: ${email.from}, Subject: ${email.subject}`);

  try {
    const result = await ticketController.createTicket({
      from: email.from,
      subject: email.subject,
      body: email.text
    });

    if (!result.success) {
      return await email.send({
        to: email.from,
        subject: "Re: " + email.subject,
        text: "Error processing your email."
      });
    }

    return await email.send({
      to: email.from,
      subject: "Re: " + email.subject,
      text: `Ticket created: ${result.data.ticketId}`
    });
  } catch (err) {
    console.error(`[Email] Error:`, err);
  }
}

2. Email Utilities - Send emails from code

When to create:

• ✅ Send notifications from controllers
• ✅ Email reports from crons
• ✅ Send alerts or confirmations

File naming: [utilityName].ts

Pattern:

// backend/email/notifications.ts
import { email } from "https://esm.town/v/std/email";

export async function sendNotification(to: string, subject: string, body: string) {
  try {
    await email.send({ to, subject, text: body });
    return { success: true, data: null, error: null };
  } catch (err: any) {
    return { success: false, data: null, error: err.message };
  }
}

Rules for email triggers:

• Use .email.tsx extension
• Export default async function accepting email: Email
• Call controllers for business logic
• Return email response using email.send()
• Configure email address in Val Town UI

Rules for email utilities:

• Use .ts extension (regular TypeScript)
• Export functions that send emails
• Return {success, data, error} format
• Can be called by controllers, services, or crons

All three trigger types follow the same architecture:

HTTP Route  ┐
Cron        ├──→  Controller → Service → External API
Email       ┘

Controllers can also call email utilities:

Controller → Email Utility → email.send()

frontend/index.html:

• Include Pico CSS: <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.min.css" />
• Add custom styles for root padding:

  <style>
    #root > div {
      padding-block: var(--pico-block-spacing-vertical);
    }
  </style>

• Include error catching script (see AGENTS.md)
• Root div: <div id="root"></div>
• Load React: <script type="module" src="/frontend/index.tsx"></script>

frontend/index.tsx:

• JSX import source and pinned React versions (see AGENTS.md for React configuration)
• Import App component
• Render <App /> to root element

frontend/hooks/useHealth.ts:

• Custom React hook for fetching /api/health data
• Returns { health, loading, error }
• Fetches on component mount (empty dependency array)
• Handles HTTP errors gracefully

frontend/components/App.tsx:

• JSX import source and pinned React imports
• Default pattern: Displays /api/health endpoint data
  • H1: Shows project.username/project.name from health data
  • H2: Clickable link to /api/health endpoint (opens in new tab)
  • Displays full health JSON in <pre> tag
  • Link to Val Town environment variables (constructed from project.links.self.project + "/environment-variables")
• Uses useHealth() custom hook for data fetching
• Handles loading and error states
• This is the baseline default view - as /api/health grows with more system info, it automatically appears here

frontend/components/NotionBlock.tsx:

• Render different Notion block types (paragraph, heading, lists, code, etc.)
• Use semantic HTML (Pico CSS provides automatic styling)

frontend/components/NotionProperty.tsx:

• Import getPropertyValue from shared/utils
• Render different property types (title, select, date, checkbox, etc.)
• Use semantic HTML (Pico CSS provides automatic styling)

IMPORTANT Documentation Pattern: Use top-level README.md files ONLY. Do NOT create README files in subdirectories.

Create README.md in these locations:

• Project root (/README.md) - Overview and architecture

  • Project purpose and identity
  • Quick start guide
  • Architecture overview diagram
  • Links to detailed documentation

• backend/README.md - Complete backend documentation

  • All 4 layers documented: Routes, Controllers, Services, Utils
  • Directory structure visualization
  • Example patterns for each layer
  • Rules and best practices
  • How to add new features (10-step checklist)
  • Error handling strategy
  • Authentication patterns
  • Import patterns

• frontend/README.md - Complete frontend documentation

  • React component structure
  • Styling guidelines (Pico CSS)
  • State management patterns
  • API integration patterns
  • Component examples

• shared/README.md - Complete shared code documentation

  • types.ts guidelines and examples
  • utils.ts guidelines and examples
  • Browser/Deno compatibility rules
  • When to use shared/utils.ts vs backend/utils/
  • Testing compatibility guide
  • Common use cases with examples

❌ DO NOT create README files in:

• backend/controllers/
• backend/routes/
• backend/services/
• backend/utils/ ← Document in /backend/README.md instead
• frontend/components/
• Any other subdirectories

Why Top-Level Only:

• Easier to maintain (fewer files to update)
• Complete context in one place per layer
• No duplication across multiple files
• Simpler navigation for developers

When Adding Features:

When you add new functionality to the codebase, ALWAYS update the relevant top-level README:

1. Added controller/service/route/util? → Update /backend/README.md
2. Added React component? → Update /frontend/README.md
3. Added shared type/util? → Update /shared/README.md
4. Changed architecture? → Update root /README.md and CLAUDE.md

README Content Standards:

Each top-level README should include:

• Directory structure visualization
• Purpose and responsibilities of each file/subdirectory
• Code examples for common patterns
• Rules and guidelines (what goes where)
• Best practices
• How to add new features
• Cross-references to other documentation

Document required environment variables:

• NOTION_API_KEY - Notion integration token (required)
• WEBHOOK_SECRET - Shared secret for API and webhook authentication (required for production, optional for development)
• RECENT_PAGES_LOOKBACK_HOURS - Hours to look back for recent pages (optional, defaults to 24)
• Add service-specific keys as needed (e.g., FIRECRAWLER_API_KEY)

After scaffolding, verify:

• ✅ All directories exist
• ✅ main.http.tsx exports app.fetch
• ✅ Service functions return {success, data, error}
• ✅ Controller functions return {success, data, error, details?}
• ✅ Frontend has JSX import source and pinned React versions
• ✅ Shared code has no Deno/browser-specific APIs
• ✅ README files explain each directory's purpose

Scaffolding complete! Now see CLAUDE.md for:

• MVC architecture patterns
• Import patterns
• Adding new features
• Error handling strategy
• Authentication rules
• Integration with AGENTS.md

For Val Town platform patterns (redirects, React config, standard library usage), see AGENTS.md.