Complete Instructions for Implementing Glimpse Endpoint Caching

Implement a comprehensive caching system for /glimpse/:id endpoints to reduce Notion API response times from 1-6 seconds to milliseconds by using blob storage with automated cache population.

• The /glimpse/:id endpoint fetches data from Notion API including nested blocks via getAllBlocksRecursively
• Response times are slow (1-6 seconds) due to Notion API calls and recursive block fetching
• The system uses Hono for routing and has existing authentication middleware
• Database queries use GLANCE_DEMOS_DB_ID environment variable to identify demo pages
• Existing architecture has controllers in /backend/controllers/, services in /backend/services/, routes in /backend/routes/

function getValAndFileNameFromURL(url: string): string | null {
  try {
    const { pathname } = new URL(url);
    const [preAt] = pathname.split("@");
    if (!preAt) return null;
    const valName = preAt.split("/").filter(Boolean).pop();
    return valName ?? null;
  } catch (e) {
    console.error("Invalid URL passed to getValAndFileNameFromURL:", url, e);
    return null;
  }
}

export function blobKey(type: string, id: string, url: string = import.meta.url): string {
  if (!id || typeof id !== "string") {
    throw new Error("Invalid ID passed to blobKey");
  }

  const valName = getValAndFileNameFromURL(url);
  if (!valName) {
    throw new Error("Could not extract val name from URL");
  }

  return [valName, type, id].join("--");
}

export const blobKeyForDemos = (id: string) => blobKey("demo", id);

Add to /backend/services/notion.service.ts:

export async function getDemoPages() {
  try {
    const databaseId = Deno.env.get("GLANCE_DEMOS_DB_ID");
    if (!databaseId) {
      throw new Error("GLANCE_DEMOS_DB_ID environment variable is not set");
    }

    const response = await notion.databases.query({
      database_id: databaseId,
    });

    return {
      success: true,
      data: response.results,
      timestamp: new Date().toISOString(),
    };
  } catch (error) {
    return {
      success: false,
      error: error.message,
      timestamp: new Date().toISOString(),
    };
  }
}

Replace /backend/controllers/glimpse.controller.ts entirely:

import { Context } from "npm:hono@3.12.12";
import { getPageById, getMultiplePages } from "../services/notion.service.ts";
import { blob } from "https://esm.town/v/std/blob";
import { blobKeyForDemos } from "../utils/blob-keys.ts";

export async function getGlimpseData(id: string) {
  if (!id) {
    return { success: false, error: "Demo ID is required", status: 400 };
  }

  // Fetch the main page
  const result = await getPageById(id);

  if (!result.success) {
    return { success: false, error: "Failed to fetch demo data", details: result.error, status: 500 };
  }

  // Filter out button properties from the main page
  if (result.data?.properties) {
    const filteredProperties = Object.fromEntries(
      Object.entries(result.data.properties).filter(([key, value]) => value?.type !== "button")
    );

    result.data.properties = filteredProperties;
  }

  // Extract related page IDs from "Glimpse content" relation property
  const glimpseContentProperty = result.data?.properties?.["Glimpse content"];
  let glimpseContent = [];

  if (glimpseContentProperty?.type === "relation" && glimpseContentProperty.relation?.length > 0) {
    const relatedPageIds = glimpseContentProperty.relation.map((rel: any) => rel.id);

    // Fetch all related pages with their content
    const relatedPagesResult = await getMultiplePages(relatedPageIds);

    if (!relatedPagesResult.success) {
      return {
        success: false,
        error: "Failed to fetch related pages",
        details: relatedPagesResult.error,
        status: 500
      };
    }

    // Process and sort the related pages
    glimpseContent = relatedPagesResult.data
      .map((item: any) => {
        // Filter out button properties from related pages too
        const filteredProperties = item.page.properties ?
          Object.fromEntries(
            Object.entries(item.page.properties).filter(([key, value]) => (value as any)?.type !== "button")
          ) : {};

        return {
          id: item.id,
          properties: filteredProperties,
          blocks: item.blocks.results
        };
      })
      // Sort by Order property
      .sort((a: any, b: any) => {
        const orderA = a.properties?.Order?.number || 0;
        const orderB = b.properties?.Order?.number || 0;
        return orderA - orderB;
      });
  }

  // Add the glimpse content to the response
  result.data.glimpseContent = glimpseContent;

  return { success: true, data: result };
}

export async function glimpseHandler(c: Context) {
  const id = c.req.param("id");

  if (!id) {
    return c.json({ error: "Demo ID is required" }, 400);
  }

  // First try to get cached data
  try {
    const cachedData = await blob.getJSON(blobKeyForDemos(id));
    if (cachedData) {
      console.log(`Serving cached data for glimpse ID: ${id}`);
      return c.json(cachedData);
    }
  } catch (error) {
    console.log(`No cached data found for glimpse ID: ${id}, falling back to live data`);
  }

  // Cache miss - fall back to live Notion data
  const result = await getGlimpseData(id);

  if (!result.success) {
    return c.json({ error: result.error, details: result.details }, result.status);
  }

  return c.json(result.data);
}

Create /backend/crons/cache_glimpse_data.ts:

import { blob } from "https://esm.town/v/std/blob";
import { getDemoPages } from "../services/notion.service.ts";
import { getGlimpseData } from "../controllers/glimpse.controller.ts";
import { blobKeyForDemos } from "../utils/blob-keys.ts";

export default async function cacheGlimpseData() {
  console.log("Starting glimpse data caching process...");

  try {
    // Get all demo pages from the database
    const demosResult = await getDemoPages();

    if (!demosResult.success) {
      console.error("Failed to fetch demo pages:", demosResult.error);
      return;
    }

    const demoPages = demosResult.data;
    console.log(`Found ${demoPages.length} demo pages to cache`);

    let successCount = 0;
    let errorCount = 0;

    // Process each demo page
    for (const page of demoPages) {
      const pageId = page.id;

      try {
        console.log(`Caching glimpse data for page ID: ${pageId}`);

        // Get the glimpse data using the same logic as the endpoint
        const glimpseResult = await getGlimpseData(pageId);

        if (glimpseResult.success) {
          // Store in blob storage
          await blob.setJSON(blobKeyForDemos(pageId), glimpseResult.data);
          console.log(`Successfully cached data for page ID: ${pageId}`);
          successCount++;
        } else {
          console.error(`Failed to get glimpse data for page ID ${pageId}:`, glimpseResult.error);
          errorCount++;
        }
      } catch (error) {
        console.error(`Error processing page ID ${pageId}:`, error.message);
        errorCount++;
      }
    }

    console.log(`Caching process completed. Success: ${successCount}, Errors: ${errorCount}`);
  } catch (error) {
    console.error("Fatal error in caching process:", error.message);
  }
}

Use the change_val_type tool:

change_val_type("/backend/crons/cache_glimpse_data.ts", "cron")

CRITICAL: After implementation, go to Val Town web UI and set the cron schedule to run every minute (* * * * *).

Create temporary test file:

import { getDemoPages } from "./backend/services/notion.service.ts";

export default async function testDatabaseQuery(req: Request) {
  const result = await getDemoPages();
  return new Response(JSON.stringify({
    success: result.success,
    count: result.success ? result.data.length : 0,
    sampleIds: result.success ? result.data.slice(0, 3).map(p => p.id) : [],
    error: result.success ? null : result.error
  }), {
    headers: { "Content-Type": "application/json" }
  });
}

Expected: success: true, count: > 0, array of page IDs

Create temporary test file:

import { getGlimpseData } from "./backend/controllers/glimpse.controller.ts";
import { blob } from "https://esm.town/v/std/blob";
import { blobKeyForDemos } from "./backend/utils/blob-keys.ts";

export default async function testCaching(req: Request) {
  const testPageId = "FIRST_PAGE_ID_FROM_TEST_1"; // Replace with actual ID

  const glimpseResult = await getGlimpseData(testPageId);
  if (!glimpseResult.success) {
    return new Response(JSON.stringify({ error: glimpseResult.error }), { status: 500 });
  }

  const blobKey = blobKeyForDemos(testPageId);
  await blob.setJSON(blobKey, glimpseResult.data);

  const retrieved = await blob.getJSON(blobKey);

  return new Response(JSON.stringify({
    success: true,
    blobKey,
    dataStored: !!retrieved,
    hasNestedBlocks: !!retrieved?.data?.glimpseContent?.some(item =>
      item.blocks?.some(block => block.children && block.children.length > 0)
    )
  }));
}

Expected: success: true, dataStored: true, nested blocks preserved

Create temporary test file:

import { glimpseHandler } from "./backend/controllers/glimpse.controller.ts";

export default async function testCachedEndpoint(req: Request) {
  const testPageId = "CACHED_PAGE_ID_FROM_TEST_2";

  const mockContext = {
    req: { param: (name: string) => name === "id" ? testPageId : undefined },
    json: (data: any, status?: number) => new Response(JSON.stringify(data), {
      status: status || 200,
      headers: { "Content-Type": "application/json" }
    })
  };

  const response = await glimpseHandler(mockContext as any);
  const responseData = JSON.parse(await response.text());

  return new Response(JSON.stringify({
    success: true,
    status: response.status,
    servedFromCache: true, // Check logs for "Serving cached data" message
    hasData: !!responseData?.data
  }));
}

Expected: success: true, status: 200, logs show "Serving cached data"

• Response times: 1-6 seconds
• Every request hits Notion API
• Recursive block fetching on every request

• Cache Hit: 50-200ms response time
• Cache Miss: 1-6 seconds (fallback to live data)
• Cache Population: Automatic every minute via cron

• Check: Val Town web UI cron schedule is set to * * * * *
• Verify: Check cron execution logs with requests tool

• Check: Look for "Serving cached data" or "No cached data found" in endpoint logs
• Verify: Manually test blob key exists: blob.getJSON(blobKeyForDemos(pageId))

• Check: Compare live vs cached data structure
• Verify: getAllBlocksRecursively function is working correctly
• Expected: Blocks with children property containing nested arrays

• Check: Cron has run at least once (check cron logs)
• Verify: Cache keys exist in blob storage
• Debug: Look for cache miss logs in endpoint requests

1. Controllers: Business logic stays in /backend/controllers/
2. Services: External API calls in /backend/services/
3. Utilities: Backend-only helpers in /backend/utils/
4. Crons: Scheduled tasks in /backend/crons/
5. No Code Duplication: Both endpoint and cron use same getGlimpseData() function
6. Error Resilience: Cron continues on individual failures, endpoint falls back to live data
7. Data Integrity: Complete preservation of nested block structure and all metadata

• ✅ All demo pages automatically cached every minute
• ✅ Cache-first endpoint with live data fallback
• ✅ Response times under 200ms for cached data
• ✅ Complete preservation of nested block structure
• ✅ Error logging and graceful degradation
• ✅ No breaking changes to existing API contract