A Hono API server that fetches, caches, and processes Groq documentation pages with token counting.

• Fetches documentation pages from Groq's console
• Caches page content, metadata, and token counts in SQLite
• Token counting using tiktoken (GPT-4 encoding)
• RESTful API endpoints for accessing pages and managing cache

On first run, the cache will be empty. You should populate it by running:

POST /cache/recalculate

This will:

• Fetch all pages from the URLs list
• Calculate token counts for each page
• Store everything in the SQLite cache
• Return a summary of what was cached

Important: This will take some time as it processes all 121+ pages and calculates tokens for each. Be patient!

Check that the cache was populated:

GET /cache/stats

This returns:

{
  "cachedPages": 121,
  "totalTokens": 1234567
}

You should run /cache/recalculate in these scenarios:

1. First time setup - Cache is empty
2. URL list changes - You've added or removed URLs from the urls array
3. Content updates - Documentation pages have been updated and you want fresh data
4. Token count needed - You need accurate token counts for new content

For single page updates, you can use:

POST /cache/clear/:path

This clears the cache for a specific page. The next time that page is requested via /page/:path, it will be fetched fresh and recached.

• Weekly: Run recalculate to catch any documentation updates
• After major docs changes: Clear and recalculate
• When adding new pages: Update the urls array, then run recalculate

Get the root docs page (cached if available).

Get a specific page by path. Examples:

• /page/api-reference
• /page/agentic-tooling/compound-beta
• /page/model/llama-3.1-8b-instant

Response includes:

• url - The source URL
• content - Full page content with frontmatter
• charCount - Character count
• tokenCount - Token count (calculated with tiktoken)
• All frontmatter fields flattened (title, description, image, etc.)

Caching: Responses are cached. First request fetches and caches, subsequent requests are instant.

Get a list of all available page paths.

Response:

[
  "docs",
  "agentic-tooling",
  "api-reference",
  ...
]

Get metadata for all pages (does not use cache - fetches fresh).

Response:

{
  "pages": [
    {
      "url": "...",
      "charCount": 1234,
      "frontmatter": {...}
    }
  ],
  "contents": [...],
  "totalPages": 121,
  "totalChars": 1234567
}

Get cache statistics.

Response:

{
  "cachedPages": 121,
  "totalTokens": 1234567
}

Clear the entire cache.

Response:

{
  "message": "Cache cleared"
}

Clear cache for a specific page.

Example:

POST /cache/clear/api-reference

Response:

{
  "message": "Cache cleared for api-reference"
}

Recalculate all pages (bypasses cache, fetches fresh, recaches everything).

Response:

{
  "message": "Recalculated 121 pages",
  "results": [
    {
      "path": "api-reference",
      "url": "https://console.groq.com/docs/api-reference.md",
      "charCount": 1234,
      "tokenCount": 567
    },
    ...
  ]
}

Important: This can take several minutes depending on:

• Number of pages (currently 121)
• Network speed
• Token calculation time

1. First Request:

   • Check cache → Not found
   • Fetch from URL
   • Calculate tokens
   • Store in cache
   • Return data

2. Subsequent Requests:

   • Check cache → Found
   • Return cached data immediately

Cache is stored in SQLite with the following schema:

CREATE TABLE groq_docs_cache (
  url TEXT PRIMARY KEY,
  content TEXT NOT NULL,
  charCount INTEGER NOT NULL,
  tokenCount INTEGER,
  frontmatter TEXT NOT NULL,
  cachedAt INTEGER NOT NULL
)

Cache is invalidated when:

• You manually clear it via /cache/clear
• You recalculate via /cache/recalculate
• Cache is cleared for a specific page via /cache/clear/:path

Note: Cache does NOT automatically expire. If documentation changes, you must manually recalculate.

1. Add URL to the urls array in main.tsx:

   const urls = [
     // ... existing URLs
     "https://console.groq.com/docs/new-page.md",
   ];

2. Run recalculate:

   POST /cache/recalculate

3. Verify:

   GET /cache/stats
   GET /list  # Should include your new page

Token counts are calculated using tiktoken with the gpt-4 encoding (cl100k_base). This is the same encoding used by:

• GPT-4
• GPT-3.5-turbo
• Many other OpenAI models

Token counts are:

• Calculated on first fetch
• Stored in cache
• Returned in API responses
• Expensive to compute (which is why caching is important)

Run /cache/recalculate to refresh everything.

1. Check /list to see if the path exists
2. Verify the URL is in the urls array
3. Ensure the path matches the URL structure (e.g., api-reference for /docs/api-reference.md)

1. Clear cache for that page: POST /cache/clear/:path
2. Request the page again: GET /page/:path
3. Or recalculate everything: POST /cache/recalculate

• Use /page/:path endpoints (cached) instead of /data (uncached)
• Check cache stats: GET /cache/stats
• Ensure cache is populated before production use

deno run --allow-net --allow-env main.tsx

The app is configured to work with Val Town. Export uses:

export default (typeof Deno !== "undefined" && Deno.env.get("valtown")) ? app.fetch : app;