MD Crawl

A website that crawls URLs recursively and outputs markdown content optimized for LLM consumption.

Features

  • Input a URL to crawl
  • Set crawl depth (default: 3)
  • Crawls only URLs on the same domain
  • Outputs a markdown representation of all pages
  • Includes a sitemap at the top for better context
  • Copy button with success state

Project Structure

  • backend/ - Server-side code
    • index.ts - Main entry point with Hono API
    • crawler.ts - Website crawling logic
  • frontend/ - Client-side code
    • index.html - Main HTML template
    • index.tsx - Frontend React code
    • components/ - React components