\n Configure Web Crawl\n

\n\n

\n \n URLs to Crawl (one per line)\n \n setUrls(e.target.value)}\n placeholder=\"https://example.com https://another-site.com\"\n className=\"w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent\"\n rows={4}\n required\n />\n

\n Enter one URL per line. Make sure to include http:// or https://\n

\n\n

\n \n Max Crawl Depth\n \n setMaxDepth(parseInt(e.target.value))}\n min=\"1\"\n max=\"5\"\n className=\"w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent\"\n />\n

\n How many levels deep to crawl (1-5)\n

\n\n

\n \n Max Pages\n \n setMaxPages(parseInt(e.target.value))}\n min=\"1\"\n max=\"500\"\n className=\"w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent\"\n />\n

\n Maximum number of pages to crawl\n

\n\n

\n \n Delay Between Requests (ms)\n \n setDelay(parseInt(e.target.value))}\n min=\"100\"\n max=\"10000\"\n step=\"100\"\n className=\"w-full px-3 py-2 border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent\"\n />\n

\n Delay to be respectful to servers\n

\n\n

\n setRespectRobots(e.target.checked)}\n className=\"w-4 h-4 text-blue-600 bg-gray-100 border-gray-300 rounded focus:ring-blue-500 focus:ring-2\"\n />\n

\n \n Respect robots.txt\n \n

\n Follow robots.txt guidelines (recommended)\n

\n\n {error && (\n

\n ⚠️\n

{error}

\n )}\n\n

\n \n {isSubmitting ? (\n <>\n

\n Starting Crawl...\n \n ) : (\n <>\n 🚀 Start Crawling\n \n )}\n \n

\n );\n}","deleted","label",[170,171,153],"frontend","components","permalink","https://www.val.town/x/diegoivo/todo-app-conversion/code/frontend/components/CrawlerForm.tsx","extension","actionData","errors"]