[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_270":271,"_272":273},"routes/_app.x.$handle.$name",{"_95":94,"_32":44,"_103":85,"_27":104,"_13":14,"_105":-5,"_11":106,"_107":108,"_109":110,"_111":112,"_113":114,"_20":-5,"_115":-5,"_116":-5,"_117":25,"_118":119,"_120":57,"_121":-5,"_122":-5,"_123":124},"routes/_app.x.$handle.$name.branch.$branch.version.$version.diff",{"_11":12,"_13":14,"_15":16,"_17":-5,"_18":-5},"project",{"_42":43,"_32":44,"_37":45,"_46":-5,"_47":25,"_48":49,"_50":49,"_51":-5,"_52":-5,"_18":53,"_54":55,"_56":57,"_58":57,"_59":-5,"_60":61,"_62":63,"_64":-5,"_65":-5,"_66":-5,"_67":-5,"_68":69,"_70":69,"_71":72,"_73":72,"_74":72,"_75":69,"_76":69,"_77":78,"_79":80,"_81":82,"_83":25},"version",76,"changes",[19],"aiChatInfo","userId",{"_20":21,"_22":23,"_24":25,"_26":25,"_27":28,"_29":28},"file",{"_30":31,"_32":28,"_33":20,"_34":36,"_37":38,"_39":14,"_40":41,"_24":25},"previous",{"_30":31,"_32":28,"_33":20,"_34":35},"deleted",false,"deletedHere","path","README.md","latestPath","fileId","38a799a2-7612-11f0-bec5-0224a6c84d84","name","fileType","content","# Quarter Price Auction Scraper API\n\nA simple, focused JSON API that scrapes **every item** from **every auction** across **all pages** on quarterpriceauction.com.\n\n## Structure\n\n```\n├── backend/\n│ ├── index.ts # Main HTTP endpoint\n│ └── scrapers/\n│ ├── mainScraper.ts # Main orchestration logic\n│ ├── auctionDiscovery.ts # Finds all auction URLs\n│ ├── auctionScraper.ts # Scrapes individual auctions\n│ ├── paginationDetector.ts # Detects page count per auction\n│ └── itemExtractor.ts # Extracts item data from HTML/REDUX\n├── shared/\n│ └── types.ts # TypeScript interfaces\n└── index.ts # Entry point\n```\n\n## Usage\n\n### Main Endpoint\n\n- `GET /` - **Returns ALL items from ALL auctions** (JSON format)\n\n### Other Endpoints\n\n- `GET /csv` - **Returns ALL items from ALL auctions** (CSV format with download)\n- `GET /health` - Health check endpoint\n- `GET /info` - API information and available endpoints\n- `GET /fresh` - Force fresh scrape (no cache)\n- `GET /metadata` - Get metadata and sample items only\n\n## Response Format\n\n### JSON Format (`GET /`)\n\nReturns **all items** from **all auctions**:\n\n```json\n{\n \"timestamp\": \"2025-08-16T18:25:00.000Z\",\n \"totalItems\": 450,\n \"totalAuctions\": 6,\n \"items\": [\n {\n \"title\": \"Google Pixel 9a with Gemini - Unlocked Android Smartphone...\",\n \"auctionTitle\": \"August 22 Auction - Quarter Price\",\n \"auctionUrl\": \"https://www.quarterpriceauction.com/auctions/30135-august-22-auction\",\n \"itemUrl\": \"https://www.quarterpriceauction.com/auctions/30135/lot/291744-google-pixel...\",\n \"imageUrl\": \"https://d3j17a2r8lnfte.cloudfront.net/qrt/2025/8/medium/DG6mWsSZbO4GZPZh7Trv5Fza.jpeg\",\n \"lotNumber\": \"291744\",\n \"currentBid\": \"$45.00\",\n \"startingBid\": \"$1.00\",\n \"bidCount\": 12,\n \"timeLeft\": \"2d 4h 15m\",\n \"category\": \"Electronics\"\n }\n ],\n \"metadata\": {\n \"reduxDataFound\": true,\n \"auctionUrlsFound\": 6,\n \"scrapedAuctions\": 6,\n \"totalPagesScraped\": 45,\n \"processingTimeMs\": 180000\n }\n}\n```\n\n### CSV Format (`GET /csv`)\n\nReturns the same data as a downloadable CSV file with headers:\n\n```csv\nTitle,Auction,Lot Number,Current Bid,Starting Bid,Bid Count,Time Left,Category,Auction URL,Item URL,Image URL,Description\n\"Google Pixel 9a with Gemini - Unlocked Android Smartphone...\",\"August 22 Auction - Quarter Price\",\"291744\",\"$45.00\",\"$1.00\",\"12\",\"2d 4h 15m\",\"Electronics\",\"https://www.quarterpriceauction.com/auctions/30135-august-22-auction\",\"https://www.quarterpriceauction.com/auctions/30135/lot/291744-google-pixel...\",\"https://d3j17a2r8lnfte.cloudfront.net/qrt/2025/8/medium/DG6mWsSZbO4GZPZh7Trv5Fza.jpeg\",\"\"\n```\n\n- **Content-Type**: `text/csv`\n- **Filename**: `quarter-price-auction-items-YYYY-MM-DD.csv`\n- **Download**: Automatically triggers download in browsers\n\n## Features\n\n- **🔍 Comprehensive Scraping**: Extracts every item from every auction\n- **📄 Multi-Page Support**: Automatically detects and handles pagination\n- **🎯 Detailed Item Data**: Titles, lot numbers, prices, images, categories\n- **🧠 Intelligent Parsing**: Uses REDUX data extraction + HTML fallbacks\n- **⚡ Performance Optimized**: High-performance concurrent processing, no artificial delays\n- **🛡️ Error Resilient**: Graceful error handling with detailed metadata\n- **📊 Rich Metadata**: Processing statistics and data source information\n- **📁 Multiple Formats**: JSON and CSV output formats\n\n## Technical Implementation\n\n### Multi-Level Scraping Strategy\n\n1. **Auction Discovery**: Finds all auction URLs from main page\n2. **Pagination Detection**: Uses binary search to find max pages per auction\n3. **Item Extraction**: Tries REDUX data first, falls back to HTML parsing\n4. **Data Enhancement**: Extracts lot numbers, prices, bid counts from containers\n\n### Performance Features\n\n- **Rate Limiting**: 300-500ms delays between requests\n- **Caching**: 10-minute cache for comprehensive scraping\n- **Binary Search**: Efficient pagination detection\n- **Memory Management**: Processes auctions sequentially\n\n### Item Data Fields\n\n- `title` - Full item name/description\n- `auctionTitle` - Which auction the item belongs to\n- `auctionUrl` - Link to the auction page\n- `itemUrl` - Direct link to the individual item/lot\n- `imageUrl` - Product image from CDN\n- `lotNumber` - Auction lot number\n- `currentBid` - Current highest bid amount\n- `startingBid` - Starting bid amount\n- `bidCount` - Number of bids placed\n- `timeLeft` - Time remaining in auction\n- `category` - Item category\n\n## Performance Notes\n\n- **High-Performance Concurrent Processing**: ~2 minutes 15 seconds for 2,956+ items\n- **Concurrent auction processing**: All auctions scraped simultaneously \n- **Concurrent page processing**: All pages within auctions scraped in parallel\n- **Smart pagination detection**: Uses concurrent requests and binary search\n- **No artificial delays**: Optimized for high-performance servers\n- **Throughput**: ~22 items per second, ~0.44 pages per second\n- Results are cached to improve subsequent request performance\n\n## ✅ MISSION\n\n**Goal**: Simple JSON list of every item across all auctions with metadata.\n\n**Status**: ✅ Accomplished - Clean, organized, focused implementation.","# Quarter Price Auction Scraper API\n\nA simple, focused JSON API that scrapes **every item** from **every auction** across **all pages** on quarterpriceauction.com.\n\n## Structure\n\n```\n├── backend/\n│ ├── index.ts # Main HTTP endpoint\n│ └── scrapers/\n│ ├── mainScraper.ts # Main orchestration logic\n│ ├── auctionDiscovery.ts # Finds all auction URLs\n│ ├── auctionScraper.ts # Scrapes individual auctions\n│ ├── paginationDetector.ts # Detects page count per auction\n│ └── itemExtractor.ts # Extracts item data from HTML/REDUX\n├── shared/\n│ └── types.ts # TypeScript interfaces\n└── index.ts # Entry point\n```\n\n## Usage\n\n### Main Endpoint\n\n- `GET /` - **Returns ALL items from ALL auctions** (JSON format)\n\n### Other Endpoints\n\n- `GET /csv` - **Returns ALL items from ALL auctions** (CSV format with download)\n- `GET /health` - Health check endpoint\n- `GET /info` - API information and available endpoints\n- `GET /fresh` - Force fresh scrape (no cache)\n- `GET /metadata` - Get metadata and sample items only\n\n## Response Format\n\n### JSON Format (`GET /`)\n\nReturns **all items** from **all auctions**:\n\n```json\n{\n \"timestamp\": \"2025-08-16T18:25:00.000Z\",\n \"totalItems\": 450,\n \"totalAuctions\": 6,\n \"items\": [\n {\n \"title\": \"Google Pixel 9a with Gemini - Unlocked Android Smartphone...\",\n \"auctionTitle\": \"August 22 Auction - Quarter Price\",\n \"auctionUrl\": \"https://www.quarterpriceauction.com/auctions/30135-august-22-auction\",\n \"itemUrl\": \"https://www.quarterpriceauction.com/auctions/30135/lot/291744-google-pixel...\",\n \"imageUrl\": \"https://d3j17a2r8lnfte.cloudfront.net/qrt/2025/8/medium/DG6mWsSZbO4GZPZh7Trv5Fza.jpeg\",\n \"lotNumber\": \"291744\",\n \"currentBid\": \"$45.00\",\n \"startingBid\": \"$1.00\",\n \"bidCount\": 12,\n \"timeLeft\": \"2d 4h 15m\",\n \"category\": \"Electronics\"\n }\n ],\n \"metadata\": {\n \"reduxDataFound\": true,\n \"auctionUrlsFound\": 6,\n \"scrapedAuctions\": 6,\n \"totalPagesScraped\": 45,\n \"processingTimeMs\": 180000\n }\n}\n```\n\n### CSV Format (`GET /csv`)\n\nReturns the same data as a downloadable CSV file with headers:\n\n```csv\nTitle,Auction,Lot Number,Current Bid,Starting Bid,Bid Count,Time Left,Category,Auction URL,Item URL,Image URL,Description\n\"Google Pixel 9a with Gemini - Unlocked Android Smartphone...\",\"August 22 Auction - Quarter Price\",\"291744\",\"$45.00\",\"$1.00\",\"12\",\"2d 4h 15m\",\"Electronics\",\"https://www.quarterpriceauction.com/auctions/30135-august-22-auction\",\"https://www.quarterpriceauction.com/auctions/30135/lot/291744-google-pixel...\",\"https://d3j17a2r8lnfte.cloudfront.net/qrt/2025/8/medium/DG6mWsSZbO4GZPZh7Trv5Fza.jpeg\",\"\"\n```\n\n- **Content-Type**: `text/csv`\n- **Filename**: `quarter-price-auction-items-YYYY-MM-DD.csv`\n- **Download**: Automatically triggers download in browsers\n\n## Features\n\n- **🔍 Comprehensive Scraping**: Extracts every item from every auction\n- **📄 Multi-Page Support**: Automatically detects and handles pagination\n- **🎯 Detailed Item Data**: Titles, lot numbers, prices, images, categories\n- **🧠 Intelligent Parsing**: Uses REDUX data extraction + HTML fallbacks\n- **⚡ Performance Optimized**: High-performance concurrent processing, no artificial delays\n- **🛡️ Error Resilient**: Graceful error handling with detailed metadata\n- **📊 Rich Metadata**: Processing statistics and data source information\n- **📁 Multiple Formats**: JSON and CSV output formats\n\n## Technical Implementation\n\n### Multi-Level Scraping Strategy\n\n1. **Auction Discovery**: Finds all auction URLs from main page\n2. **Pagination Detection**: Uses binary search to find max pages per auction\n3. **Enhanced Item Extraction**: \n - **Apollo State** (preferred): Extracts from `window.__APOLLO_STATE__` for most detailed data\n - **REDUX Data** (fallback): Uses `window.REDUX_DATA` extraction\n - **HTML Parsing** (final fallback): Traditional HTML parsing\n4. **Data Enhancement**: Calculates time remaining, formats prices, extracts lot numbers\n\n### Performance Features\n\n- **Rate Limiting**: 300-500ms delays between requests\n- **Caching**: 10-minute cache for comprehensive scraping\n- **Binary Search**: Efficient pagination detection\n- **Memory Management**: Processes auctions sequentially\n\n### Item Data Fields\n\n- `title` - Full item name/description\n- `auctionTitle` - Which auction the item belongs to\n- `auctionUrl` - Link to the auction page\n- `itemUrl` - Direct link to the individual item/lot\n- `imageUrl` - Product image from CDN\n- `lotNumber` - Auction lot number\n- `currentBid` - Current highest bid amount\n- `startingBid` - Starting bid amount\n- `bidCount` - Number of bids placed\n- `timeLeft` - Time remaining in auction\n- `category` - Item category\n\n## Performance Notes\n\n- **High-Performance Concurrent Processing**: ~2 minutes 15 seconds for 2,956+ items\n- **Concurrent auction processing**: All auctions scraped simultaneously \n- **Concurrent page processing**: All pages within auctions scraped in parallel\n- **Smart pagination detection**: Uses concurrent requests and binary search\n- **No artificial delays**: Optimized for high-performance servers\n- **Throughput**: ~22 items per second, ~0.44 pages per second\n- Results are cached to improve subsequent request performance\n\n## ✅ MISSION\n\n**Goal**: Simple JSON list of every item across all auctions with metadata.\n\n**Status**: ✅ Accomplished - Clean, organized, focused implementation.","createdAt",["D",1755371119721],"branchVersion","branchId","27c8a162-7612-11f0-80a7-0224a6c84d84","id","27c86a44-7612-11f0-80a7-0224a6c84d84","QPAScrape",["D",1754848079739],"deletedAt","legacyEnvVars","privacy","public","httpPrivacy","description","imageUrl","4555dd0e-707e-4444-acd3-049d206eb1db","pinnedFileId","471c270a-7612-11f0-bbfb-0224a6c84d84","pinPreview",true,"pinMigrated","screenshotVersion","likeCount",0,"orgId","bced381e-9ab2-4b55-929d-e7c5064f7f38","tursoName","tursoHostname","tursoMeasuredAt","pinned","userPerm","r","valPerm","blobPerm","rw","emailPerm","sqlitePerm","projectPerm","telemetryPerm","user",{"_95":94,"_102":98},"org",{"_95":94,"_96":-5,"_97":98,"_18":53,"_99":100,"_101":25},"branch",{"_13":84,"_32":85,"_42":41,"_86":43,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":-5,"_92":44,"_93":94},"isOwned",79,"main","projectId","forkedBranchId","forkedBranchVersion","forkedProjectPrivacy","forkedProjectId","forkedProjectOrgId","projectName","userHandle","wolf","handle","displayName","avatar","https://img.clerk.com/eyJ0eXBlIjoicHJveHkiLCJzcmMiOiJodHRwczovL2ltYWdlcy5jbGVyay5kZXYvdXBsb2FkZWQvaW1nXzJvVXpPVUZRTjZGTGd2Q2t4c0RRdFpNZWJ5UCJ9","tier","pro","canRestrictVals","profileImageUrl","branchName","","adoptOrgId",{"_42":43,"_32":44,"_37":266,"_46":-5,"_47":25,"_48":49,"_50":49,"_51":-5,"_52":-5,"_18":53,"_54":55,"_56":57,"_58":57,"_59":-5,"_60":61,"_62":63,"_64":-5,"_65":-5,"_66":-5,"_67":-5,"_68":69,"_70":69,"_71":72,"_73":72,"_74":72,"_75":69,"_76":69,"_77":267,"_79":268,"_81":269,"_83":25},"tree",[168,177,179,180],"files",[168,169,170,171,172,173,174,175,176,177,178,179,180],"counts",{"_157":61,"_158":159,"_160":61,"_161":162,"_163":164},"contents",[125,126,127,128],"upstreamProject","upstreamBranch","emptyEnvVars","flags",[],"hasHttpFiles","grants","orgs","customDomains",{},{"_42":151,"_32":152,"_30":153,"_131":-5,"_40":41,"_39":154,"_133":57,"_37":155,"_13":61,"_33":149,"_34":-5,"_24":25,"_138":-5,"_27":156},{"_42":144,"_32":145,"_30":146,"_131":-5,"_40":41,"_39":147,"_133":57,"_37":148,"_13":61,"_33":149,"_34":-5,"_24":25,"_138":-5,"_27":150},{"_42":140,"_32":28,"_30":31,"_131":-5,"_40":41,"_39":14,"_133":25,"_37":141,"_13":142,"_33":20,"_34":36,"_24":25,"_138":-5,"_27":143},{"_42":129,"_32":130,"_30":55,"_131":-5,"_40":41,"_39":132,"_133":57,"_37":134,"_13":135,"_33":136,"_34":137,"_24":25,"_138":-5,"_27":139},"23834182-7ace-11f0-9a73-0224a6c84d84","index.ts","parentId",37,"latest",["D",1755368622606],25,"http","// Main entry point - redirect to backend\nexport { default } from \"./backend/index.ts\";","label",[130],"f3e9582a-7ad3-11f0-b075-0224a6c84d84",["D",1755371119721],9,[28],"ee869844-7acd-11f0-9f72-0224a6c84d84","shared","70d7061f-95d3-4643-9a03-b479b7308bb0",30,["D",1755368533698],"directory",[145],"f3970530-7acd-11f0-b736-0224a6c84d84","backend","fc20f944-0b9e-4e36-9a6d-25006bf4c3db",31,["D",1755368542201],[152],"pulls","branches",1,"forks","envVars",{"_165":61,"_167":61},"connections",{"_165":61,"_166":61},"count","unfilledCount","emptyCount",{"_42":151,"_32":152,"_30":153,"_131":-5,"_40":41,"_39":154,"_133":57,"_37":263,"_13":61,"_33":149,"_34":-5,"_24":25,"_138":-5,"_182":264,"_27":265},{"_42":259,"_32":217,"_30":210,"_131":153,"_40":41,"_39":154,"_133":57,"_37":260,"_13":61,"_33":149,"_34":-5,"_24":25,"_138":-5,"_182":261,"_27":262},{"_42":252,"_32":253,"_30":254,"_131":210,"_40":41,"_39":154,"_133":57,"_37":255,"_13":61,"_33":192,"_34":256,"_24":25,"_138":-5,"_182":257,"_27":258},{"_42":243,"_32":244,"_30":245,"_131":210,"_40":41,"_39":246,"_133":57,"_37":247,"_13":248,"_33":192,"_34":249,"_24":25,"_138":-5,"_182":250,"_27":251},{"_42":235,"_32":236,"_30":237,"_131":210,"_40":41,"_39":238,"_133":57,"_37":239,"_13":213,"_33":192,"_34":240,"_24":25,"_138":-5,"_182":241,"_27":242},{"_42":226,"_32":227,"_30":228,"_131":210,"_40":41,"_39":229,"_133":25,"_37":230,"_13":231,"_33":192,"_34":232,"_24":25,"_138":-5,"_182":233,"_27":234},{"_42":218,"_32":219,"_30":220,"_131":210,"_40":41,"_39":221,"_133":57,"_37":222,"_13":159,"_33":192,"_34":223,"_24":25,"_138":-5,"_182":224,"_27":225},{"_42":207,"_32":208,"_30":209,"_131":210,"_40":41,"_39":211,"_133":57,"_37":212,"_13":213,"_33":192,"_34":214,"_24":25,"_138":-5,"_182":215,"_27":216},{"_42":199,"_32":130,"_30":200,"_131":153,"_40":41,"_39":201,"_133":57,"_37":202,"_13":203,"_33":192,"_34":204,"_24":25,"_138":-5,"_182":205,"_27":206},{"_42":144,"_32":145,"_30":146,"_131":-5,"_40":41,"_39":147,"_133":57,"_37":196,"_13":61,"_33":149,"_34":-5,"_24":25,"_138":-5,"_182":197,"_27":198},{"_42":188,"_32":189,"_30":190,"_131":146,"_40":41,"_39":147,"_133":57,"_37":191,"_13":61,"_33":192,"_34":193,"_24":25,"_138":-5,"_182":194,"_27":195},{"_42":140,"_32":28,"_30":31,"_131":-5,"_40":41,"_39":14,"_133":25,"_37":185,"_13":142,"_33":20,"_34":36,"_24":25,"_138":-5,"_182":186,"_27":187},{"_42":129,"_32":130,"_30":55,"_131":-5,"_40":41,"_39":132,"_133":57,"_37":181,"_13":135,"_33":136,"_34":137,"_24":25,"_138":-5,"_182":183,"_27":184},["D",1755368622606],"children",[],[130],["D",1755371119721],[],[28],"ee868ad4-7acd-11f0-9f72-0224a6c84d84","types.ts","ee868bc4-7acd-11f0-9f72-0224a6c84d84",["D",1755368533698],"script","export interface AuctionItem {\n title: string;\n currentBid?: string;\n timeLeft?: string;\n imageUrl?: string;\n auctionUrl?: string;\n itemUrl?: string;\n description?: string;\n lotNumber?: string;\n auctionTitle?: string;\n startingBid?: string;\n bidCount?: number;\n category?: string;\n}\n\nexport interface ScrapedData {\n timestamp: string;\n totalItems: number;\n totalAuctions: number;\n items: AuctionItem[];\n error?: string;\n metadata?: {\n reduxDataFound: boolean;\n auctionUrlsFound: number;\n scrapedAuctions: number;\n totalPagesScraped: number;\n processingTimeMs: number;\n };\n}",[],[145,189],["D",1755368533698],[178],[145],"4b904918-7ad3-11f0-9056-0224a6c84d84","219af112-7ace-11f0-a140-0224a6c84d84",75,["D",1755370837280],11,"import { Hono } from \"https://esm.sh/hono@3.11.7\";\nimport { stringify } from \"https://esm.sh/csv-stringify@6.4.6/sync\";\nimport { scrapeAllAuctionItems } from \"./scrapers/mainScraper.ts\";\n\nconst app = new Hono();\n\n// Unwrap Hono errors to see original error details\napp.onError((err) => Promise.reject(err));\n\n// Main endpoint - returns all items from all auctions\napp.get(\"/\", async (c) => {\n const data = await scrapeAllAuctionItems();\n \n return c.json(data, {\n headers: {\n \"Cache-Control\": \"public, max-age=300\", // Reduced cache for testing\n }\n });\n});\n\n// Health check endpoint\napp.get(\"/health\", (c) => {\n return c.json({ \n status: \"ok\", \n timestamp: new Date().toISOString(),\n service: \"Quarter Price Auction Scraper\"\n });\n});\n\n// API info endpoint\napp.get(\"/info\", (c) => {\n return c.json({\n name: \"Quarter Price Auction Scraper API\",\n description: \"Simple scraper for all items across all auctions from quarterpriceauction.com\",\n endpoints: {\n \"/\": \"Get ALL items from ALL auctions (JSON)\",\n \"/csv\": \"Get ALL items from ALL auctions (CSV format)\",\n \"/health\": \"Health check\",\n \"/info\": \"API information\",\n \"/fresh\": \"Force fresh scrape (no cache)\",\n \"/metadata\": \"Get metadata and sample items only\"\n },\n version: \"3.0.0\",\n features: [\n \"Scrapes every item from every auction\",\n \"Handles pagination automatically\",\n \"Extracts detailed item information\",\n \"JSON and CSV output formats\",\n \"High-performance concurrent processing\"\n ]\n });\n});\n\n// CSV endpoint - returns all items as CSV\napp.get(\"/csv\", async (c) => {\n const data = await scrapeAllAuctionItems();\n \n if (data.error) {\n return c.json({ error: data.error }, 500);\n }\n \n // Convert items to CSV format\n const csvData = stringify(data.items, {\n header: true,\n columns: [\n { key: 'title', header: 'Title' },\n { key: 'auctionTitle', header: 'Auction' },\n { key: 'lotNumber', header: 'Lot Number' },\n { key: 'currentBid', header: 'Current Bid' },\n { key: 'startingBid', header: 'Starting Bid' },\n { key: 'bidCount', header: 'Bid Count' },\n { key: 'timeLeft', header: 'Time Left' },\n { key: 'category', header: 'Category' },\n { key: 'auctionUrl', header: 'Auction URL' },\n { key: 'itemUrl', header: 'Item URL' },\n { key: 'imageUrl', header: 'Image URL' },\n { key: 'description', header: 'Description' }\n ]\n });\n \n return new Response(csvData, {\n headers: {\n \"Content-Type\": \"text/csv; charset=utf-8\",\n \"Content-Disposition\": `attachment; filename=\"quarter-price-auction-items-${new Date().toISOString().split('T')[0]}.csv\"`,\n \"Cache-Control\": \"public, max-age=300\",\n }\n });\n});\n\n// Fresh scrape endpoint (no cache)\napp.get(\"/fresh\", async (c) => {\n const data = await scrapeAllAuctionItems();\n \n return c.json(data, {\n headers: {\n \"Cache-Control\": \"no-cache, no-store, must-revalidate\",\n }\n });\n});\n\n// Metadata only endpoint for performance testing\napp.get(\"/metadata\", async (c) => {\n const data = await scrapeAllAuctionItems();\n \n return c.json({\n timestamp: data.timestamp,\n totalItems: data.totalItems,\n totalAuctions: data.totalAuctions,\n metadata: data.metadata,\n sampleItems: data.items.slice(0, 3) // Just show first 3 items\n }, {\n headers: {\n \"Cache-Control\": \"no-cache, no-store, must-revalidate\",\n }\n });\n});\n\nexport default app.fetch;",[],[152,130],"fe273e56-7ace-11f0-a0e5-0224a6c84d84","paginationDetector.ts","f964cf56-7acd-11f0-9fe5-0224a6c84d84","31dd3db8-c538-401b-86da-04105849d3b1",43,["D",1755368989420],2,"/**\n * Detects the maximum number of pages for an auction\n */\nexport async function detectMaxPages(auctionUrl: string, firstPageHtml: string, itemCount: number): Promise {\n let maxPage = 1;\n \n // If we found close to 50 items on page 1, there are likely more pages\n if (itemCount >= 45) {\n console.log(`${itemCount} items found on page 1, checking for more pages...`);\n \n try {\n // Test if page 2 exists\n const testPage2 = await fetch(`${auctionUrl}?page=2`);\n if (testPage2.ok) {\n const testHtml = await testPage2.text();\n // Check if page 2 has content\n if (testHtml.length > 100000 && testHtml.includes('lot-img')) {\n console.log(`Page 2 exists for ${auctionUrl}, finding max page...`);\n \n // Use concurrent requests to find max page more efficiently\n maxPage = await findMaxPageConcurrently(auctionUrl);\n console.log(`Found ${maxPage} pages for ${auctionUrl}`);\n }\n }\n } catch (e) {\n console.log(`Error detecting pages for ${auctionUrl}:`, e);\n }\n }\n \n return maxPage;\n}\n\n/**\n * Finds max page using concurrent requests for faster detection\n */\nasync function findMaxPageConcurrently(auctionUrl: string): Promise {\n // Test multiple pages concurrently to find the boundary faster\n const testPages = [5, 10, 15, 20, 25, 30];\n \n const pagePromises = testPages.map(async (page) => {\n try {\n const response = await fetch(`${auctionUrl}?page=${page}`);\n if (response.ok) {\n const html = await response.text();\n const isValid = html.length > 100000 && html.includes('lot-img');\n return { page, isValid };\n }\n return { page, isValid: false };\n } catch (e) {\n return { page, isValid: false };\n }\n });\n \n const results = await Promise.all(pagePromises);\n \n // Find the highest valid page\n let maxValidPage = 1;\n for (const result of results) {\n if (result.isValid && result.page > maxValidPage) {\n maxValidPage = result.page;\n }\n }\n \n // If we found a high valid page, do a more precise binary search in that range\n if (maxValidPage > 1) {\n const preciseMax = await binarySearchMaxPage(auctionUrl, maxValidPage, Math.min(maxValidPage + 10, 50));\n return preciseMax;\n }\n \n return maxValidPage;\n}\n\n/**\n * Binary search for exact max page in a smaller range\n */\nasync function binarySearchMaxPage(auctionUrl: string, low: number, high: number): Promise {\n let lastValidPage = low;\n \n while (low <= high) {\n const mid = Math.floor((low + high) / 2);\n try {\n const testResponse = await fetch(`${auctionUrl}?page=${mid}`);\n if (testResponse.ok) {\n const testHtml = await testResponse.text();\n if (testHtml.length > 100000 && testHtml.includes('lot-img')) {\n lastValidPage = mid;\n low = mid + 1;\n } else {\n high = mid - 1;\n }\n } else {\n high = mid - 1;\n }\n } catch (e) {\n high = mid - 1;\n }\n }\n \n return lastValidPage;\n}",[],[152,217,208],"scrapers","f56fe204-7ace-11f0-b506-0224a6c84d84","mainScraper.ts","1ce292d8-7ace-11f0-a9b5-0224a6c84d84",42,["D",1755368974797],"import type { ScrapedData } from \"../../shared/types.ts\";\nimport { discoverAuctionUrls } from \"./auctionDiscovery.ts\";\nimport { scrapeAuctionItems } from \"./auctionScraper.ts\";\n\n/**\n * Main scraper function - gets all items from all auctions\n */\nexport async function scrapeAllAuctionItems(): Promise {\n const startTime = Date.now();\n \n try {\n // Discover all auction URLs\n const auctionUrls = await discoverAuctionUrls();\n \n if (auctionUrls.length === 0) {\n return {\n timestamp: new Date().toISOString(),\n totalItems: 0,\n totalAuctions: 0,\n items: [],\n error: \"No auction URLs found\"\n };\n }\n \n const allItems = [];\n let totalPagesScraped = 0;\n \n // Process auctions concurrently\n const auctionPromises = auctionUrls.map(async (auctionUrl) => {\n try {\n console.log(`Processing auction: ${auctionUrl}`);\n const auctionResult = await scrapeAuctionItems(auctionUrl);\n console.log(`Completed auction ${auctionUrl}: ${auctionResult.items.length} items, ${auctionResult.pagesScraped} pages`);\n return auctionResult;\n } catch (error) {\n console.log(`Failed to process auction ${auctionUrl}:`, error);\n return { items: [], pagesScraped: 0 };\n }\n });\n \n // Wait for all auctions to complete\n const auctionResults = await Promise.all(auctionPromises);\n \n // Combine results\n for (const result of auctionResults) {\n allItems.push(...result.items);\n totalPagesScraped += result.pagesScraped;\n }\n \n const processingTime = Date.now() - startTime;\n \n return {\n timestamp: new Date().toISOString(),\n totalItems: allItems.length,\n totalAuctions: auctionUrls.length,\n items: allItems,\n metadata: {\n reduxDataFound: true, // We always try REDUX first\n auctionUrlsFound: auctionUrls.length,\n scrapedAuctions: auctionUrls.length,\n totalPagesScraped,\n processingTimeMs: processingTime\n }\n };\n \n } catch (error) {\n console.error(\"Scraping error:\", error);\n return {\n timestamp: new Date().toISOString(),\n totalItems: 0,\n totalAuctions: 0,\n items: [],\n error: error instanceof Error ? error.message : \"Unknown error occurred\"\n };\n }\n}",[],[152,217,219],"3fd17746-7ad3-11f0-aa81-0224a6c84d84","itemExtractor.ts","0be423e8-7ace-11f0-b25a-0224a6c84d84",73,["D",1755370817578],10,"import type { AuctionItem } from \"../../shared/types.ts\";\n\n/**\n * Enhanced extraction that tries Apollo state, then REDUX data, then HTML\n */\nexport async function extractItemsWithGraphQL(html: string, auctionTitle: string, auctionUrl: string): Promise {\n // Try Apollo state first (most detailed data)\n const apolloItems = extractItemsFromApolloState(html, auctionTitle, auctionUrl);\n if (apolloItems.length > 0) {\n console.log(`Extracted ${apolloItems.length} items from Apollo state`);\n return apolloItems;\n }\n \n // Fall back to REDUX data\n const reduxMatch = html.match(/window\\.REDUX_DATA\\s*=\\s*({.*?});/s);\n if (reduxMatch) {\n try {\n const reduxData = JSON.parse(reduxMatch[1]);\n const reduxItems = extractItemsFromReduxData(reduxData, auctionTitle, auctionUrl);\n if (reduxItems.length > 0) {\n console.log(`Extracted ${reduxItems.length} items from REDUX data`);\n return reduxItems;\n }\n } catch (e) {\n console.log(\"Failed to parse REDUX data, falling back to HTML\");\n }\n }\n \n // Final fallback to HTML parsing\n console.log(\"Falling back to HTML parsing\");\n return extractItemsFromHtml(html, auctionTitle, auctionUrl);\n}\n\n/**\n * Extracts items from Apollo state data (most detailed)\n */\nfunction extractItemsFromApolloState(html: string, auctionTitle: string, auctionUrl: string): AuctionItem[] {\n const apolloMatch = html.match(/window\\.__APOLLO_STATE__\\s*=\\s*({.*?});/s);\n if (!apolloMatch) {\n return [];\n }\n \n try {\n const apolloState = JSON.parse(apolloMatch[1]);\n const items: AuctionItem[] = [];\n \n // Look for AuctionLot entries in Apollo state\n for (const [key, value] of Object.entries(apolloState)) {\n if (key.startsWith('AuctionLot.') && value && typeof value === 'object') {\n const lotData = value as any;\n const item = createAuctionItemFromApollo(lotData, auctionTitle, auctionUrl);\n if (item) {\n items.push(item);\n }\n }\n }\n \n return items;\n } catch (e) {\n console.log(\"Failed to parse Apollo state\");\n return [];\n }\n}\n\n/**\n * Creates an AuctionItem from Apollo state lot data\n */\nfunction createAuctionItemFromApollo(lotData: any, auctionTitle: string, auctionUrl: string): AuctionItem | null {\n if (!lotData.title) {\n return null;\n }\n \n const item: AuctionItem = {\n title: lotData.title,\n auctionTitle: lotData.auction?.title || auctionTitle,\n auctionUrl: auctionUrl,\n lotNumber: lotData.auction_lot_id || lotData.lot_number,\n itemUrl: lotData.public_url\n };\n \n // Current bid\n if (lotData.winning_bid_amount && lotData.winning_bid_amount > 0) {\n item.currentBid = `${lotData.winning_bid_amount}`;\n }\n \n // Starting bid\n if (lotData.starting_bid && lotData.starting_bid > 0) {\n item.startingBid = `${lotData.starting_bid}`;\n }\n \n // Bid count\n if (lotData.bid_count !== undefined && lotData.bid_count !== null) {\n item.bidCount = lotData.bid_count;\n }\n \n // Description\n if (lotData.description_plain && lotData.description_plain !== item.title) {\n item.description = lotData.description_plain;\n }\n \n // Image URL - get the first image's medium variant\n if (lotData.images && Array.isArray(lotData.images) && lotData.images.length > 0) {\n const firstImage = lotData.images[0];\n if (firstImage.asset_url_image_variants?.medium) {\n item.imageUrl = firstImage.asset_url_image_variants.medium;\n } else if (firstImage.cached_assets && Array.isArray(firstImage.cached_assets)) {\n // Find medium variant in cached assets\n const mediumAsset = firstImage.cached_assets.find((asset: any) => asset.variant === 'medium');\n if (mediumAsset) {\n item.imageUrl = mediumAsset.url;\n }\n }\n }\n \n // Time calculation from end_time\n if (lotData.end_time) {\n const endTime = new Date(lotData.end_time);\n const now = new Date();\n const timeDiff = endTime.getTime() - now.getTime();\n \n if (timeDiff > 0) {\n const days = Math.floor(timeDiff / (1000 * 60 * 60 * 24));\n const hours = Math.floor((timeDiff % (1000 * 60 * 60 * 24)) / (1000 * 60 * 60));\n const minutes = Math.floor((timeDiff % (1000 * 60 * 60)) / (1000 * 60));\n \n if (days > 0) {\n item.timeLeft = `${days}d ${hours}h ${minutes}m`;\n } else if (hours > 0) {\n item.timeLeft = `${hours}h ${minutes}m`;\n } else {\n item.timeLeft = `${minutes}m`;\n }\n } else {\n item.timeLeft = 'Ended';\n }\n }\n \n // Extract category from dynamic fields\n if (lotData.dynamic_fields && Array.isArray(lotData.dynamic_fields)) {\n for (const field of lotData.dynamic_fields) {\n if (field.label === 'Condition' && field.data?.value) {\n item.category = field.data.value;\n break;\n }\n }\n }\n \n return item;\n}\n\n/**\n * Extracts lot IDs from HTML page\n */\nfunction extractLotIdsFromHtml(html: string): string[] {\n const lotIds: string[] = [];\n \n // Look for lot URLs in links\n const lotUrlPattern = /\\/auctions\\/\\d+\\/lot\\/(\\d+)/g;\n let match;\n while ((match = lotUrlPattern.exec(html)) !== null) {\n if (match[1] && !lotIds.includes(match[1])) {\n lotIds.push(match[1]);\n }\n }\n \n // Look for data attributes with lot IDs\n const dataLotPattern = /data-lot[^=]*=[\"'](\\d+)[\"']/gi;\n while ((match = dataLotPattern.exec(html)) !== null) {\n if (match[1] && !lotIds.includes(match[1])) {\n lotIds.push(match[1]);\n }\n }\n \n // Look for lot IDs in JavaScript/JSON data\n const jsLotPattern = /\"(?:lot_id|lotId|auction_lot_id)\"\\s*:\\s*\"?(\\d+)\"?/gi;\n while ((match = jsLotPattern.exec(html)) !== null) {\n if (match[1] && !lotIds.includes(match[1])) {\n lotIds.push(match[1]);\n }\n }\n \n // Look for lot IDs in REDUX data\n const reduxMatch = html.match(/window\\.REDUX_DATA\\s*=\\s*({.*?});/s);\n if (reduxMatch) {\n try {\n const reduxData = JSON.parse(reduxMatch[1]);\n const reduxLotIds = extractLotIdsFromReduxData(reduxData);\n for (const lotId of reduxLotIds) {\n if (!lotIds.includes(lotId)) {\n lotIds.push(lotId);\n }\n }\n } catch (e) {\n console.log(\"Failed to parse REDUX data for lot IDs\");\n }\n }\n \n return lotIds;\n}\n\n/**\n * Extracts lot IDs from REDUX data\n */\nfunction extractLotIdsFromReduxData(reduxData: any): string[] {\n const lotIds: string[] = [];\n \n const possibleItemSources = [\n reduxData.lots,\n reduxData.items,\n reduxData.auction?.lots,\n reduxData.auction?.items,\n reduxData.lotList,\n reduxData.currentAuction?.lots,\n reduxData.auctionLots\n ];\n \n for (const source of possibleItemSources) {\n if (Array.isArray(source)) {\n for (const item of source) {\n if (item && typeof item === 'object') {\n const lotId = extractLotIdFromItem(item);\n if (lotId && !lotIds.includes(lotId)) {\n lotIds.push(lotId);\n }\n }\n }\n } else if (source && typeof source === 'object') {\n // Handle object with item IDs as keys\n for (const [key, item] of Object.entries(source)) {\n if (item && typeof item === 'object') {\n const lotId = extractLotIdFromItem(item as any) || key;\n if (lotId && !lotIds.includes(lotId)) {\n lotIds.push(lotId);\n }\n }\n }\n }\n }\n \n return lotIds;\n}\n\n/**\n * Extracts lot ID from a single item object\n */\nfunction extractLotIdFromItem(item: any): string | null {\n const lotFields = [\n 'auction_lot_id', 'auctionLotId', 'lot_id', 'lotId', 'lot_number', 'lotNumber',\n 'item_id', 'itemId', 'id', 'lot', 'item_number', 'itemNumber'\n ];\n \n for (const field of lotFields) {\n if (item[field] && item[field].toString().trim()) {\n return item[field].toString();\n }\n }\n \n return null;\n}\n\n/**\n * Extracts items from REDUX data\n */\nexport function extractItemsFromReduxData(reduxData: any, auctionTitle: string, auctionUrl: string): AuctionItem[] {\n const items: AuctionItem[] = [];\n \n // Look for lots/items in various possible locations in the redux state\n const possibleItemSources = [\n reduxData.lots,\n reduxData.items,\n reduxData.auction?.lots,\n reduxData.auction?.items,\n reduxData.lotList,\n reduxData.currentAuction?.lots,\n reduxData.auctionLots\n ];\n \n for (const source of possibleItemSources) {\n if (Array.isArray(source)) {\n for (const item of source) {\n if (item && typeof item === 'object') {\n const auctionItem = createAuctionItem(item, auctionTitle, auctionUrl);\n items.push(auctionItem);\n }\n }\n } else if (source && typeof source === 'object') {\n // Handle object with item IDs as keys\n for (const [key, item] of Object.entries(source)) {\n if (item && typeof item === 'object') {\n const auctionItem = createAuctionItem(item as any, auctionTitle, auctionUrl, key);\n items.push(auctionItem);\n }\n }\n }\n }\n \n return items;\n}\n\n/**\n * Extracts items from HTML\n */\nexport function extractItemsFromHtml(html: string, auctionTitle: string, auctionUrl: string): AuctionItem[] {\n const items: AuctionItem[] = [];\n \n // Look for images with class=\"lot-img\" which contain the item titles in alt text\n const lotImages = html.match(/]*class[^>]*lot-img[^>]*>/gi) || [];\n console.log(`Found ${lotImages.length} lot images`);\n \n for (const imgTag of lotImages) {\n const item: AuctionItem = {\n title: 'Unknown Item',\n auctionTitle,\n auctionUrl\n };\n \n // Extract title from alt attribute\n const altMatch = imgTag.match(/alt=[\"']([^\"']+)[\"']/i);\n if (altMatch && altMatch[1]) {\n item.title = altMatch[1].replace(/&/g, '&').trim();\n }\n \n // Extract image URL\n const srcMatch = imgTag.match(/src=[\"']([^\"']+)[\"']/i);\n if (srcMatch && srcMatch[1]) {\n item.imageUrl = srcMatch[1];\n }\n \n items.push(item);\n }\n \n // Enhance items with additional data from lot containers\n enhanceItemsFromContainers(html, items, auctionTitle, auctionUrl);\n \n // Try to extract prices from the page's JavaScript data or hidden elements\n extractPricesFromPageData(html, items);\n \n // Remove duplicates and items without meaningful data\n const uniqueItems = items.filter((item, index, self) => {\n const isDuplicate = index !== self.findIndex(other => other.title === item.title);\n if (isDuplicate) return false;\n return item.title !== 'Unknown Item' && item.title.length > 3;\n });\n \n console.log(`Extracted ${uniqueItems.length} unique items from HTML`);\n return uniqueItems;\n}\n\n/**\n * Creates an AuctionItem from raw data\n */\nfunction createAuctionItem(itemData: any, auctionTitle: string, auctionUrl: string, key?: string): AuctionItem {\n const auctionItem: AuctionItem = {\n title: itemData.title || itemData.name || itemData.description || `Item ${key || 'Unknown'}`,\n auctionTitle,\n auctionUrl\n };\n \n // Enhanced price extraction from various possible field names\n const priceFields = [\n 'current_bid', 'currentBid', 'current_price', 'currentPrice', 'price', 'bid_amount', 'bidAmount',\n 'high_bid', 'highBid', 'winning_bid', 'winningBid', 'top_bid', 'topBid'\n ];\n \n const startingPriceFields = [\n 'starting_bid', 'startingBid', 'start_bid', 'startBid', 'opening_bid', 'openingBid',\n 'reserve_price', 'reservePrice', 'min_bid', 'minBid', 'starting_price', 'startingPrice'\n ];\n \n // Find current bid\n for (const field of priceFields) {\n if (itemData[field] !== undefined && itemData[field] !== null) {\n let price = itemData[field].toString();\n if (!price.startsWith('$') && parseFloat(price) > 0) {\n price = '$' + price;\n }\n if (price !== '$0' && price !== '$0.00') {\n auctionItem.currentBid = price;\n break;\n }\n }\n }\n \n // Find starting bid\n for (const field of startingPriceFields) {\n if (itemData[field] !== undefined && itemData[field] !== null) {\n let price = itemData[field].toString();\n if (!price.startsWith('$') && parseFloat(price) > 0) {\n price = '$' + price;\n }\n if (price !== '$0' && price !== '$0.00') {\n auctionItem.startingBid = price;\n break;\n }\n }\n }\n \n // Enhanced bid count extraction\n const bidCountFields = [\n 'bid_count', 'bidCount', 'bids', 'num_bids', 'numBids', 'total_bids', 'totalBids',\n 'bidder_count', 'bidderCount', 'bidders'\n ];\n \n for (const field of bidCountFields) {\n if (itemData[field] !== undefined && itemData[field] !== null) {\n const count = parseInt(itemData[field]);\n if (!isNaN(count) && count >= 0) {\n auctionItem.bidCount = count;\n break;\n }\n }\n }\n \n // Enhanced time left extraction\n const timeFields = [\n 'time_left', 'timeLeft', 'remaining_time', 'remainingTime', 'ends_in', 'endsIn',\n 'time_remaining', 'timeRemaining', 'auction_end', 'auctionEnd'\n ];\n \n for (const field of timeFields) {\n if (itemData[field] && itemData[field].toString().trim()) {\n auctionItem.timeLeft = itemData[field].toString().trim();\n break;\n }\n }\n \n // Enhanced lot number extraction\n const lotFields = [\n 'lot_number', 'lotNumber', 'lot_id', 'lotId', 'item_number', 'itemNumber',\n 'id', 'lot', 'item_id', 'itemId', 'auction_lot', 'auctionLot'\n ];\n \n for (const field of lotFields) {\n if (itemData[field] && itemData[field].toString().trim()) {\n auctionItem.lotNumber = itemData[field].toString();\n break;\n }\n }\n \n // Category extraction\n const categoryFields = ['category', 'type', 'classification', 'group', 'section'];\n for (const field of categoryFields) {\n if (itemData[field] && itemData[field].toString().trim()) {\n auctionItem.category = itemData[field].toString().trim();\n break;\n }\n }\n \n // Description extraction\n if (itemData.description && itemData.description !== auctionItem.title) {\n auctionItem.description = itemData.description;\n }\n \n // Image URL extraction\n const imageFields = [\n 'image_url', 'imageUrl', 'image', 'photo_url', 'photoUrl', 'photo', 'picture_url', 'pictureUrl',\n 'thumbnail_url', 'thumbnailUrl', 'thumb_url', 'thumbUrl', 'img_url', 'imgUrl'\n ];\n \n for (const field of imageFields) {\n if (itemData[field] && itemData[field].toString().trim()) {\n auctionItem.imageUrl = itemData[field].toString().trim();\n break;\n }\n }\n \n // Item URL extraction\n const urlFields = [\n 'url', 'lot_url', 'lotUrl', 'item_url', 'itemUrl', 'link', 'permalink', 'href'\n ];\n \n for (const field of urlFields) {\n if (itemData[field] && itemData[field].toString().trim()) {\n const itemUrl = itemData[field].toString().trim();\n auctionItem.itemUrl = itemUrl.startsWith('/') ? \n `https://www.quarterpriceauction.com${itemUrl}` : itemUrl;\n break;\n }\n }\n \n return auctionItem;\n}\n\n/**\n * Extracts prices from JavaScript data or hidden elements in the page\n */\nfunction extractPricesFromPageData(html: string, items: AuctionItem[]): void {\n // Look for JSON data in script tags that might contain pricing\n const scriptMatches = html.match(/]*>(.*?)<\\/script>/gs) || [];\n \n for (const script of scriptMatches) {\n // Skip if it's too large (likely not data)\n if (script.length > 50000) continue;\n \n // Look for price-related JSON data\n const priceDataMatches = script.match(/\\{[^}]*(?:price|bid|amount)[^}]*\\}/gi) || [];\n \n for (const dataMatch of priceDataMatches) {\n try {\n // Try to extract lot number and price from JSON-like structures\n const lotMatch = dataMatch.match(/(?:lot|id)['\":\\s]*(\\d+)/i);\n const priceMatch = dataMatch.match(/(?:price|bid|amount)['\":\\s]*['\"$]?([\\d,.]+)/i);\n \n if (lotMatch && priceMatch) {\n const lotNumber = lotMatch[1];\n const price = '$' + priceMatch[1].replace(/[^\\d.]/g, '');\n \n // Find the corresponding item and update its price\n const item = items.find(item => item.lotNumber === lotNumber);\n if (item && !item.currentBid && parseFloat(price.replace('$', '')) > 0) {\n item.currentBid = price;\n }\n }\n } catch (e) {\n // Ignore parsing errors\n }\n }\n }\n \n // Look for data attributes in HTML elements\n const dataAttributeMatches = html.match(/<[^>]*data-[^>]*(?:price|bid|amount)[^>]*>/gi) || [];\n \n for (const element of dataAttributeMatches) {\n const lotMatch = element.match(/data-(?:lot|id)[='\"]*(\\d+)/i);\n const priceMatch = element.match(/data-(?:price|bid|amount)[='\"]*([\\d,.]+)/i);\n \n if (lotMatch && priceMatch) {\n const lotNumber = lotMatch[1];\n const price = '$' + priceMatch[1].replace(/[^\\d.]/g, '');\n \n const item = items.find(item => item.lotNumber === lotNumber);\n if (item && !item.currentBid && parseFloat(price.replace('$', '')) > 0) {\n item.currentBid = price;\n }\n }\n }\n \n // Look for price information in CSS classes or IDs\n const priceElementMatches = html.match(/<[^>]*(?:class|id)[^>]*(?:price|bid|amount)[^>]*>([^<]*)([^<]*\\$[\\d,.]+[^<]*) item.lotNumber === lotNumber);\n if (item && !item.currentBid) {\n item.currentBid = price[0];\n }\n }\n }\n }\n }\n}\n\n/**\n * Enhances items with additional data from lot containers\n */\nfunction enhanceItemsFromContainers(html: string, items: AuctionItem[], auctionTitle: string, auctionUrl: string): void {\n const lotContainerPatterns = [\n /]*class[^>]*lot[^>]*>[\\s\\S]*?<\\/div>/gi,\n /]*class[^>]*lotList[^>]*>[\\s\\S]*?<\\/div>/gi\n ];\n \n for (const pattern of lotContainerPatterns) {\n const matches = html.match(pattern) || [];\n \n for (const lotHtml of matches.slice(0, 200)) {\n if (lotHtml.length < 100) continue;\n \n const containerImages = lotHtml.match(/]*class[^>]*lot-img[^>]*>/gi) || [];\n if (containerImages.length === 0) continue;\n \n const imgTag = containerImages[0];\n const altMatch = imgTag.match(/alt=[\"']([^\"']+)[\"']/i);\n if (!altMatch) continue;\n \n const title = altMatch[1].replace(/&/g, '&').trim();\n let existingItem = items.find(item => item.title === title);\n \n if (!existingItem) {\n existingItem = {\n title,\n auctionTitle,\n auctionUrl\n };\n \n const srcMatch = imgTag.match(/src=[\"']([^\"']+)[\"']/i);\n if (srcMatch && srcMatch[1]) {\n existingItem.imageUrl = srcMatch[1];\n }\n \n items.push(existingItem);\n }\n \n // Extract additional data\n enhanceItemWithContainerData(existingItem, lotHtml);\n }\n }\n}\n\n/**\n * Enhances a single item with data from its container HTML\n */\nfunction enhanceItemWithContainerData(item: AuctionItem, containerHtml: string): void {\n // Look for prices with more aggressive patterns\n const pricePatterns = [\n /\\$[\\d,]+\\.?\\d*/g, // Standard $123.45 or $123\n /\\$\\s*[\\d,]+\\.?\\d*/g, // $123.45 with spaces\n /[\\d,]+\\.?\\d*\\s*dollars?/gi, // 123.45 dollars\n /price[:\\s]*\\$?[\\d,]+\\.?\\d*/gi, // price: $123.45\n /bid[:\\s]*\\$?[\\d,]+\\.?\\d*/gi, // bid: $123.45\n /current[:\\s]*\\$?[\\d,]+\\.?\\d*/gi, // current: $123.45\n /starting[:\\s]*\\$?[\\d,]+\\.?\\d*/gi, // starting: $123.45\n ];\n \n let foundPrices: string[] = [];\n \n for (const pattern of pricePatterns) {\n const matches = containerHtml.match(pattern) || [];\n foundPrices.push(...matches);\n }\n \n // Clean and deduplicate prices\n foundPrices = [...new Set(foundPrices)]\n .map(price => {\n // Normalize price format\n let cleaned = price.replace(/[^\\d.,]/g, '');\n if (cleaned && !cleaned.startsWith('$')) {\n cleaned = '$' + cleaned;\n }\n return cleaned;\n })\n .filter(price => price && price !== '$' && parseFloat(price.replace(/[$,]/g, '')) > 0);\n \n // Assign prices based on context and availability\n if (foundPrices.length > 0 && !item.currentBid) {\n // Look for context clues to determine which price is which\n const currentBidMatch = containerHtml.match(/current[^$]*(\\$[\\d,]+\\.?\\d*)/i);\n const startingBidMatch = containerHtml.match(/start[^$]*(\\$[\\d,]+\\.?\\d*)/i);\n \n if (currentBidMatch) {\n item.currentBid = currentBidMatch[1];\n } else if (foundPrices.length > 0) {\n // Use the first/highest price as current bid\n item.currentBid = foundPrices[0];\n }\n \n if (startingBidMatch) {\n item.startingBid = startingBidMatch[1];\n } else if (foundPrices.length > 1) {\n // Use the second/lower price as starting bid\n item.startingBid = foundPrices[foundPrices.length - 1];\n }\n }\n \n // Look for lot numbers with more patterns\n const lotNumberPatterns = [\n containerHtml.match(/lot[^0-9]*(\\d+)/i),\n containerHtml.match(/item[^0-9]*(\\d+)/i),\n containerHtml.match(/#(\\d+)/),\n containerHtml.match(/data-lot[^0-9]*(\\d+)/i),\n containerHtml.match(/id[^0-9]*(\\d+)/i),\n containerHtml.match(/(\\d{6,})/), // 6+ digit numbers (likely lot numbers)\n ];\n \n for (const pattern of lotNumberPatterns) {\n if (pattern && pattern[1] && !item.lotNumber) {\n item.lotNumber = pattern[1];\n break;\n }\n }\n \n // Look for links to individual lot pages\n const linkMatch = containerHtml.match(/]*href=[\"']([^\"']+)[\"']/i);\n if (linkMatch && linkMatch[1] && !item.itemUrl) {\n let itemUrl = linkMatch[1];\n if (itemUrl.startsWith('/')) {\n itemUrl = 'https://www.quarterpriceauction.com' + itemUrl;\n }\n if (itemUrl.includes('lot') || itemUrl.includes('item')) {\n item.itemUrl = itemUrl;\n }\n }\n \n // Look for bid count with more patterns\n const bidCountPatterns = [\n containerHtml.match(/(\\d+)\\s*bids?/i),\n containerHtml.match(/bids?[:\\s]*(\\d+)/i),\n containerHtml.match(/(\\d+)\\s*bidders?/i),\n ];\n \n for (const pattern of bidCountPatterns) {\n if (pattern && pattern[1] && !item.bidCount) {\n item.bidCount = parseInt(pattern[1]);\n break;\n }\n }\n \n // Look for time left with more patterns\n const timeLeftPatterns = [\n containerHtml.match(/(\\d+[dhms])/i),\n containerHtml.match(/(\\d+:\\d+:\\d+)/i),\n containerHtml.match(/(\\d+\\s*(?:days?|hours?|minutes?|mins?|seconds?|secs?))/i),\n containerHtml.match(/time[^:]*:\\s*([^<]+)/i),\n containerHtml.match(/ends?[^:]*:\\s*([^<]+)/i),\n ];\n \n for (const pattern of timeLeftPatterns) {\n if (pattern && pattern[1] && !item.timeLeft) {\n item.timeLeft = pattern[1].trim();\n break;\n }\n }\n}",[],[152,217,227],"f62b490a-7ad2-11f0-8695-0224a6c84d84","graphqlItemScraper.ts","a7d0c852-7ad2-11f0-aa51-0224a6c84d84",70,["D",1755370694014],"import type { AuctionItem } from \"../../shared/types.ts\";\n\n/**\n * GraphQL query to get individual lot details\n */\nconst GET_LOT_QUERY = `\nquery get_lot($auction_lot_id: String!) {\n lot(auction_lot_id: $auction_lot_id) {\n auction_id\n auction_lot_id\n lot_number\n title\n description\n description_plain\n winning_bid_amount\n starting_bid\n bid_count\n start_time\n end_time\n quantity\n images {\n asset_url_image_variants {\n medium\n large\n }\n }\n auction {\n title\n }\n dynamic_fields {\n label\n data {\n value\n }\n }\n category {\n category_id\n }\n public_url\n }\n}`;\n\n/**\n * Scrapes individual item details using GraphQL API\n */\nexport async function scrapeItemWithGraphQL(auctionLotId: string): Promise {\n try {\n const response = await fetch('https://www.quarterpriceauction.com/api', {\n method: 'POST',\n headers: {\n 'Content-Type': 'application/json',\n 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0',\n 'Accept': '*/*',\n 'x-crunch': '1'\n // Removed x-app-token to test without authentication\n },\n body: JSON.stringify({\n operationName: 'get_lot',\n variables: { auction_lot_id: auctionLotId },\n query: GET_LOT_QUERY\n })\n });\n\n if (!response.ok) {\n console.log(`GraphQL request failed for lot ${auctionLotId}: ${response.status}`);\n // Let's also log the response text to see what the error is\n const errorText = await response.text();\n console.log(`Error response: ${errorText}`);\n return null;\n }\n\n const data = await response.json();\n \n if (!data.data?.lot) {\n console.log(`No lot data found for ${auctionLotId}`);\n console.log(`Response data:`, JSON.stringify(data, null, 2));\n return null;\n }\n\n return convertGraphQLToAuctionItem(data.data.lot);\n \n } catch (error) {\n console.log(`Error fetching lot ${auctionLotId} via GraphQL:`, error);\n return null;\n }\n}\n\n/**\n * Converts GraphQL lot data to AuctionItem format\n */\nfunction convertGraphQLToAuctionItem(lotData: any): AuctionItem {\n const item: AuctionItem = {\n title: lotData.title || 'Unknown Item',\n lotNumber: lotData.auction_lot_id || lotData.lot_number,\n auctionTitle: lotData.auction?.title || 'Unknown Auction',\n itemUrl: lotData.public_url || `https://www.quarterpriceauction.com/auctions/${lotData.auction_id}/lot/${lotData.auction_lot_id}`,\n };\n\n // Current bid amount\n if (lotData.winning_bid_amount && lotData.winning_bid_amount > 0) {\n item.currentBid = `$${lotData.winning_bid_amount}`;\n }\n\n // Starting bid\n if (lotData.starting_bid && lotData.starting_bid > 0) {\n item.startingBid = `$${lotData.starting_bid}`;\n }\n\n // Bid count\n if (lotData.bid_count !== undefined && lotData.bid_count !== null) {\n item.bidCount = lotData.bid_count;\n }\n\n // Description\n if (lotData.description_plain && lotData.description_plain !== item.title) {\n item.description = lotData.description_plain;\n }\n\n // Image URL - prefer medium size\n if (lotData.images && lotData.images.length > 0) {\n const firstImage = lotData.images[0];\n if (firstImage.asset_url_image_variants?.medium) {\n item.imageUrl = firstImage.asset_url_image_variants.medium;\n } else if (firstImage.asset_url_image_variants?.large) {\n item.imageUrl = firstImage.asset_url_image_variants.large;\n }\n }\n\n // Time left calculation\n if (lotData.end_time) {\n const endTime = new Date(lotData.end_time);\n const now = new Date();\n const timeDiff = endTime.getTime() - now.getTime();\n \n if (timeDiff > 0) {\n const days = Math.floor(timeDiff / (1000 * 60 * 60 * 24));\n const hours = Math.floor((timeDiff % (1000 * 60 * 60 * 24)) / (1000 * 60 * 60));\n const minutes = Math.floor((timeDiff % (1000 * 60 * 60)) / (1000 * 60));\n \n if (days > 0) {\n item.timeLeft = `${days}d ${hours}h ${minutes}m`;\n } else if (hours > 0) {\n item.timeLeft = `${hours}h ${minutes}m`;\n } else {\n item.timeLeft = `${minutes}m`;\n }\n } else {\n item.timeLeft = 'Ended';\n }\n }\n\n // Extract category from dynamic fields\n if (lotData.dynamic_fields && Array.isArray(lotData.dynamic_fields)) {\n for (const field of lotData.dynamic_fields) {\n if (field.label === 'Condition' && field.data?.value) {\n item.category = field.data.value;\n break;\n }\n }\n }\n\n // Set auction URL based on auction_id\n if (lotData.auction_id) {\n item.auctionUrl = `https://www.quarterpriceauction.com/auctions/${lotData.auction_id}`;\n }\n\n return item;\n}\n\n/**\n * Batch scrape multiple items using GraphQL\n */\nexport async function scrapeItemsBatchWithGraphQL(auctionLotIds: string[]): Promise {\n const items: AuctionItem[] = [];\n const batchSize = 5; // Limit concurrent requests to avoid rate limiting\n \n console.log(`Scraping ${auctionLotIds.length} items via GraphQL in batches of ${batchSize}`);\n \n for (let i = 0; i < auctionLotIds.length; i += batchSize) {\n const batch = auctionLotIds.slice(i, i + batchSize);\n \n const batchPromises = batch.map(lotId => scrapeItemWithGraphQL(lotId));\n const batchResults = await Promise.allSettled(batchPromises);\n \n for (const result of batchResults) {\n if (result.status === 'fulfilled' && result.value) {\n items.push(result.value);\n }\n }\n \n // Small delay between batches to be respectful\n if (i + batchSize < auctionLotIds.length) {\n await new Promise(resolve => setTimeout(resolve, 200));\n }\n }\n \n console.log(`Successfully scraped ${items.length} items via GraphQL`);\n return items;\n}",[],[152,217,236],"c43d30c0-7ad2-11f0-8cdc-0224a6c84d84","auctionScraper.ts","16820a4a-7ace-11f0-90f1-0224a6c84d84",66,["D",1755370610241],6,"import type { AuctionItem } from \"../../shared/types.ts\";\nimport { extractItemsFromReduxData, extractItemsFromHtml, extractItemsWithGraphQL } from \"./itemExtractor.ts\";\nimport { detectMaxPages } from \"./paginationDetector.ts\";\n\n/**\n * Scrapes all items from a single auction across all pages\n */\nexport async function scrapeAuctionItems(auctionUrl: string): Promise<{items: AuctionItem[], pagesScraped: number}> {\n const items: AuctionItem[] = [];\n let pagesScraped = 0;\n \n try {\n console.log(`Starting to scrape auction: ${auctionUrl}`);\n \n // Get the first page\n const response = await fetch(auctionUrl);\n if (!response.ok) {\n throw new Error(`HTTP error! status: ${response.status}`);\n }\n \n const html = await response.text();\n pagesScraped++;\n \n // Extract auction title\n const auctionTitleMatch = html.match(/]*>([^<]+)<\\/title>/i);\n const auctionTitle = auctionTitleMatch ? \n auctionTitleMatch[1].replace(/Quarter Price.*?-\\s*/, '').trim() : \n auctionUrl.split('/').pop()?.replace(/-/g, ' ') || 'Unknown Auction';\n \n // Extract items from the first page\n let pageItems = await extractItemsFromPage(html, auctionTitle, auctionUrl);\n items.push(...pageItems);\n \n console.log(`Found ${pageItems.length} items on page 1 of ${auctionUrl}`);\n \n // Detect how many pages this auction has\n const maxPage = await detectMaxPages(auctionUrl, html, pageItems.length);\n \n // Scrape remaining pages concurrently\n if (maxPage > 1) {\n const pagePromises = [];\n \n for (let page = 2; page <= Math.min(maxPage, 50); page++) {\n const pagePromise = scrapePageConcurrently(auctionUrl, page, auctionTitle);\n pagePromises.push(pagePromise);\n }\n \n // Wait for all pages to complete\n const pageResults = await Promise.allSettled(pagePromises);\n \n // Process results\n for (let i = 0; i < pageResults.length; i++) {\n const page = i + 2; // Pages start at 2\n const result = pageResults[i];\n \n if (result.status === 'fulfilled') {\n const { items: pageItems, success } = result.value;\n if (success && pageItems.length > 0) {\n items.push(...pageItems);\n pagesScraped++;\n console.log(`Found ${pageItems.length} items on page ${page} of ${auctionUrl}`);\n } else if (!success) {\n console.log(`Page ${page} appears to be empty or invalid`);\n }\n } else {\n console.log(`Error scraping page ${page} of ${auctionUrl}:`, result.reason);\n }\n }\n }\n \n console.log(`Completed scraping ${auctionUrl}: ${items.length} total items across ${pagesScraped} pages`);\n \n } catch (error) {\n console.log(`Error scraping auction ${auctionUrl}:`, error);\n }\n \n return { items, pagesScraped };\n}\n\n/**\n * Extracts items from a single page using GraphQL API when possible\n */\nasync function extractItemsFromPage(html: string, auctionTitle: string, auctionUrl: string): Promise {\n // Try GraphQL extraction first (this will fall back to REDUX/HTML if needed)\n return await extractItemsWithGraphQL(html, auctionTitle, auctionUrl);\n}\n\n/**\n * Checks if a page has valid content\n */\nfunction isValidPage(html: string): boolean {\n return html.length > 100000 && \n html.includes('lot-img') && \n !html.includes('No items found') && \n !html.includes('No lots found');\n}\n\n/**\n * Scrapes a single page concurrently\n */\nasync function scrapePageConcurrently(auctionUrl: string, page: number, auctionTitle: string): Promise<{items: AuctionItem[], success: boolean}> {\n try {\n const pageUrl = `${auctionUrl}?page=${page}`;\n \n const pageResponse = await fetch(pageUrl);\n if (!pageResponse.ok) {\n return { items: [], success: false };\n }\n \n const pageHtml = await pageResponse.text();\n \n // Check if page has content\n if (!isValidPage(pageHtml)) {\n return { items: [], success: false };\n }\n \n // Extract items from this page\n const pageItems = await extractItemsFromPage(pageHtml, auctionTitle, auctionUrl);\n \n return { items: pageItems, success: pageItems.length > 0 };\n \n } catch (error) {\n console.log(`Error scraping page ${page} of ${auctionUrl}:`, error);\n return { items: [], success: false };\n }\n}",[],[152,217,244],"f396fc16-7acd-11f0-b736-0224a6c84d84","auctionDiscovery.ts","f396fd06-7acd-11f0-b736-0224a6c84d84",["D",1755368542201],"/**\n * Discovers all auction URLs from the main page\n */\nexport async function discoverAuctionUrls(): Promise {\n try {\n const response = await fetch(\"https://www.quarterpriceauction.com/\");\n \n if (!response.ok) {\n throw new Error(`HTTP error! status: ${response.status}`);\n }\n \n const html = await response.text();\n \n // Extract auction URLs from the HTML\n const auctionUrls = [...new Set((html.match(/href=[\"'][^\"']*\\/auctions\\/[^\"']*[\"']/g) || [])\n .map(m => m.replace(/href=[\"']/, '').replace(/[\"']$/, ''))\n .filter(url => url.includes('/auctions/') && !url.includes('#'))\n .map(url => url.startsWith('/') ? `https://www.quarterpriceauction.com${url}` : url))];\n \n console.log(`Found ${auctionUrls.length} auction URLs`);\n return auctionUrls;\n \n } catch (error) {\n console.error(\"Error discovering auction URLs:\", error);\n return [];\n }\n}",[],[152,217,253],"f397072e-7acd-11f0-b736-0224a6c84d84",["D",1755368542201],[170,171,172,173,174,175],[152,217],["D",1755368542201],[169,176],[152],["D",1754848079739],{"_95":94,"_102":98},{"_95":94,"_96":-5,"_97":98,"_18":53,"_99":100,"_101":25},{"_13":84,"_32":85,"_42":41,"_86":43,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":-5,"_92":44,"_93":94},"ctx",{"_77":-5,"_122":303,"_304":305,"_306":307,"_308":25,"_309":25,"_310":25,"_311":57,"_312":-5,"_313":-7,"_314":-7,"_315":61,"_316":25,"_317":25,"_318":-5,"_319":25,"_320":25,"_321":25,"_322":-5},"ENV",{"_274":275,"_276":277,"_278":85,"_279":280,"_281":282,"_283":284,"_285":286,"_287":288,"_289":290,"_291":292,"_293":294,"_295":296,"_297":298,"_299":300,"_301":302},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","e87eb04e707cf64e7e532c8e8ec395f209e17f15","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]

Diff · v76