This val automates IndexNow submission by:

1. Crawling sitemap URLs.
2. Tracking submission history in project-scoped SQLite.
3. Submitting only changed URLs in batches (up to 10,000 URLs per POST).

• indexnow.http.ts: primary HTTP entrypoint.
  • / serves the dashboard UI.
  • /run performs IndexNow submission runs (JSON response).
  • /monitor serves monitor JSON (or dashboard via ?view=dashboard if needed).
• indexnow.cron.ts: scheduled trigger that calls the HTTP handler at /run.
• indexnow.monitor.http.ts: monitor implementation module used by indexnow.http.ts.

The recommended setup is a cron trigger every hour for sitemap polling, plus /run for immediate/manual control.

Run history stores why a run happened using reason:

• manual_run_endpoint: run came through HTTP POST /run.
• cron_entrypoint: run came through indexnow.cron.ts.

This is entrypoint provenance. It does not distinguish scheduled cron fires from someone manually running the cron file. To keep provenance meaningful, treat indexnow.cron.ts as scheduler-only and use /run for all manual runs.

• INDEXNOW_RUN_TOKEN (required): auth token required for /run (manual and cron).
• INDEXNOW_FORCE_TOKEN (optional but recommended): separate auth token required when using force=1.
• INDEXNOW_ENDPOINT (optional): defaults to https://api.indexnow.org/indexnow.
• INDEXNOW_MAX_URLS_PER_RUN (optional): defaults to 5000.
• INDEXNOW_CIRCUIT_BREAKER_MAX_URLS (optional): defaults to 200; blocks unusually large non-force submissions.
• INDEXNOW_ALERTS_ENABLED (optional): defaults to enabled; set to 0/false to disable alert emails.

Site configs live in config/sites/*.json with one site per file. At least one file must exist or the val will throw on startup.

Example: config/sites/hanayou.studio.json

{
  "host": "www.hanayou.studio",
  "sitemapUrl": "https://www.hanayou.studio/sitemap.xml",
  "keyLocation": "https://www.hanayou.studio/2dc12b205bfe46d19a75077e3991f7ce.txt",
  "key": "optional-explicit-key",
  "includePrefixes": ["/journal/", "/guides/"],
  "excludePrefixes": ["/tag/", "/author/"]
}

• Dashboard: /
• Run endpoint: /run
• /run is POST only and requires auth:
  • Authorization: Bearer <INDEXNOW_RUN_TOKEN> or x-indexnow-token: <INDEXNOW_RUN_TOKEN>
• /run defaults run reason to manual_run_endpoint.
• x-indexnow-reason: cron_entrypoint is reserved for the cron entrypoint.
• Dry run: POST /run?dryRun=1
• Force resubmit currently eligible URLs: POST /run?force=1
  • also requires x-indexnow-force-token: <INDEXNOW_FORCE_TOKEN>
• On non-dry runs, the val emails the owner when:
  • any URLs are successfully submitted, or
  • there are attention conditions (errors, pending, deferred, or circuit breaker blocks).
• Target a single configured site: POST /run?site=www.hanayou.studio
• Response includes stage-specific URL arrays:
• siteRuns[].filteredUrls for URLs that passed filters.
• siteRuns[].changedUrls for URLs detected as changed.
• siteRuns[].queuedUrls for changed URLs after run-cap limiting.
• siteRuns[].submittedCount counts only HTTP 200 responses.
• HTTP 202 responses are tracked as pending in siteRuns[].pendingCount.
• siteRuns[].circuitBreakerTriggered indicates the submission was blocked by the circuit breaker.

Example:

curl -X POST "https://<your-val>.web.val.run/run?dryRun=1&site=www.hanayou.studio" \
  -H "x-indexnow-token: <INDEXNOW_RUN_TOKEN>"

Force example:

curl -X POST "https://<your-val>.web.val.run/run?force=1&site=www.hanayou.studio" \
  -H "x-indexnow-token: <INDEXNOW_RUN_TOKEN>" \
  -H "x-indexnow-force-token: <INDEXNOW_FORCE_TOKEN>"

• Path: /monitor on indexnow.http.ts (JSON default)
• Purpose: IndexNow health snapshot from config + SQLite state
• Query params:
• site (optional): limit report to one configured host.
• runs (optional): recent run history per site (default 50, max 500).
• view (optional): dashboard for HTML UI, otherwise JSON.

Response highlights:

• totals.everSubmitted200Urls: URLs that have ever had a confirmed HTTP 200 submission.
• totals.upToDateUrls: URLs whose latest sitemap lastmod is already covered by the last successful submission.
• totals.needsSubmissionUrls: URLs never successfully submitted, or changed since last successful submission.
• totals.retryPendingUrls: URLs that need submission and whose latest attempt was a non-200/202 failure.
• sites[].latestRun: latest run summary per site.
• sites[].recentRuns: recent run history per site.

Dashboard example:

https://<your-val>.web.val.run/?site=www.hanayou.studio&runs=50

SQLite tables:

• indexnow_url_state_v1: per-URL state (seen/submitted timestamps, lastmod, status).
• indexnow_run_log_v1: per-run summaries for observability and debugging.

This val uses https://esm.town/v/std/sqlite/main.ts so data is scoped to this val project.

• Circuit breaker applies only to non-dry, non-force runs.
• If queued URLs for a site exceed INDEXNOW_CIRCUIT_BREAKER_MAX_URLS, the run is blocked before any IndexNow POST happens.
• The run logs an error and marks circuitBreakerTriggered: true for that site.
• To intentionally override for bulk submissions, use force=1 with x-indexnow-force-token.

1. Daily/incident check:
   • Open / dashboard and confirm site health cards are green.
   • Review recent runs for pending, deferred, or Error statuses.
2. Manual safe run flow:
   • Run dry run first: POST /run?dryRun=1.
   • Review changedCount and queuedCount.
   • If expected, run real submission: POST /run.
3. If circuit breaker triggers:
   • Confirm the spike is legitimate (content release, migration, etc.).
   • Prefer tuning INDEXNOW_MAX_URLS_PER_RUN/schedule before forcing.
   • Only then use force=1 with x-indexnow-force-token.
4. Alert triage:
   • Submission alert with no issues: informational (URLs submitted successfully).
   • Alert with pending: retry/watch next run; investigate endpoint throttling.
   • Alert with deferred: consider run-cap tuning or additional runs.
   • Alert with errors/breaker: requires action before trusting indexing progress.
5. External verification:
   • Use Bing Webmaster Tools to verify ingestion trends (expect lag).