This val automates IndexNow submission by:

1. Crawling sitemap URLs.
2. Tracking submission history in project-scoped SQLite.
3. Submitting only changed URLs with explicit limits at different stages (per-site run URL limit and IndexNow POST chunk size).

0. Read the IndexNow docs (indexnow.org/documentation) and make sure your key file is already hosted on your domain at a public URL like https://www.example.org/<key>.txt.
1. Add at least one site file in config/sites/*.json with host, sitemapUrl, and keyLocation (see Site Config Files).
2. Set required env vars: INDEXNOW_RUN_TOKEN (and INDEXNOW_FORCE_TOKEN if you want force-mode protection) (see Environment Variables).
3. Open / to confirm the dashboard loads.
4. Run a safe dry run first: POST /run?dryRun=1 with x-indexnow-token (see API Reference).
5. After dry run looks correct, enable cron via indexnow.cron.ts for ongoing automation.

• INDEXNOW_PER_SITE_RUN_URL_LIMIT (default 5000): per-site cap for queued changed URLs in a single run.
• IndexNow POST payload size (10,000 URLs): protocol hard limit per HTTP request to IndexNow.
• Monitor query runs (default 50, max 500): number of run-history records returned, not URL count.

• indexnow.http.ts: primary HTTP entrypoint.
  • / serves the dashboard UI.
  • /run performs IndexNow submission runs (JSON response).
  • /monitor serves monitor JSON.
• indexnow.cron.ts: scheduled trigger that calls the HTTP handler at /run.
• indexnow.monitor.http.ts: monitor implementation module used by indexnow.http.ts.

The recommended setup is a cron trigger every hour for sitemap polling, plus /run for immediate/manual control.

Run history stores why a run happened using reason:

• manual_run_endpoint: run came through HTTP POST /run.
• cron_entrypoint: run came through indexnow.cron.ts.

This is entrypoint provenance. It does not distinguish scheduled cron fires from someone manually running the cron file. To keep provenance meaningful, treat indexnow.cron.ts as scheduler-only and use /run for all manual runs.

• INDEXNOW_RUN_TOKEN (required): auth token required for /run (manual and cron).
• INDEXNOW_FORCE_TOKEN (optional but recommended): separate auth token required when using force=1.
• INDEXNOW_ENDPOINT (optional): defaults to https://api.indexnow.org/indexnow.
• INDEXNOW_PER_SITE_RUN_URL_LIMIT (optional): defaults to 5000; per-site queue cap per run before batching into IndexNow POSTs.
• INDEXNOW_RUN_URL_LIMIT (legacy alias): still accepted for backward compatibility; remove from val settings after migrating.
• INDEXNOW_MAX_URLS_PER_RUN (legacy alias): still accepted for backward compatibility; remove from val settings after migrating.
• INDEXNOW_ALERTS_ENABLED (optional): defaults to enabled; set to 0/false to disable alert emails.
• INDEXNOW_ALERT_TO_<SITE> (optional): per-site alert recipient email (for example INDEXNOW_ALERT_TO_HANAYOU). If unset, alerts for that site go to the val owner email.

Site configs live in config/sites/*.json with one site per file. At least one file must exist or the val will throw on startup.

Example: config/sites/hanayou.studio.json

{
  "host": "www.hanayou.studio",
  "sitemapUrl": "https://www.hanayou.studio/sitemap.xml",
  "keyLocation": "https://www.hanayou.studio/2dc12b205bfe46d19a75077e3991f7ce.txt",
  "alertToEnvVar": "INDEXNOW_ALERT_TO_HANAYOU",
  "key": "optional-explicit-key",
  "includePrefixes": ["/journal/", "/guides/"],
  "excludePrefixes": ["/tag/", "/author/"]
}

Notes:

• Keep raw email addresses out of config/sites/*.json.
• Use alertToEnvVar to point to an environment variable that contains the recipient email.
• If alertToEnvVar is omitted (or env var is empty), the val sends alerts to the default owner email.

Base URL pattern: https://<your-val>.web.val.run

Auth model:

• POST /run is token-protected with INDEXNOW_RUN_TOKEN.
• POST /run?force=1 additionally requires INDEXNOW_FORCE_TOKEN.
• GET / and GET /monitor are read-only/public.

Time semantics:

• JSON timestamps are ISO-8601 UTC strings.
• The dashboard (GET /) renders those timestamps in the viewer's local browser timezone.

Purpose: Render the HTML operator dashboard.

Auth: none.

Query parameters:

Response:

• 200 text/html dashboard UI.
• Runtime/config errors bubble as server errors.

Example:

https://<your-val>.web.val.run/?site=www.hanayou.studio&runs=50

Purpose: Execute sitemap discovery and IndexNow submission workflow.

Auth headers:

Query parameters:

Status codes:

Pragmatic response fields:

• Top-level: reason (manual_run_endpoint or cron_entrypoint), dryRun, force, runUrlLimit, hasErrors, startedAt, finishedAt.
• Per site (siteRuns[]): host, filteredCount, changedCount, queuedCount, submittedCount, pendingCount, deferredCount, errors.
• Per site URL arrays: filteredUrls, changedUrls, queuedUrls.

Examples:

Dry run (safe):

curl -X POST "https://<your-val>.web.val.run/run?dryRun=1&site=www.hanayou.studio" \
  -H "x-indexnow-token: <INDEXNOW_RUN_TOKEN>"

Real run:

curl -X POST "https://<your-val>.web.val.run/run?site=www.hanayou.studio" \
  -H "x-indexnow-token: <INDEXNOW_RUN_TOKEN>"

Force run:

curl -X POST "https://<your-val>.web.val.run/run?force=1&site=www.hanayou.studio" \
  -H "x-indexnow-token: <INDEXNOW_RUN_TOKEN>" \
  -H "x-indexnow-force-token: <INDEXNOW_FORCE_TOKEN>"

Purpose: Return JSON health/status data for automation, debugging, and external checks.

Auth: none.

Query parameters:

Status codes:

Pragmatic response fields:

• Top-level: generatedAt, runHistoryLimit, issuesOnly, selectedHost, selectedRunId.
• Site aggregate status: totals (especially upToDateUrls, needsSubmissionUrls, retryPendingUrls).
• Run history: sites[].latestRun, sites[].recentRuns.
• Drill-down details: sites[].selectedRunUrlStatuses (when run is provided and data exists).

Example:

curl "https://<your-val>.web.val.run/monitor?site=www.hanayou.studio&runs=500&issues=1"

• submittedCount counts successful IndexNow HTTP 200 responses only.
• pendingCount tracks URLs from IndexNow HTTP 202 responses.
• deferredCount means URLs were eligible/changed but capped by INDEXNOW_PER_SITE_RUN_URL_LIMIT in that run.

SQLite tables:

• indexnow_url_state_v1: per-URL state (seen/submitted timestamps, lastmod, status).
• indexnow_run_log_v1: per-run summaries for observability and debugging.

This val uses https://esm.town/v/std/sqlite/main.ts so data is scoped to this val project.

• INDEXNOW_PER_SITE_RUN_URL_LIMIT is applied per site for each run.
• If more changed URLs are discovered than the limit, the remainder is deferred to future runs.
• To intentionally resubmit unchanged URLs (still within limit), use force=1 with x-indexnow-force-token.

1. Daily/incident check:
   • Open / dashboard and confirm site health cards are green.
   • Review recent runs for pending, deferred, or Error statuses.
2. Manual safe run flow:
   • Run dry run first: POST /run?dryRun=1.
   • Review changedCount and queuedCount.
   • If expected, run real submission: POST /run.
3. If deferred URLs accumulate:
   • Confirm this is expected (large content release, migration, etc.).
   • Tune INDEXNOW_PER_SITE_RUN_URL_LIMIT and/or schedule frequency.
   • Use force=1 only when you intentionally want to resubmit unchanged URLs.
4. Alert triage:
   • Submission alert with no issues: informational (URLs submitted successfully).
   • Alert with pending: retry/watch next run; investigate endpoint throttling.
   • Alert with deferred: consider run URL-limit tuning or additional runs.
   • Alert with errors: requires action before trusting indexing progress.
5. External verification:
   • Use Bing Webmaster Tools to verify ingestion trends (expect lag).