URL Archiver

Archive one or more URLs into a rich, per-provider record. The page type is detected automatically and routed to a matching extraction provider. Any HTTP method is accepted; URLs come via query params or body (JSON or form).

Try it: Try the app

Endpoint: https://url-archiver.val.run

# Single URL (GET)
curl "https://url-archiver.val.run/?url=https://github.com/denoland/deno"

# Many URLs (GET, repeated params)
curl "https://url-archiver.val.run/?url=https://a.com&url=https://b.com"

# Many URLs (POST, JSON)
curl -X POST https://url-archiver.val.run \
  -H 'content-type: application/json' \
  -d '{"url":["https://a.com","https://b.com"],"options":{"wordsPerMinute":250}}'

URL aliases (any may be a string or array): url, urls, site, sites, website, website, link, links.

Recognized types & providers

kind	Handled URLs	Source of metadata
`git`	GitHub, GitLab, Bitbucket, Codeberg/Gitea	provider repo APIs (stars, forks, topics, …)
`music`	Spotify, Deezer, SoundCloud, Bandcamp, YouTube Music	oEmbed + ID extraction
`music`	MusicBrainz	MusicBrainz `ws/2` API (keyless)
`music`	ListenBrainz, Last.fm	page meta (+ Last.fm API if key set)
`video`	YouTube, Vimeo, Dailymotion	oEmbed + ID extraction
`wikipedia`	`*.wikipedia.org/wiki/…`	article extract + REST `summary` (keyless)
`paper`	`doi.org/…`, arXiv, PubMed	Crossref / arXiv Atom / NCBI eutils (all keyless)
`file`	direct asset downloads (`*.pdf`, images, …)	—
`article`	everything else	`@extractus/article-extractor`

Example outputs

// https://github.com/denoland/deno
{ "kind": "git", "owner": "denoland", "repo": "deno", "language": "Rust",
  "stars": 108390, "forks": 6373, "license": "MIT", "topics": ["deno", "typescript", "rust"], … }

// https://doi.org/10.1038/s41586-020-2649-2
{ "kind": "paper", "doi": "10.1038/s41586-020-2649-2", "title": "Array programming with NumPy",
  "authors": ["Charles R. Harris", "…"], "journal": "Nature", "published": "2020-9-16", "abstract": "…", … }

// https://musicbrainz.org/recording/a4fd9f68-…
{ "kind": "music", "musicbrainz_entity": "recording", "title": "Never Gonna Give You Up",
  "artist_credit": "Rick Astley", "duration_ms": 91000, … }

Optional env vars

No credentials are required for any provider — everything public/keyless works out of the box. The only optional key enables richer Last.fm enrichment:

LASTFM_API_KEY — enrich Last.fm artist/album/track pages via the Last.fm API.

👉 Add LASTFM_API_KEY here: https://www.val.town/x/heidi/url-archiver/environment-variables?key=LASTFM_API_KEY

Layout

main.ts — HTTP handler, URL collection, classification & routing
src/classify.ts — URL → kind detection
src/git.ts, src/music.ts, src/video.ts, src/paper.ts, src/wikipedia.ts — providers
src/util.ts — shared helpers (oEmbed, meta-tag parsing, entity decoding)

Rendering mermaid diagram...

Everything is best-effort: if a provider API is down or the page doesn't match, the archiver still returns a useful base record (IDs, host, original URL, title) rather than failing.