Avatar

tmcw

đź‘· Building Val Town
158 public vals
Joined August 31, 2022

NYC Charging Stations Data Analysis

CleanShot 2023-09-21 at 12.07.43@2x.png

https://tmcw-nychargingstations.web.val.run/

This analyzes some open data about electric car charging stations in New York State and bakes a website from it. Data is messy!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import { fetch } from "https://esm.town/v/std/fetch";
export const nyChargingStations = (async () => {
const { csvParse } = await import("npm:d3-dsv");
const { group } = await import("npm:d3-array");
const { micromark } = await import("npm:micromark");
const rows = await fetch(
"https://data.ny.gov/api/views/7rrd-248n/rows.csv?accessType=DOWNLOAD&sorting=true",
)
.then((r) => r.text())
.then(csvParse);
const facets = ["Groups With Access Code", "Access Days Time", "ZIP"];
let output = `# New York City Charging Station Analytics
From: [NYC Open Data](https://data.ny.gov/Energy-Environment/Electric-Vehicle-Charging-Stations-in-New-York/7rrd-248n) \n\n`;
for (let facet of facets) {
const groups = group(rows, (r) => r[facet]);
output += `# ${facet}\n` +
Array.from(groups.entries(), ([k, v]) => [k, v.length]).sort((
[a, b],
[a1, b1],
) => b1 - b)
.map(([a, b]) => `- ${a}: ${b}`)
.join("\n") +
"\n\n";
}
return new Response(
`<link rel="stylesheet" href="https://unpkg.com/missing.css@1.1.1"><main>` +
micromark(output),
{
headers: {
"Content-Type": "text/html",
},
},
);
});

This tracks the data produced by my iPhone trade-in values val.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
// set by tmcw.trackIphoneTradein at 2023-10-14T05:08:46.031Z
export let iphoneTradeInValues = [{
"date": "2023-09-13T18:38:18.512Z",
"rows": [{
"name": "iPhone 14 Pro Max",
"val": "Up to $650"
}, {
"name": "iPhone 14 Pro",
"val": "Up to $570"
}, {
"name": "iPhone 14 Plus",
"val": "Up to $470"
}, {
"name": "iPhone 14",
"val": "Up to $430"
}, {
"name": "iPhone SE (3rd generation)",
"val": "Up to $160"
}, {
"name": "iPhone 13 Pro Max",
"val": "Up to $580"
}, {
"name": "iPhone 13 Pro",
"val": "Up to $480"
}, {
"name": "iPhone 13",
"val": "Up to $370"
}, {
"name": "iPhone 13 mini",
"val": "Up to $320"
}, {
"name": "iPhone 12 Pro Max",
"val": "Up to $450"
}, {
"name": "iPhone 12 Pro",
"val": "Up to $360"
}, {
"name": "iPhone 12",
"val": "Up to $250"
}, {
"name": "iPhone 12 mini",
"val": "Up to $200"
}, {
"name": "iPhone SE (2nd generation)",
"val": "Up to $80"
}, {
"name": "iPhone 11 Pro Max",
"val": "Up to $300"
}, {
"name": "iPhone 11 Pro",
"val": "Up to $250"
}, {
"name": "iPhone 11",
"val": "Up to $200"
}, {
"name": "iPhone XS Max",
"val": "Up to $170"
}, {
"name": "iPhone XS",
"val": "Up to $140"
}, {
"name": "iPhone XR",
"val": "Up to $140"
}, {
"name": "iPhone X",
"val": "Up to $120"
}, {
"name": "iPhone 8 Plus",
"val": "Up to $90"
}, {
"name": "iPhone 8",
"val": "Up to $60"
}, {
"name": "iPhone 7 Plus",
"val": "Up to $50"
}, {
"name": "iPhone 7",
"val": "Up to $40"
}]
}, {
"date": "2023-10-14T05:08:45.994Z",
"rows": [{
"name": "iPhone 14 Pro Max",
"val": "Up to $650"
}, {
"name": "iPhone 14 Pro",
"val": "Up to $570"
}, {
"name": "iPhone 14 Plus",
"val": "Up to $470"
}, {
"name": "iPhone 14",
"val": "Up to $430"
}, {
"name": "iPhone SE (3rd generation)",
"val": "Up to $160"
}, {
"name": "iPhone 13 Pro Max",
"val": "Up to $580"
}, {

Track iPhone trade-in prices

iPhone trade in values - to trade phones back to Apple themselves - fluctuate over time, and eventually they no longer accept certain kinds of phones. You can still trade in an iPhone 7 - released in 2016, about 6 years ago as of this writing. But no longer does an iPhone 6 have value back to Apple.

This tracks those values - it runs every month and records prices to iphoneTradeInValues, which I'll visualize once there's some data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import { fetch } from "https://esm.town/v/std/fetch";
export const trackIphoneTradein = async () => {
const cheerio = await import("npm:cheerio@1.0.0-rc.12");
const page = await fetch(
"https://www.apple.com/shop/browse/overlay/tradein_landing/iphone_values",
).then((r) => r.text());
const $ = cheerio.load(page);
const rows = $("tr").map((row, elem) => {
const [name, val] = $(elem).find("td").map((idx, elem) => $(elem).text())
.toArray();
return { name, val };
}).toArray().filter((row) => row.name && row.val);
return rows;
};

The story behind HTTP 200 "OK"

What's in an HTTP response? I've been writing software for the web since the early 2000s and have incrementally learned things about HTTP. There are status codes, like "200" and "404". There are headers, for Content-Type and headers to control cache settings. There are different versions of HTTP itself, like 1.1, 2, and 3. HTTP requests and responses can contain data, in the message body.

hello.png

But there's one thing I didn't notice until yesterday. A quirk that was included in the HTTP 1.1 specification with an authors note that it's mostly there for historical reasons: the reason-phrase.

None of this information is useful. The reason-phrase is barely supported on the web and was always an oddity, but keep reading if you like oddities!

If you're used to JavaScript’s fetch() method to make HTTP requests, you've seen the reason-phrase under a different name: statusText:

Create val(await fetch('https://example.com/')).statusText

What is statusText? I had assumed that it was something that JavaScript itself provides, by looking up the status code 200 and matching it with the text "OK". I was wrong!

When I look at a raw HTTP response, I see the first few lines are like this:

HTTP/1.1 200 OK
Date: Thu, 17 Aug 2023 15:16:42 GMT
Content-Type: text/plain;charset=UTF-8

The reason phrase

So what is that text? I dug around in the HTTP 1.0 specification and found the section Status Code and Reason Phrase.

The Status-Code element is a 3-digit integer result code of the attempt to understand and satisfy the request. The Reason-Phrase is intended to give a short textual description of the Status-Code. The Status-Code is intended for use by automata and the Reason-Phrase is intended for the human user. The client is not required to examine or display the Reason-Phrase.

That also lists recommended reason phrases, like OK for 200 and Not Found for 404. And notes that you can choose different phrases without affecting the protocol.

The HTTP 1.1 specification adds a little color about the reason-phrase:

quote.png

So, with a HTTP server, you can customize your reason phrase! Here's an example with a val on Val Town:

Create vallet customReason = (req) => new Response("", { statusText: 'Hello world!', });

Unfortunately, this doesn't work! The response that Val Town produces is reorganized and optimized by Cloudflare, which upgrades requests and responses from HTTP 1.1 to HTTP 2. And sadly, HTTP 2 dropped support for the custom reason-phrase.

RIP the reason-phrase. It was present even in a 1992 draft of the HTTP specification, and was a weird and under-appreciated way to pilfer extra information in a response. Now, thanks to HTTP/2 and the commonplace use of proxies and CDNs like Cloudflare, it's no longer usable. It was fun while it lasted.

1
2
3
4
5
6
export let reasonPhrase = () => {
return new Response("Hello world", {
status: 200,
statusText: "This can be customized!",
});
};

WebFinger

This is a not-quite-complete ActivityPub implementation based on my blog post about building an AP implementation. It includes enough to look up bot@tmcw-activitypub.web.val.run on Mastodon and get some basic information.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
export const activitypub = async (req: Request) => {
const { Hono } = await import("npm:hono@3.4.3");
const app = new Hono();
const DOMAIN = "tmcw-activitypub.web.val.run";
const USERS = new Map([
[
`acct:bot@${DOMAIN}`,
{
subject: `acct:bot@${DOMAIN}`,
aliases: [],
links: [
{
rel: "http://webfinger.net/rel/profile-page",
type: "text/html",
href: `https://${DOMAIN}/profile`,
},
{
rel: "self",
type: "application/activity+json",
href: `https://${DOMAIN}/u/bot`,
},
],
},
],
]);
const publicKey = `-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA7LBs3Qyuh93lRboTNXLN
hj4n92oK5Qg4oS8Cc81AXh04hD7nQSSKBhtarbHy2yPXeiKA+H3EGbcflsLvZCo2
B3OPNo2nGTCMyJM8HWDf/7JCOHHcy4tZC1ldjrItkb1YDABWwfoXxyBiGTyTVXjL
sBX5ArTGUPwctMSOdxlJp0ttFn5WDIHiPzxbSaEX/fzTy+HKr9RvYPu/hWWpXA/W
8QQRacZjslweupZFGCGPX1zJ+P0FSe81uV6N5cOPpy+vFkBQrvApwCSIyp/n7Rfq
UtU+zi/ru+wSxyvnoZPZa+zOXst8+pk7lIbmI6dyJ2+wijkykAxKt2DnDXWOSUGM
R+aNjc6tt8xp2MwmhAz91f1rIt2+jOhkPZ0m6aLV3B86J3iI0BIHXzQNydDtz5/P
EOj79vnnDwjCrWmbsfsIVCmECQDS7EW6Lzdc98GyiD/vyA4Epg3QgpeN4r7fELZj
8IfJJ7J8Z8nYewRoCVNnfvXpR26y+CLftMUi9LtPP1N78er1IdvZEer/8RIAc58r
S5VmDYBBfEduxPh/l3tn4A5Ri8smue26yG+wPkBj3CSqaOaNFxxZPgXcbI2OePrH
81goKk17g+5O0sZJGv+EAeFM1OQPXKqyu0DLY6PHJKGSho/B/BNWQ34vZnQhQF1r
++VZAcLEeqny/Cn6CHoeu5cCAwEAAQ==
-----END PUBLIC KEY-----`;
app.get("/u/bot", (ctx) => {
return ctx.json({
"@context": [
"https://www.w3.org/ns/activitystreams",
"https://w3id.org/security/v1",
],
id: `https://${DOMAIN}/u/bot`,
type: "Person",
preferredUsername: `bot`,
summary: `Photos from ${DOMAIN}.com`,
inbox: `https://${DOMAIN}/api/inbox`,
followers: `https://${DOMAIN}/u/bot/followers`,
icon: {
type: "Image",
mediaType: "image/jpeg",
url: "https://macwright.com/graphics/about.jpg",
},
publicKey: {
id: `https://${DOMAIN}/u/bot#main-key`,
owner: `https://${DOMAIN}/u/bot`,
publicKeyPem: publicKey,
},
});
});
app.get("/.well-known/webfinger", (ctx) => {
const u = new URL(ctx.req.url);
const resource = u.searchParams.get("resource");
if (resource && USERS.has(resource)) {
return new Response(JSON.stringify(USERS.get(resource)), {
status: 200,
headers: { "Content-Type": "application/jrd+json" },
});
}
return new Response("", {
status: 404,
headers: { "Content-Type": "text/html" },
});
});
return app.fetch(req);
};

Generate a pdf with pdf-lib

👉 Visit the web site generated by this Val

pdf-lib is an incredible pure-JavaScript module that generates PDFs! In this example, it's generating a simple page that embeds the PDF, and exposes another route (with Hono) that serves the PDF.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
export const pdfExample = (async (req) => {
const { Hono } = await import("npm:hono");
const app = new Hono();
app.get("/", async (c) => {
return new Response(
`<html><body><h1>This PDF is generated with Val Town:</h1>
<iframe src='/pdf' width="100%" height="500"></iframe></body>
</html>`,
{
headers: {
"Content-Type": "text/html; charset=utf-8",
},
},
);
});
app.get("/pdf", async (c) => {
const { PDFDocument, StandardFonts, rgb } = await import("npm:pdf-lib");
// Create a new PDFDocument
const pdfDoc = await PDFDocument.create();
// Embed the Times Roman font
const timesRomanFont = await pdfDoc.embedFont(StandardFonts.TimesRoman);
// Add a blank page to the document
const page = pdfDoc.addPage();
// Get the width and height of the page
const { width, height } = page.getSize();
// Draw a string of text toward the top of the page
const fontSize = 30;
page.drawText("Creating PDFs in JavaScript is awesome!", {
x: 50,
y: height - 4 * fontSize,
size: fontSize,
font: timesRomanFont,
color: rgb(0, 0.53, 0.71),
});
// Serialize the PDFDocument to bytes (a Uint8Array)
const pdfBytes = await pdfDoc.save();
return new Response(pdfBytes, {
headers: {
"Content-Type": "application/pdf",
},
});
});
return app.fetch(req);
});

Response with ReadableStream example

Demonstrates our ability to handle streaming responses. We do, in practice, read the streams and then return the response - basically buffering - but with the support for the Response object we include the option to pass a ReadableStream instance as the first argument.

Note that it's required that the ReadableStream returns Uint8Array items.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
export let streamExample = (() => {
const te = new TextEncoder();
let iterator = "<h1>Test</h1>".split("").map((t) => te.encode(t)).values();
const rs = new ReadableStream({
async pull(controller) {
const { value, done } = await iterator.next();
if (done) {
controller.close();
}
else {
controller.enqueue(value);
}
},
});
return new Response(rs, {
headers: {
"Content-Type": "text/html",
},
});
});
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import { fetch } from "https://esm.town/v/std/fetch";
export let blocks_inner = async (req, res) => {
const [_, user, id, ...file] = req.path.split("/");
const gist = await fetch(`https://api.github.com/gists/${id}`).then((r) =>
r.json()
);
res.set("Cache-Control", "public,max-age=64800");
if (file.filter(Boolean).length) {
if (gist.files[file.filter(Boolean).join("/")].raw_url.endsWith(".js")) {
res.set("Content-Type", "application/javascript");
return res.send(
await fetch(gist.files[file.filter(Boolean).join("/")].raw_url).then(
(r) => r.text()
),
);
}
return res.redirect(gist.files[file.filter(Boolean).join("/")].raw_url);
}
const index = gist.files["index.html"];
if (!index) {
return res.end({ message: "Gist did not contain index.html" });
}
return res.send(index.content.replace("http://d3js.org", "https://d3js.org"));
};
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import { fetch } from "https://esm.town/v/std/fetch";
import { big_stories_ranks } from "https://esm.town/v/tmcw/big_stories_ranks";
import process from "node:process";
export let big_story = (async () => {
const nytimes =
await (await fetch(
`https://api.nytimes.com/svc/topstories/v2/home.json?api-key=${process.env.nytimes_api_key}`,
)).json();
const now = Date.now();
const newDataPoints = nytimes.results.slice(0, 20).forEach(
({ title, url, created_date, section, uri }, rank) => {
if (big_stories_ranks[url]) {
big_stories_ranks[url].ranks.push([now, rank]);
}
else {
big_stories_ranks[url] = {
title,
url,
section,
ranks: [[now, rank]],
};
}
},
);
for (let url of Object.keys(big_stories_ranks)) {
// Prune stories over 3 days ago
if (
big_stories_ranks[url].ranks.every((rank) => {
return rank[0] < Date.now() - (1000 * 60 * 60 * 24 * 3);
})
) {
delete big_stories_ranks[url];
}
}
});

This val is simple, and just a way to test the express API.

1
export let express_example = (req, res) => res.send("hi");