${esc(diag.title)}

[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_166":167,"_168":169},"routes/_app.x.$handle.$name",{"_11":12,"_13":14,"_15":-7,"_18":19,"_17":-7,"_126":-5,"_20":127,"_22":128,"_129":130,"_131":132,"_26":133,"_24":134,"_135":-5,"_136":-5,"_137":51,"_138":139,"_140":43,"_141":-5,"_142":-5,"_143":144},"routes/_app.x.$handle.$name.code.$",{"_11":12,"_13":14,"_15":16,"_17":-7,"_18":19,"_20":21,"_22":23,"_24":25,"_26":27,"_28":-5,"_29":-5,"_30":-5,"_31":-5,"_32":-5},"handle","anmeimei","name","us-topnews-merge","branchName","main","version","path","README.md","project",{"_33":70,"_13":14,"_44":71,"_72":-5,"_73":51,"_74":75,"_76":75,"_77":78,"_79":-5,"_80":81,"_82":58,"_83":43,"_84":-5,"_85":-5,"_86":87,"_88":89,"_90":-5,"_91":-5,"_92":-5,"_93":-5,"_94":95,"_96":95,"_97":98,"_99":98,"_100":98,"_101":95,"_102":95,"_103":104,"_105":106,"_107":108,"_109":51},"tree",[54,55],"file",{"_33":34,"_13":19,"_35":36,"_37":-5,"_38":39,"_40":41,"_42":43,"_44":45,"_17":46,"_47":24,"_48":49,"_50":51,"_52":-5,"_18":53},"contents",[],"upstream","pull","ahead","behind","envVars","id","a220d9a9-12d5-4e34-baf5-47e2d1b03d70","fileId","5b5faa18-a80f-11f1-8197-1607ee4eb77e","parentId","branchId","23607ccd-58ae-46e5-8871-5f1a571d8515","branchVersion",29,"latest",true,"createdAt",["D",1789140945989],10,"fileType","content","# us-topnews-merge\n\nVal Town: https://www.val.town/x/anmeimei/us-topnews-merge Base:\nhttps://anmeimei--1349d12aa7df11f1abc71607ee4eb77e.web.val.run\n\n| 路由 | 内容 |\n| --------- | -------------------------------------------- |\n| / | CNBC Top + CBS Top 合并去重（重复保留 CNBC） |\n| /politics | CNBC 政治与政策，扣掉已在 Top News 出现过的 |\n| /economy | CNBC 经济，同上 |\n| /tech | CNBC 科技，同上 |\n\n源：\n\n- CNBC Top https://www.cnbc.com/id/100003114/device/rss/rss.html\n- CBS Top https://www.cbsnews.com/latest/rss/main\n- Politics https://www.cnbc.com/id/10000113/device/rss/rss.html\n- Economy https://www.cnbc.com/id/20910258/device/rss/rss.html\n- Tech https://www.cnbc.com/id/19854910/device/rss/rss.html\n\n参数（任何路由通用，多个参数用 & 连接，只有第一个用 ?）： ?sim=0.45 阈值 ?tag=0\n去来源前缀 ?diag=0 去诊断条目 ?debug=1 诊断网页\n\nfeed 第一条是「合并诊断」，置顶。guid 见文末 2026-09-04 那节（已改成不带日期）。\n\n## 实测 2026-09-03\n\n/ CNBC30 + CBS30 → 保留 56，丢弃 4 /politics CNBC30 → 保留 22，丢弃 8 /economy\nCNBC30 → 保留 28，丢弃 2 /tech CNBC30 → 保留 21，丢弃 9（9\n条全部是「同一篇」精确命中）\n\n板块 vs Top News 基本都是同一篇文章（链接相同），走精确匹配，不靠相似度猜。跨\nCBS/CNBC 才需要近似匹配。sim=0.36 会误杀 CBS 理财软文标题，故默认 0.45。\n\n## 2026-09-04 修复：标题完全相同但日期相隔很远的重复\n\nCBS 的 latest feed 会把 60 Minutes 旧片段按**原始播出日期**重发\n（实测出现过同一条 \"When Social Security mistakenly overpays benefits\" 的\n2025-07-27 / 2024-06-23 / 2023-12-20 三份）。原来的近似判重有 WINDOW_HOURS=36\n的时间窗，相隔几年的两条永远不会被拿来比较，所以三条全留下了。\n\n修法：在近似比较之前加一道「标题归一化后完全相同 → 同一篇」，这一步不看日期。\n归一化 = 转小写 + 非字母数字全替换成空格 + trim。零误判风险。\n本地用那三条真实标题跑过，2 条被丢、1 条保留，走的是 same-title 分支。\n\n注意 CBS 这类旧稿是它自己有意重推的内容，不是脏数据，所以只去重、不按年龄过滤。\n\n## 2026-09-04 诊断条目修正（已上线）\n\n两处改动，均在 `diagEntry()`：\n\n1. **日期按美东算**，不再用 UTC。\n `new Date().toLocaleDateString(\"en-CA\", { timeZone: \"America/New_York\" })`\n 之前用 `toISOString().slice(0,10)`，美东晚 8 点后 UTC 已经跳到第二天，\n 所以标题会显示 9/4 而实际是 9/3。\n\n2. **诊断条目的 guid 去掉日期**：`diag-${base.pathname}`。每个 feed\n 永远只有唯一一条「合并诊断」，内容随刷新就地更新，不再每天新增、不会积攒。\n\n验证（2026-09-03 ET）：\n\n| 路由 | guid | 诊断条数 | item 总数 |\n| ----------- | ---------------- | -------- | --------- |\n| `/` | `diag-/` | 1 | 57 |\n| `/politics` | `diag-/politics` | 1 | 24 |\n| `/economy` | `diag-/economy` | 1 | 30 |\n| `/tech` | `diag-/tech` | 1 | 21 |\n\n副作用：改 guid 会在 NetNewsWire 里各留下一条旧的孤儿诊断（四个 feed 各一条），\nNNW 会自行清掉。这是最后一次——以后 guid 固定，不会再产生孤儿。\n\n## 2026-09-04 查证：三个板块之间要不要互相去重 → 不需要\n\n问题：Tech / Politics / Economy 之间是否也有重复，要不要做成级联去重（Tech 只扣\nTop；Politics 扣 Top+Tech；Economy 扣 Top+Tech+Politics）。\n\n实测（当天四个源各 30 条，按文章 URL 比对）：\n\n| 比对 | 重叠 |\n| ------------------- | ---- |\n| Tech ∩ Politics | 1 |\n| Tech ∩ Economy | 0 |\n| Politics ∩ Economy | 0 |\n| Tech ∩ Top News | 10 |\n| Politics ∩ Top News | 7 |\n| Economy ∩ Top News | 1 |\n\n那唯一 1 条（Musk 中期选举花钱）本身就在 Top News 里，两个板块已经各自扣掉了，\n所以实际可见的跨板块重复是 0 条。又把模糊阈值放宽到 0.40 并去掉 36\n小时时间窗重跑，三个板块两两之间零个近似标题。\n\n原因是结构性的：CNBC 每篇文章基本只归一个板块，Top News\n是跨板块挑出来的一层覆盖，所以「板块 ↔ Top」重叠大，「板块 ↔ 板块」几乎不重叠。\n\n结论：不做级联。理由不只是没必要，还有代价—— 跨板块用 sim=0.45 会误杀（Economy\n一篇关税报道 vs Politics 另一篇关税报道\n是两篇不同文章，不是重复），且引入顺序依赖、/economy 每次多抓两个源。以每天 0~1\n条真重复换这些，不划算。\n\n如果以后真的反复看到同一篇出现两次，再加**只按 URL 精确匹配、不做模糊**的级联，\n那个零误杀风险。注意：这个结论基于一天的切片（90 条）， Musk\n那条证明「一篇挂两个板块」是可能的，只是罕见。\n\n## 2026-09-06 方向反转：改成「Top News 扣板块」\n\n### 起因\n\n用户在 Top News 和 CNBC: Tech 里看到同一条新闻。查下来去重逻辑没坏（当时 /tech\n的诊断里就把它列为「同一篇」丢弃了），但暴露了两个成因：\n\n**成因一（竞态）**：CNBC 先把稿子发到板块，过一会儿才提拔进 Top News。\nNetNewsWire 在这个空窗里抓了板块 feed，就存下一份；等它进了 Top News 又存一份。\n\n**成因二（结构性，更严重）**：CNBC Top News 只有 30 条、约 1.5 天翻一轮，而板块\nfeed 能追到两三周前。用短窗口当参照系，绝大部分板块条目根本没被检查过。\n2026-09-06 实测：\n\n| feed | 时间跨度 | 比 Top News 窗口还老的条目 |\n| ------------------ | ------------------------- | -------------------------- |\n| Top News（参照系） | 09-04 11:00 → 09-06 00:24 | — |\n| Tech | 09-02 12:43 → 09-04 14:47 | 21 / 26（81%） |\n| Politics | 09-01 18:53 → 09-04 19:48 | 21 / 24（88%） |\n| Economy | 08-19 21:29 → 09-04 17:16 | 25 / 26（96%） |\n\n### 改法\n\n**反过来：Top News 扣掉三个板块，板块恢复全量。**\n\n参照系（三个板块合起来 08-19 → 09-05，2.5 周）完整覆盖被过滤对象（Top News\n09-04 → 09-05，1.5 天），成因二的窟窿直接消失，不需要持久化存储。\n\n实测 CNBC Top 30 条里 14 条（47%）已经在某个板块里，反转后 Top News 从 57 条降到\n43 条。\n\n### 三个配套决定\n\n**一、Top News 扣板块只做精确链接 / 同标题匹配，不做相似度。** 主 feed\n误杀一条的代价远大于漏掉一条重复。CNBC 与 CBS 之间仍然用 sim=0.45。\n\n**二、板块条目压后 6 小时再输出（?delay=）。** 等 CNBC 把该提拔的提拔完，★\n第一次渲染就准。\n用户读完就标已读、不会回头看，所以「首次准确」比「事后补上」重要。\n代价：板块新闻晚 6 小时出现，这段时间它在任何 feed 里都看不到。\n（用户明确说不介意——他只用 NetNewsWire 每天 catch-up 一次，不做实时。）\n\n**三、板块里也上了 Top News 的条目标 ★（?star=0 关掉）。**\n反转之后，编辑部「这是今天头条」的信号会随着条目移出 Top News 而丢失， ★\n把这个信号保留在用户真正会读完的地方。\n\n### 剩下没治的\n\nCNBC 偶尔先发 Top News、后发板块的话，那一刻 Top News 会显示它，\n等板块拿到才扣掉 —— NNW 两边各存一份。治它只能延迟 Top News，不划算。从实测看\nCNBC 是先板块后头条（最近 12 小时属于板块的 5\n条全都已在板块里），这个方向应该很少。\n\n### 实测 2026-09-05（美东）\n\n| 路由 | 条目 | 带 ★ |\n| ----------- | ------------- | ---- |\n| `/` | 43（丢弃 17） | — |\n| `/tech` | 30 | 4 |\n| `/politics` | 30 | 6 |\n| `/economy` | 29 | 3 |\n\n`?delay=24` 时 /tech 变成 27 条，压后功能正常。\n\n## 2026-09-07 剔除 CBS 的电视片段（`/video/`）\n\n### 起因\n\n用户反馈：CBS Top News 每天推送太密集，而且「很多什么 9/11\n的好多条，这些我确实不 care」。 CNBC 的密度刚刚好，但它基本只覆盖钱相关的事，缺\nLindsay Clancy 这类社会新闻。\n\n### 判断依据\n\nCBS 的 feed 里有很大一部分条目链接是 `/video/`（电视节目片段），其余是\n`/news/`（文字稿）。两次抽样：主 feed 25 条里 11 条是 `/video/`；这次上线后 CBS\n抓到 30 条里 16 条是 `/video/`——大约一半。\n\n把 `/video/` 条目和同一天的 `/news/`\n条目配对，**没有配上的残余基本都不是硬新闻**：\n\n| video-only 残余（抽样） | 类型 |\n| ------------------------------------------------- | -------------------- |\n| 2 friends hit the road for a good cause | 软性栏目 |\n| Exclusive discounts from CBS Mornings Deals | 带货广告 |\n| Meet the psychiatrist behind \"therapy speak\" | 访谈片段 |\n| Is quitting good for you? | 软性栏目 |\n| American farmers sell herds over high costs | 节目片段 |\n| 9/6: CBS Weekend News / 9/6: Face The Nation | 整期节目片头 |\n| 9/6/2026: Sharswood; Boom Chicago | 地方台整期节目 |\n| Connecticut school honors Sept. 11 attacks victim | 纪念类 |\n| Lindsay Clancy 庭审片段 | **真新闻，唯一一条** |\n\n硬新闻（飓风、空难、庭审）文字和视频两种格式都会有；video-only\n的漏网基本是软内容。\n\n唯一一条真新闻 Lindsay Clancy，在 cbsnews.com 上搜到大量 `/news/` 文章\n（`/news/lindsay-clancy-mistrial-what-happens-next/`、\n`/news/trump-lindsay-clancy-case-mistrial-hung-jury-horrible-thing/`，还有若干\n`/boston/news/…`），\n说明这类连续跟进的大案子不会因为剔掉视频而丢——今天漏了明天还在。\n\n用户自己的判断（「CBS 的 text 和 video 基本都讲一件事儿，真正重要的新闻两个\nformat 都会有」）\n结论对，理由只对一半：不是每条视频都有对应文字，而是**没有对应文字的那些本来就不值得读**。\n\n### 实现\n\n```ts\nconst isVideo = (it: Item) => /\\/video\\//.test(it.link || \"\");\n```\n\n`collect()` 加了 `keepVideo` 参数，默认剔除，`?video=1` 保留。 `counts`\n统计的是过滤后的数量（所以诊断里 CBS 显示 14 而不是 30），\n剔掉多少条单独写在诊断第一段：「剔除 CBS 电视片段 N 条」。\n\n板块路由的 `videoOut` 只算板块自己的（永远是 0，CNBC 没有 video 链接），\n不把「Top News 参照系剔掉的」算进去——那会让 /tech 的诊断显示一个跟它无关的数字。\n\n### 实测 2026-09-07（美东）\n\n| 路由 | 条目 | `/video/` 链接 |\n| ----------- | ---------------------- | -------------- |\n| `/` | 32（31 条新闻 + 诊断） | 0 |\n| `/tech` | 30 | 0（4 条带 ★） |\n| `/politics` | 29 | 0 |\n| `/economy` | 31 | 0 |\n\n`?video=1` 恢复：Top News 41 条新闻，其中 11 条是 `/video/`。四个路由都没有 NaN\n/ undefined。\n\n### 密度参考（当时测的其他候选源）\n\n| 源 | 每天条数 |\n| ---------------------- | -------- |\n| CBS Top News（未过滤） | ~30 |\n| CNBC Top News | ~20 |\n| PBS NewsHour | ~19 |\n| NPR News | ~8 |\n\n## 2026-09-11 Top News 加发布时间上限（`?maxage=`）\n\n### 起因\n\n用户两三天没 catch-up，回来发现 US Top News 里混着 4 月 / 7 月 / 8 月的 CBS\n文章，而且清楚记得这几天反复标过已读。\n\n### 查证（读了 Mac 上 NetNewsWire 的 iCloud 库，只读副本）\n\n该 feed 171 条未读，按**到达日期**分组：\n\n| 到达日 | 条数 |\n| ------ | ---- |\n| 09-04 | 1 |\n| 09-08 | 7 |\n| 09-09 | 63 |\n| 09-10 | 69 |\n| 09-11 | 31 |\n\n那 11 条\"很久远\"的条目，发布日期从 2026-04-30 到 08-29，**到达日期全部是\n09-09**。\n\n所以不是同步丢失，也不是标过又变回未读：CBS 在 09-09 往 feed 里吐了一批旧存档，\nNetNewsWire 按发布日期排序，它们一进来就沉到列表最底部，从上往下读永远扫不到。\n\n佐证：09-11 当天 CBS 源里仍挂着一条发布日期 2024-08-01 的「The 60 Minutes\nChannel」，只是恰好是\n`/video/`，被视频过滤器顺手挡掉了。这个源确实会往外吐旧条目。\n\n### 改法\n\nTop News 的输出加发布时间上限，默认 10 天，`?maxage=` 可调，`?maxage=0` 不限。\n**只作用于 `/` 自己的条目**——不作用于板块参照集，也不作用于板块路由。\n\n为什么板块不加：Top News\n是「现在在发生什么」，十天前的东西躺在列表底部对读者零价值。板块不是。实测\n09-11 当天 `/economy` 最老一条发布于 08-25（17 天前），加 10\n天上限会直接砍掉一大半；而且板块的时间深度还要拿去给 Top News 当去重参照系，\n砍短了会让 2026-09-06 那次反转解决的窗口盲区重新出现。\n\n### 实测 2026-09-11（美东）\n\n| 路由 | 条目 | 最老发布日 | 备注 |\n| ------------ | ---- | ----------- | ---------------------------------------- |\n| `/` | 34 | 09-10 | 默认 maxage=10，当天没有超龄条目 |\n| `/?maxage=1` | 33 | 09-10 20:28 | 剔除 1 条，诊断第一段有提示 |\n| `/?maxage=2` | 34 | 09-10 01:26 | 剔除 0 条 |\n| `/tech` | 24 | 09-09 | 不受影响 |\n| `/politics` | 25 | 09-08 | 不受影响 |\n| `/economy` | 28 | **08-25** | 不受影响（这条就是不给板块加上限的理由） |\n\n四个路由都没有 NaN / undefined。\n\n### 部署踩的坑（以后改这个 val 的人看这里）\n\nVal Town 存盘时会跑 prettier，把长语句折成多行。用「整行匹配 +\n在该行之后插入」的方式\n改代码时，如果目标语句在线上已经被折成多行，插入点会落在语句中间、把它劈成两半——\n这次就把 `const delayH =` 和它下面的三元表达式劈开了，线上直接 500。\n改之前先把目标区域**按行打印出来**确认实际排版，不要假设它和本地文件长得一样。\n\n## 相关（同一套 RSS 系统的其他 val）\n\n- 财新滚动要闻 / 晨读荐闻：val `anmeimei/caixin-briefing-rss`\n- 财新 T早报（tzaobao.ts）也在同一个 val 里\n- MIT Tech Review 三个 AI feed 合并去重：val `anmeimei/mit-tr-ai`","deleted",false,"label",[19],{"_33":34,"_13":19,"_35":36,"_37":-5,"_38":39,"_40":41,"_42":43,"_44":67,"_17":46,"_47":24,"_48":49,"_50":51,"_52":-5,"_64":68,"_18":69},{"_33":56,"_13":57,"_35":58,"_37":-5,"_38":39,"_40":59,"_42":43,"_44":60,"_17":61,"_47":62,"_48":63,"_50":51,"_52":-5,"_64":65,"_18":66},"41230479-d468-45cc-b8bd-2e8b791cac14","main.ts","1349d12a-a7df-11f1-abc7-1607ee4eb77e",28,["D",1789140796980],17,"http","/**\n * CNBC / CBS 新闻聚合去重\n *\n * 路由：\n * / Top News: CNBC Top + CBS Top 合并去重，再扣掉三个板块里已有的\n * /politics CNBC 政治与政策，全量\n * /economy CNBC 经济，全量\n * /tech CNBC 科技，全量\n *\n * 方向是「Top News 扣板块」，不是反过来。原因：CNBC Top News 只有 30 条、约 1.5 天就翻一轮，\n * 而板块 feed 能追到两周前。拿短窗口当参照系会漏掉八到九成六的条目（2026-09-06 实测）。\n * 反过来，板块的时间跨度完整覆盖 Top News，参照系就不再漏。\n *\n * 板块条目压后 delay 小时（默认 6）再输出，等 CNBC 把该提拔的提拔进 Top News，\n * 这样 ★（表示这条也上了 Top News）第一次渲染就是准的 —— NetNewsWire 会就地更新已存文章，\n * 但用户读完就标已读、不会回头看，所以首次准确比事后补上重要。\n * 代价是板块新闻晚 delay 小时出现，期间它在任何 feed 里都看不到。\n *\n * Top News 扣板块时只做精确链接 / 同标题匹配，不做相似度 —— 主 feed 误杀一条的代价太大。\n * CNBC 与 CBS 之间仍然用相似度合并（两家写同一件事用词不同）。\n *\n * 参数（任何路由都能用）：\n * ?sim=0.45 相似度阈值，只作用于 CNBC/CBS 之间\n * ?delay=6 板块压后几小时，只影响板块 feed\n * ?star=0 板块里不标 ★\n * ?video=1 保留 CBS 的电视片段（默认剔除）\n * ?maxage=10 Top News 只收 10 天内发布的，0=不限，只影响 Top News\n * ?tag=0 去掉标题前的来源前缀\n * ?diag=0 不要 feed 里那条「合并诊断」\n * ?debug=1 改为输出人类可读的诊断网页\n *\n * URL 写法：多个参数之间用 & 连接，只有第一个用 ?\n */\n\ntype Item = {\n tag: string;\n pri: number;\n title: string;\n link: string;\n guid: string;\n date: Date;\n desc: string;\n};\n\ntype Src = { tag: string; pri: number; url: string };\n\n// pri 小的优先保留 → 重复时保留 CNBC 的版本。\nconst TOP: Src[] = [\n {\n tag: \"CNBC\",\n pri: 0,\n url: \"https://www.cnbc.com/id/100003114/device/rss/rss.html\",\n },\n { tag: \"CBS\", pri: 1, url: \"https://www.cbsnews.com/latest/rss/main\" },\n];\n\nconst SECTIONS: Record = {\n politics: {\n title: \"CNBC 政治与政策（全量 · ★ 表示也上了 Top News）\",\n src: {\n tag: \"CNBC\",\n pri: 0,\n url: \"https://www.cnbc.com/id/10000113/device/rss/rss.html\",\n },\n },\n economy: {\n title: \"CNBC 经济（全量 · ★ 表示也上了 Top News）\",\n src: {\n tag: \"CNBC\",\n pri: 0,\n url: \"https://www.cnbc.com/id/20910258/device/rss/rss.html\",\n },\n },\n tech: {\n title: \"CNBC 科技（全量 · ★ 表示也上了 Top News）\",\n src: {\n tag: \"CNBC\",\n pri: 0,\n url: \"https://www.cnbc.com/id/19854910/device/rss/rss.html\",\n },\n },\n};\n\nconst SECTION_SRCS: Src[] = Object.values(SECTIONS).map((s) => s.src);\n/* section items are held back N hours so CNBC has time to promote them into Top News; then the star is right the first time */\nconst DEFAULT_DELAY_H = 6;\n\n/* CBS 偶尔会往 feed 里吐一批旧存档：2026-09-09 一次进来 11 条，最老的发布于 4 月 30 日。\n 它们按发布日期排到列表最底部，人从上往下读永远扫不到，于是永远卡在未读里。\n Top News 是「现在在发生什么」，超龄的直接不收。0 = 不限。\n 板块不设这个上限 —— 板块靠两周多的深度给 Top News 做去重参照，砍了会丢内容。 */\nconst DEFAULT_MAX_AGE_D = 10;\n\nconst DEFAULT_SIM = 0.45;\nconst WINDOW_HOURS = 36; // 只在这个时间窗内比较，避免跨天误判\n\nconst UA =\n \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 \" +\n \"(KHTML, like Gecko) Chrome/126.0 Safari/537.36\";\n\nasync function fetchText(url: string): Promise {\n const res = await fetch(url, { headers: { \"User-Agent\": UA } });\n if (!res.ok) throw new Error(`${res.status} ${url}`);\n return new TextDecoder(\"utf-8\").decode(await res.arrayBuffer());\n}\n\nconst decodeEntities = (s: string) =>\n s.replace(//g, \"$1\")\n .replace(/ /g, \" \").replace(/’|’/g, \"'\")\n .replace(/‘|‘/g, \"'\").replace(/“|“/g, '\"')\n .replace(/”|”/g, '\"').replace(/—|—/g, \"—\")\n .replace(/&/g, \"&\").replace(/</g, \"<\").replace(/>/g, \">\")\n .replace(/"/g, '\"').replace(/'|'/g, \"'\");\n\nconst tagText = (xml: string, name: string) => {\n const m = xml.match(\n new RegExp(`<${name}[^>]*>([\\\\s\\\\S]*?)<\\\\/${name}>`, \"i\"),\n );\n return m ? decodeEntities(m[1]).replace(/<[^>]*>/g, \"\").trim() : \"\";\n};\n\nfunction parseFeed(xml: string, src: Src): Item[] {\n const out: Item[] = [];\n for (const m of xml.matchAll(/][\\s\\S]*?<\\/item>/gi)) {\n const raw = m[0];\n const title = tagText(raw, \"title\");\n if (!title) continue;\n const link = tagText(raw, \"link\");\n const guid = tagText(raw, \"guid\") || link;\n const d = new Date(tagText(raw, \"pubDate\"));\n out.push({\n tag: src.tag,\n pri: src.pri,\n title,\n link,\n guid,\n date: isNaN(d.getTime()) ? new Date() : d,\n desc: tagText(raw, \"description\").slice(0, 400),\n });\n }\n return out;\n}\n\n/* CBS feed 里约三分之一是电视节目片段（/video/），含带货广告和整期节目片头。\n 实测硬新闻（飓风、空难、庭审）都有 /news/ 版本，video-only 的残余基本是软性栏目。\n 默认剔除，?video=1 保留。 */\nconst isVideo = (it: Item) => /\\/video\\//.test(it.link || \"\");\n\nasync function collect(srcs: Src[], keepVideo = true) {\n const counts: Record = {};\n const errors: string[] = [];\n const items: Item[] = [];\n let videoDropped = 0;\n await Promise.all(srcs.map(async (s) => {\n try {\n let got = parseFeed(await fetchText(s.url), s);\n if (!keepVideo) {\n const before = got.length;\n got = got.filter((it) => !isVideo(it));\n videoDropped += before - got.length;\n }\n counts[s.tag] = (counts[s.tag] ?? 0) + got.length;\n items.push(...got);\n } catch (e) {\n counts[s.tag] = counts[s.tag] ?? 0;\n errors.push(`${s.tag}: ${e}`);\n }\n }));\n return { items, counts, errors, videoDropped };\n}\n\n// ---- 近似去重 ----\n\nconst STOP = new Set(\n (\"a an the and or but of to in on at for from by with as is are was were be \" +\n \"been being it its this that these those he she they them his her their \" +\n \"will would can could may might says said say new after over into about \" +\n \"amid up down out off more most than then there here what which who whom \" +\n \"you your we our us i me my not no nor so if because while during has have\")\n .split(/\\s+/),\n);\n\nfunction tokens(title: string): string[] {\n return [\n ...new Set(\n title.toLowerCase()\n .replace(/[^a-z0-9$%\\s]/g, \" \")\n .split(/\\s+/)\n .filter(Boolean)\n .filter((w) => !STOP.has(w))\n .filter((w) => w.length >= 3 || /\\d/.test(w)),\n ),\n ];\n}\n\n/**\n * 相似度 = IDF 加权的 containment。\n * 加权：共享的专有名词（Clancy、Nvidia）权重高，\"says / new / after\" 权重低。\n * containment 而非 Jaccard：两家标题长短经常差很多，Jaccard 会被长的稀释。\n * 天花板：改写到没有共同词的重复抓不到，例如\n * \"Fed holds rates steady\" vs \"Federal Reserve leaves rates unchanged\"（0.29）。\n */\nfunction makeSim(all: string[][]) {\n const n = all.length;\n const df = new Map();\n for (const doc of all) for (const w of doc) df.set(w, (df.get(w) ?? 0) + 1);\n const wt = (w: string) => Math.log((n + 1) / ((df.get(w) ?? 0) + 0.5));\n return (a: string[], b: string[]) => {\n const B = new Set(b);\n let inter = 0, sa = 0, sb = 0;\n for (const w of a) {\n sa += wt(w);\n if (B.has(w)) inter += wt(w);\n }\n for (const w of b) sb += wt(w);\n if (sa <= 0 || sb <= 0) return 0;\n return inter / Math.min(sa, sb);\n };\n}\n\ntype Dropped = { kept: Item; dropped: Item; score: number; exact: boolean };\n\nconst key = (it: Item) => it.link || it.guid;\n\n/**\n * items 内部互相去重；再扣掉与 reference 重复的。\n * reference 为空就是单纯的合并去重。\n */\nfunction reduce(\n items: Item[],\n reference: Item[],\n simThreshold: number,\n refFuzzy = true,\n) {\n const ordered = [...items].sort(\n (x, y) => x.pri - y.pri || y.date.getTime() - x.date.getTime(),\n );\n const tk = new Map();\n for (const it of [...ordered, ...reference]) tk.set(it, tokens(it.title));\n const sim = makeSim([...tk.values()]);\n\n const norm = (t: string) =>\n t.toLowerCase().replace(/[^a-z0-9]+/g, \" \").trim();\n const titleSeen = new Map();\n for (const r of reference) titleSeen.set(norm(r.title), r);\n const refKeys = new Set(reference.map(key));\n const kept: Item[] = [];\n const dropped: Dropped[] = [];\n const seen = new Set();\n\n for (const it of ordered) {\n const k = key(it);\n if (seen.has(k)) continue;\n\n // 1) 精确命中 reference（CNBC 板块 vs CNBC Top 通常就是同一篇）\n if (refKeys.has(k)) {\n const ref = reference.find((r) => key(r) === k)!;\n dropped.push({ kept: ref, dropped: it, score: 1, exact: true });\n continue;\n }\n\n // 2) 近似命中 reference 或已保留的条目\n /* identical title = same story; ignore the time window (CBS republishes old 60 Minutes segments under their original dates, years apart) */ const nt =\n norm(it.title);\n const twin = titleSeen.get(nt);\n if (twin) {\n dropped.push({ kept: twin, dropped: it, score: 1, exact: true });\n continue;\n }\n let dup: { it: Item; score: number } | null = null;\n for (const other of (refFuzzy ? [...reference, ...kept] : kept)) {\n const dh = Math.abs(other.date.getTime() - it.date.getTime()) / 36e5;\n if (dh > WINDOW_HOURS) continue;\n const s = sim(tk.get(it)!, tk.get(other)!);\n if (s >= simThreshold && (!dup || s > dup.score)) {\n dup = { it: other, score: s };\n }\n }\n if (dup) {\n dropped.push({\n kept: dup.it,\n dropped: it,\n score: dup.score,\n exact: false,\n });\n continue;\n }\n\n seen.add(k);\n kept.push(it);\n titleSeen.set(nt, it);\n }\n\n kept.sort((x, y) => y.date.getTime() - x.date.getTime());\n return { kept, dropped };\n}\n\nconst esc = (s: string) =>\n s.replace(/&/g, \"&\").replace(//g, \">\")\n .replace(/\"/g, \""\").replace(/'/g, \"'\");\n\n// ---- 输出 ----\n\nfunction dropTable(dropped: Dropped[]) {\n if (!dropped.length) {\n return `这一轮没有判出重复。`;\n }\n return [...dropped].sort((a, b) => b.score - a.score).map((d) =>\n `${d.exact ? \"同一篇\" : d.score.toFixed(2)}` +\n `丢弃 ${esc(d.dropped.tag)} ${esc(d.dropped.title)}
` +\n `因为已有 ${esc(d.kept.tag)} ${esc(d.kept.title)}`\n ).join(\"\");\n}\n\nfunction summaryLine(\n counts: Record,\n keptN: number,\n droppedN: number,\n sim: number,\n) {\n return `阈值 sim=${sim}　抓到 ${\n Object.entries(counts).map(([k, v]) => `${k} ${v} 条`).join(\"、\")\n }　→ 保留 ${keptN} 条，丢弃 ${droppedN} 条。`;\n}\n\nfunction diagEntry(\n counts: Record,\n kept: Item[],\n dropped: Dropped[],\n sim: number,\n base: URL,\n note = \"\",\n) {\n const day = (() => {\n const iso = () => new Date().toISOString().slice(0, 10);\n try {\n const s = new Date().toLocaleDateString(\"en-CA\", {\n timeZone: \"America/New_York\",\n });\n return /^\\d{4}-\\d{2}-\\d{2}$/.test(s) ? s : iso();\n } catch {\n return iso();\n }\n })(); /* label the day in US Eastern, not UTC: after 8pm ET the UTC date has already rolled over */\n const html =\n `

${summaryLine(counts, kept.length, dropped.length, sim)}${note}

` +\n `` +\n `${\n dropTable(dropped)\n }

相似度	丢弃了什么 / 为什么

` +\n `

可调参数（跟在地址后面，第一个用 ? 之后用 &）：

` +\n `` +\n `` +\n `` +\n `` +\n `

参数	作用
`?sim=0.45`	相似度阈值，越低去重越狠（默认 0.45）
`?tag=0`	标题前不加来源前缀（板块 feed 本来就没有）
`?diag=0`	不要这条诊断
`?debug=1`	改为输出人类可读的诊断网页
`?delay=6`	板块条目压后几小时再输出（默认 6，只影响板块 feed）
`?star=0`	板块里不标 ★
`?video=1`	保留 CBS 的电视片段（默认剔除）
`?maxage=10`	Top News 只收 N 天内发布的（默认 10，0=不限；板块不受影响）

` +\n `

例：${base.origin}${base.pathname}?sim=0.35&debug=1

`;\n return {\n title:\n `合并诊断 · ${day}（丢弃 ${dropped.length} 条 / 保留 ${kept.length} 条）`,\n html,\n guid:\n `diag-${base.pathname}`, /* no date: one diagnostic per feed forever, updated in place, never accumulates */\n link: `${base.origin}${base.pathname}?sim=${sim}&debug=1`,\n };\n}\n\nfunction buildRss(\n chanTitle: string,\n chanDesc: string,\n items: Item[],\n showTag: boolean,\n diag: ReturnType | null,\n) {\n const parts: string[] = [];\n if (diag) {\n // 排在最新一条之后一分钟 → 置顶。guid 按天固定，一天只新增一条，不会反复未读。\n const newest = items.length ? items[0].date.getTime() : Date.now();\n parts.push(\n `${esc(diag.title)}${esc(diag.link)}` +\n `${esc(diag.guid)}` +\n `${new Date(newest + 60_000).toUTCString()}` +\n `${esc(diag.html)}`,\n );\n }\n for (const it of items) {\n const title = showTag ? `${it.tag} · ${it.title}` : it.title;\n parts.push(\n `${esc(title)}${esc(it.link)}` +\n `${esc(it.guid)}` +\n `${it.date.toUTCString()}` +\n `${esc(it.desc)}`,\n );\n }\n return `\n\n${esc(chanTitle)}\nhttps://www.cnbc.com/\n${esc(chanDesc)}\nen-US\n${new Date().toUTCString()}\n${parts.join(\"\")}\n`;\n}\n\nfunction debugPage(\n chanTitle: string,\n counts: Record,\n kept: Item[],\n dropped: Dropped[],\n sim: number,\n) {\n return `${\n esc(chanTitle)\n } · 诊断\n\n

${esc(chanTitle)} · 合并诊断

\n\n

${summaryLine(counts, kept.length, dropped.length, sim)}

阈值调低去重更狠（可能误杀），调高更保守（可能漏重复）。多个参数用 & 连接，\n只有第一个用 ?：/tech?sim=0.35&debug=1

\n${\n dropTable(dropped)\n }

相似度	丢弃了什么 / 为什么

`;\n}\n\nexport default async function (req: Request): Promise {\n const url = new URL(req.url);\n const sim = Math.min(\n 0.95,\n Math.max(0.2, Number(url.searchParams.get(\"sim\")) || DEFAULT_SIM),\n );\n const wantTag = url.searchParams.get(\"tag\") !== \"0\";\n const wantDiag = url.searchParams.get(\"diag\") !== \"0\";\n const wantStar = url.searchParams.get(\"star\") !== \"0\";\n const keepVideo = url.searchParams.get(\"video\") === \"1\";\n const dRaw = url.searchParams.get(\"delay\");\n const delayH = dRaw === null\n ? DEFAULT_DELAY_H\n : Math.min(48, Math.max(0, Number(dRaw) || 0));\n const mRaw = url.searchParams.get(\"maxage\");\n const maxAgeD = mRaw === null\n ? DEFAULT_MAX_AGE_D\n : Math.min(3650, Math.max(0, Number(mRaw) || 0));\n let videoOut = 0, held = 0, starred = 0, staleOut = 0;\n const path = url.pathname.replace(/^\\/|\\/$/g, \"\");\n\n const plain = (body: string, status = 200) =>\n new Response(body, {\n status,\n headers: { \"Content-Type\": \"text/plain; charset=utf-8\" },\n });\n\n if (path !== \"\" && !SECTIONS[path]) {\n return plain(\n \"可用地址：\\n / Top News（CNBC + CBS 去重）\\n\" +\n Object.entries(SECTIONS).map(([k, v]) =>\n ` /${k}${\" \".repeat(11 - k.length)}${v.title}`\n )\n .join(\"\\n\") +\n \"\\n\\n参数：?sim= ?tag=0 ?diag=0 ?delay= ?star=0 ?video=1 ?maxage= ?debug=1\",\n 404,\n );\n }\n\n let chanTitle: string, chanDesc: string, counts: Record;\n let kept: Item[], dropped: Dropped[], showTag: boolean, errors: string[];\n\n if (path === \"\") {\n const [got, secs] = await Promise.all([\n collect(TOP, keepVideo),\n collect(SECTION_SRCS, keepVideo),\n ]);\n counts = { ...got.counts, \"板块（参照）\": secs.items.length };\n errors = [...got.errors, ...secs.errors];\n videoOut = got.videoDropped + secs.videoDropped;\n /* against the section feeds use exact / same-title only, never fuzzy:\n a false positive here would delete a story from the main feed. */\n let pool = got.items;\n if (maxAgeD > 0) {\n const cutAge = Date.now() - maxAgeD * 864e5;\n pool = pool.filter((it) => it.date.getTime() >= cutAge);\n staleOut = got.items.length - pool.length;\n }\n const r = reduce(pool, secs.items, sim, false);\n kept = r.kept;\n dropped = r.dropped;\n chanTitle = \"US Top News（CNBC + CBS 去重，已扣除板块）\";\n chanDesc =\n \"CNBC Top News 与 CBS Top News 合并去重，再扣掉已经在科技/政治/经济板块 feed 里的条目。\";\n showTag = wantTag;\n } else {\n const sec = SECTIONS[path];\n const [got, top] = await Promise.all([\n collect([sec.src], keepVideo),\n collect(TOP, keepVideo),\n ]);\n counts = { ...got.counts, \"Top News（标★用）\": top.items.length };\n errors = [...got.errors, ...top.errors];\n videoOut = got.videoDropped; /* top 只是 ★ 参照，别把它剔掉的算进板块 */\n const cut = Date.now() - delayH * 36e5;\n const aged = got.items.filter((it) => it.date.getTime() <= cut);\n held = got.items.length - aged.length;\n const r = reduce(aged, [], sim);\n if (wantStar) {\n const nk = (t: string) =>\n t.toLowerCase().replace(/[^a-z0-9]+/g, \" \").trim();\n const topKeys = new Set(top.items.map(key));\n const topTitles = new Set(top.items.map((t) => nk(t.title)));\n for (const it of r.kept) {\n if (topKeys.has(key(it)) || topTitles.has(nk(it.title))) {\n it.title = \"★ \" + it.title;\n starred++;\n }\n }\n }\n kept = r.kept;\n dropped = r.dropped;\n chanTitle = sec.title;\n chanDesc = \"板块全量，不再扣除 Top News。★ 表示这条同时也上了 Top News。\";\n showTag = false;\n }\n\n const vnote = videoOut\n ? `　剔除 CBS 电视片段 ${videoOut} 条（?video=1 保留）。`\n : \"\";\n\n const snote = staleOut\n ? `　剔除发布超过 ${maxAgeD} 天的旧条目 ${staleOut} 条（?maxage=0 不限）。`\n : \"\";\n const note = path === \"\"\n ? vnote + snote\n : `${vnote}　板块延迟 ${delayH} 小时，暂缓 ${held} 条；其中 ${starred} 条标了 ★。`;\n\n if (!kept.length && dropped.length === 0) {\n return plain(\n \"一条都没抓到。把这句话告诉 Claude。\\n\" + errors.join(\"\\n\"),\n 500,\n );\n }\n\n if (url.searchParams.get(\"debug\") === \"1\") {\n return new Response(debugPage(chanTitle, counts, kept, dropped, sim), {\n headers: { \"Content-Type\": \"text/html; charset=utf-8\" },\n });\n }\n\n const diag = wantDiag\n ? diagEntry(counts, kept, dropped, sim, url, note)\n : null;\n return new Response(buildRss(chanTitle, chanDesc, kept, showTag, diag), {\n headers: {\n \"Content-Type\": \"application/rss+xml; charset=utf-8\",\n \"Cache-Control\": \"public, max-age=300\",\n },\n });\n}","children",[],[57],["D",1789140945989],[],[19],"449fec92-6e09-4aec-b796-e8acf2bfb859",["D",1788471196775],"deletedAt","legacyEnvVars","privacy","public","httpPrivacy","description","CNBC + CBS Top News merged and de-duplicated into one RSS feed","imageUrl","userId","4d930760-aac6-4857-b21b-f01a00f81422","pinnedFileId","pinPreview","pinMigrated","screenshotVersion","likeCount",0,"orgId","01a06281-3add-7588-9875-998aa2e1101c","tursoName","tursoHostname","tursoMeasuredAt","pinned","userPerm","r","valPerm","blobPerm","rw","emailPerm","sqlitePerm","projectPerm","telemetryPerm","user",{"_11":12,"_125":121},"org",{"_11":12,"_118":119,"_120":121,"_80":81,"_122":123,"_124":51},"branch",{"_17":41,"_13":16,"_33":39,"_110":70,"_111":-5,"_112":-5,"_113":-5,"_114":-5,"_115":-5,"_116":14,"_117":12},"isOwned","projectId","forkedBranchId","forkedBranchVersion","forkedProjectPrivacy","forkedProjectId","forkedProjectOrgId","projectName","userHandle","displayName","Meimei An","avatar","https://lh3.googleusercontent.com/a/ACg8ocIGi6q-4ldhq-uZuUQBjSymuORY8Y80AuFVk-RYPSZTveoWJw=s96-c","tier","free","canRestrictVals","profileImageUrl","adoptOrgId",{"_33":70,"_13":14,"_44":162,"_72":-5,"_73":51,"_74":75,"_76":75,"_77":78,"_79":-5,"_80":81,"_82":58,"_83":43,"_84":-5,"_85":-5,"_86":87,"_88":89,"_90":-5,"_91":-5,"_92":-5,"_93":-5,"_94":95,"_96":95,"_97":98,"_99":98,"_100":98,"_101":95,"_102":95,"_103":163,"_105":164,"_107":165,"_109":51},[156,157],"files",[156,157],"counts",{"_146":87,"_147":148,"_149":87,"_32":150,"_151":152},[],{"_33":34,"_13":19,"_35":36,"_37":-5,"_38":39,"_40":41,"_42":43,"_44":45,"_17":46,"_47":24,"_48":49,"_50":51,"_52":-5,"_18":145},"upstreamProject","upstreamBranch","emptyEnvVars","flags",[],"hasHttpFiles","grants","orgs","customDomains",{},[19],"pulls","branches",1,"forks",{"_153":87,"_155":87},"connections",{"_153":87,"_154":87},"count","unfilledCount","emptyCount",{"_33":34,"_13":19,"_35":36,"_37":-5,"_38":39,"_40":41,"_42":43,"_44":67,"_17":46,"_47":24,"_48":49,"_50":51,"_52":-5,"_64":160,"_18":161},{"_33":56,"_13":57,"_35":58,"_37":-5,"_38":39,"_40":59,"_42":43,"_44":60,"_17":61,"_47":62,"_48":63,"_50":51,"_52":-5,"_64":158,"_18":159},[],[57],[],[19],["D",1788471196775],{"_11":12,"_125":121},{"_11":12,"_118":119,"_120":121,"_80":81,"_122":123,"_124":51},{"_17":41,"_13":16,"_33":39,"_110":70,"_111":-5,"_112":-5,"_113":-5,"_114":-5,"_115":-5,"_116":14,"_117":12},"ctx",{"_103":-5,"_142":199,"_200":201,"_202":203,"_204":51,"_205":51,"_206":51,"_207":43,"_208":-5,"_209":-7,"_210":-7,"_211":87,"_212":51,"_213":51,"_214":-5,"_215":51,"_216":51,"_217":51,"_218":-5},"ENV",{"_170":171,"_172":173,"_174":16,"_175":176,"_177":178,"_179":180,"_181":182,"_183":184,"_185":186,"_187":188,"_189":190,"_191":192,"_193":194,"_195":196,"_197":198},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","e87eb04e707cf64e7e532c8e8ec395f209e17f15","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]