[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_165":166,"_167":168},"routes/_app.x.$handle.$name",{"_94":93,"_32":44,"_103":84,"_27":52,"_13":14,"_104":-5,"_11":105,"_106":107,"_108":109,"_110":111,"_112":113,"_20":-5,"_114":-5,"_115":-5,"_116":26,"_117":118,"_119":24,"_120":-5,"_121":-5,"_122":123},"routes/_app.x.$handle.$name.branch.$branch.version.$version.diff",{"_11":12,"_13":14,"_15":16,"_17":-5,"_18":-5},"project",{"_42":43,"_32":44,"_37":45,"_46":-5,"_47":26,"_48":49,"_50":49,"_51":52,"_53":-5,"_18":54,"_55":-5,"_56":26,"_57":-5,"_58":-5,"_59":60,"_61":62,"_63":-5,"_64":-5,"_65":-5,"_66":-5,"_67":68,"_69":68,"_70":71,"_72":71,"_73":71,"_74":68,"_75":68,"_76":77,"_78":79,"_80":81,"_82":26},"version",14,"changes",[19],"aiChatInfo","userId",{"_20":21,"_22":-7,"_23":24,"_25":26,"_27":28,"_29":28},"file",{"_30":31,"_32":28,"_33":34,"_35":36,"_37":38,"_39":14,"_40":41,"_23":26},"previous","deleted",true,"deletedHere",false,"path","report.ts","latestPath","fileId","045e97dc-52b1-11f1-b210-ee650bb23af1","name","fileType","script","content","// Claude Code Effort Benchmark — Phase 1+SWE\n// Auto-generated by scripts/generate_report.py\nexport default async function(req: Request): Promise {\n const html = `\n\n\n\n\nClaude Code Effort Benchmark\n\n\n\n\n

Agent Effort Benchmark Phase 1+SWE

Empirické měření dopadu reasoning effort napříč providery a modely

308

celkem runů

úlohy

173/308

verify pass

\\$120.2

celkem náklady

nekompletní runy

Přehled výsledků

Úloha	low	medium	high	max	Effort gradient

Pass rate per effort

Průměrná cena per effort

Tokeny per effort

Iterace (turns) per effort

Effort	n	Pass%	Avg score	Avg cost	Avg wall	Avg tools	Avg iter	Tokens	Tool breakdown

Efektivita: cena vs. kvalita

Každý bod = jeden run. Osy: cena (USD) vs. skóre (0–1). Ideální bod = levý horní roh (levné a kvalitní). Pareto frontier = zelená čára.

Tokeny a iterace napříč úlohami

Token consumption per effort (stacked)

Průměrné iterace per effort

Čas a efektivita

Průměrný wall time per effort

Cena per úspěšný run

Success rate podle typu úlohy

SWE-bench výsledky

Reálné bugy z open-source projektů.

Instance	low	medium	high	max	Finding

Syntetické úlohy

Na dobře specifikovaných úlohách jsou všechny effort úrovně funkčně ekvivalentní.

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

Na reálných bugech (django-16910) low/medium konzistentně selávají, high uspívá. Na jednoduchých bugech i low stačí.

💡 Max ≠ lepší výsledek

Max effort stojí 2–4× více než high, ale výsledek je stejný. High je sweet spot pro reálné bugy.

⚠️ Debug bez testu = nulový efekt

03_debug_order Bug B nebyl nalezen ani max effortem. Failing testy jsou nezbytný guidance.

⚠️ Python 3.14 limituje SWE-bench

Pre-2022 projekty jsou nekompatibilní (odstraňeny moduly cgi, distutils). Používejte Django 4.2+ / Sympy 1.11+.

Doporučení

Use case	Effort	Proč

\n\n\n`;\n return new Response(html, { headers: {'content-type': 'text/html;charset=utf-8'} });\n}\n","createdAt",["D",1779105566110],"branchVersion","branchId","7683ffee-501f-11f1-afc4-ee650bb23af1","id","76834068-501f-11f1-afc4-ee650bb23af1","effortBenchmark",["D",1778823148771],"deletedAt","legacyEnvVars","privacy","public","httpPrivacy","description","","imageUrl","7241cb49-222a-4e9a-af99-25b2615eda07","pinnedFileId","pinPreview","pinMigrated","screenshotVersion","likeCount",0,"orgId","019e0b3d-7f60-7428-8330-1beb489e3416","tursoName","tursoHostname","tursoMeasuredAt","pinned","userPerm","r","valPerm","blobPerm","rw","emailPerm","sqlitePerm","projectPerm","telemetryPerm","user",{"_94":93,"_102":98},"org",{"_94":93,"_95":96,"_97":98,"_18":54,"_99":100,"_101":26},"branch",{"_13":83,"_32":84,"_42":41,"_85":43,"_86":-5,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":44,"_92":93},"isOwned",23,"main","projectId","forkedBranchId","forkedBranchVersion","forkedProjectPrivacy","forkedProjectId","forkedProjectOrgId","projectName","userHandle","patrickzandl","handle","displayName","Patrick Zandl","avatar","https://lh3.googleusercontent.com/a/ACg8ocK47-vrp0IIPUC8DK3BcjE57lcyPd28xqviX34gk6fkTziKkbep=s96-c","tier","free","canRestrictVals","profileImageUrl","branchName","adoptOrgId",{"_42":43,"_32":44,"_37":161,"_46":-5,"_47":26,"_48":49,"_50":49,"_51":52,"_53":-5,"_18":54,"_55":-5,"_56":26,"_57":-5,"_58":-5,"_59":60,"_61":62,"_63":-5,"_64":-5,"_65":-5,"_66":-5,"_67":68,"_69":68,"_70":71,"_72":71,"_73":71,"_74":68,"_75":68,"_76":162,"_78":163,"_80":164,"_82":26},"tree",[152,153],"files",[152,153],"counts",{"_141":60,"_142":143,"_144":60,"_145":146,"_147":148},"contents",[124,125],"upstreamProject","upstreamBranch","emptyEnvVars","flags",[],"hasHttpFiles","grants","orgs","customDomains",{},{"_42":132,"_32":133,"_30":134,"_127":-5,"_40":41,"_39":135,"_128":26,"_37":136,"_13":137,"_33":138,"_35":139,"_23":26,"_130":-5,"_27":140},{"_42":126,"_32":28,"_30":31,"_127":-5,"_40":41,"_39":14,"_128":26,"_37":129,"_13":60,"_33":34,"_35":36,"_23":26,"_130":-5,"_27":131},"045e9700-52b1-11f1-b210-ee650bb23af1","parentId","latest",["D",1779105566110],"label",[28],"791329aa-5228-11f1-9aaf-ee650bb23af1","main.http.ts","7768c20a-501f-11f1-afc4-ee650bb23af1",13,["D",1779046920850],12,"http","// Claude Code Effort Benchmark — Phase 1+SWE\n// Auto-generated by scripts/generate_report.py\nexport default async function(req: Request): Promise {\n const html = `\n\n\n\n\nClaude Code Effort Benchmark\n\n\n\n\n

Agent Effort Benchmark Phase 1+SWE

Empirické měření dopadu reasoning effort napříč providery a modely

181

celkem runů

úlohy

121/181

verify pass

\\$120.2

celkem náklady

nekompletní runy

Přehled výsledků

Úloha	low	medium	high	max	Effort gradient

Pass rate per effort

Průměrná cena per effort

Tokeny per effort

Iterace (turns) per effort

Effort	n	Pass%	Avg score	Avg cost	Avg wall	Avg tools	Avg iter	Tokens	Tool breakdown

Efektivita: cena vs. kvalita

Každý bod = jeden run. Osy: cena (USD) vs. skóre (0–1). Ideální bod = levý horní roh (levné a kvalitní). Pareto frontier = zelená čára.

Tokeny a iterace napříč úlohami

Token consumption per effort (stacked)

Průměrné iterace per effort

Čas a efektivita

Průměrný wall time per effort

Cena per úspěšný run

Success rate podle typu úlohy

SWE-bench výsledky

Reálné bugy z open-source projektů.

Instance	low	medium	high	max	Finding

Syntetické úlohy

Na dobře specifikovaných úlohách jsou všechny effort úrovně funkčně ekvivalentní.

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

Na reálných bugech (django-16910) low/medium konzistentně selávají, high uspívá. Na jednoduchých bugech i low stačí.

💡 Max ≠ lepší výsledek

Max effort stojí 2–4× více než high, ale výsledek je stejný. High je sweet spot pro reálné bugy.

⚠️ Debug bez testu = nulový efekt

03_debug_order Bug B nebyl nalezen ani max effortem. Failing testy jsou nezbytný guidance.

⚠️ Python 3.14 limituje SWE-bench

Pre-2022 projekty jsou nekompatibilní (odstraňeny moduly cgi, distutils). Používejte Django 4.2+ / Sympy 1.11+.

Doporučení

Use case	Effort	Proč

\n\n\n`;\n return new Response(html, { headers: {'content-type': 'text/html;charset=utf-8'} });\n}\n",[133],"pulls","branches",1,"forks","envVars",{"_149":60,"_151":60},"connections",{"_149":60,"_150":60},"count","unfilledCount","emptyCount",{"_42":132,"_32":133,"_30":134,"_127":-5,"_40":41,"_39":135,"_128":26,"_37":158,"_13":137,"_33":138,"_35":139,"_23":26,"_130":-5,"_155":159,"_27":160},{"_42":126,"_32":28,"_30":31,"_127":-5,"_40":41,"_39":14,"_128":26,"_37":154,"_13":60,"_33":34,"_35":36,"_23":26,"_130":-5,"_155":156,"_27":157},["D",1779105566110],"children",[],[28],["D",1779046920850],[],[133],["D",1778823148771],{"_94":93,"_102":98},{"_94":93,"_95":96,"_97":98,"_18":54,"_99":100,"_101":26},{"_13":83,"_32":84,"_42":41,"_85":43,"_86":-5,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":44,"_92":93},"ctx",{"_76":-5,"_121":198,"_199":200,"_201":202,"_203":26,"_204":26,"_205":26,"_206":24,"_207":-5,"_208":-7,"_209":-7,"_210":60,"_211":26,"_212":26,"_213":-5,"_214":26,"_215":26,"_216":26,"_217":-5},"ENV",{"_169":170,"_171":172,"_173":84,"_174":175,"_176":177,"_178":179,"_180":181,"_182":183,"_184":185,"_186":187,"_188":189,"_190":191,"_192":193,"_194":195,"_196":197},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","e87eb04e707cf64e7e532c8e8ec395f209e17f15","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]

Přehled výsledků

Pass rate per effort

Průměrná cena per effort

Tokeny per effort

Iterace (turns) per effort

Per-check breakdown

Efektivita: cena vs. kvalita

Tokeny a iterace napříč úlohami

Token consumption per effort (stacked)

Průměrné iterace per effort

Čas a efektivita

Průměrný wall time per effort

Cena per úspěšný run

Success rate podle typu úlohy

SWE-bench výsledky

Syntetické úlohy

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

💡 Max ≠ lepší výsledek

⚠️ Debug bez testu = nulový efekt

⚠️ Python 3.14 limituje SWE-bench

Doporučení

Přehled výsledků

Pass rate per effort

Průměrná cena per effort

Tokeny per effort

Iterace (turns) per effort

Per-check breakdown

Efektivita: cena vs. kvalita

Tokeny a iterace napříč úlohami

Token consumption per effort (stacked)

Průměrné iterace per effort

Čas a efektivita

Průměrný wall time per effort

Cena per úspěšný run

Success rate podle typu úlohy

SWE-bench výsledky

Syntetické úlohy

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

💡 Max ≠ lepší výsledek

⚠️ Debug bez testu = nulový efekt

⚠️ Python 3.14 limituje SWE-bench

Doporučení

Diff · v14