[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_154":155,"_156":157},"routes/_app.x.$handle.$name",{"_95":94,"_32":45,"_104":85,"_27":53,"_13":14,"_105":-5,"_11":106,"_107":108,"_109":110,"_111":112,"_113":114,"_20":-5,"_115":-5,"_116":-5,"_117":25,"_118":119,"_120":121,"_122":-5,"_123":-5,"_124":125},"routes/_app.x.$handle.$name.branch.$branch.version.$version.diff",{"_11":12,"_13":14,"_15":16,"_17":-5,"_18":-5},"project",{"_43":44,"_32":45,"_38":46,"_47":-5,"_48":25,"_49":50,"_51":50,"_52":53,"_54":-5,"_18":55,"_56":-5,"_57":25,"_58":-5,"_59":-5,"_60":61,"_62":63,"_64":-5,"_65":-5,"_66":-5,"_67":-5,"_68":69,"_70":69,"_71":72,"_73":72,"_74":72,"_75":69,"_76":69,"_77":78,"_79":80,"_81":82,"_83":25},"version",5,"changes",[19],"aiChatInfo","userId",{"_20":21,"_22":23,"_24":25,"_26":25,"_27":28,"_29":28},"file",{"_30":31,"_32":28,"_33":34,"_35":37,"_38":39,"_40":14,"_41":42,"_24":25},"previous",{"_30":31,"_32":28,"_33":34,"_35":36},"deleted",false,"deletedHere","path","main.http.ts","latestPath","fileId","7768c20a-501f-11f1-afc4-ee650bb23af1","name","fileType","http","content","// Claude Code Effort Benchmark — Phase 1+SWE\n// Auto-generated by scripts/generate_report.py\nexport default async function(req: Request): Promise {\n const html = String.raw`\n\n\n\n\nClaude Code Effort Benchmark\n\n\n\n\n

\n \n \n GitHub\n \n \n \n

Claude Code Effort Benchmark Phase 1+SWE

Empirické měření dopadu parametru --effort · claude-opus-4-7

145

celkem runů

úlohy

98/145

verify pass

$115.3

celkem náklady

Přehled výsledků

Úloha	low	medium	high	max	Effort gradient

SWE-bench výsledky

Reálné bugy z open-source projektů.

Instance	low	medium	high	max	Finding

Django-16910: detail effort gradientu

only()+select_related() na reverse OneToOneField.

Pass rate per effort

Průměrná cena per effort

Effort	n	Pass%	Avg score	Avg cost	Avg wall	Avg tools

Syntetické úlohy

Na dobře specifikovaných úlohách jsou všechny effort úrovně funkčně ekvivalentní.

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

Na obtížných reálných bugech (django-16408, 16820, 16910) low konzistentně selhává (~25%), medium+ uspívá (90-100%). Na jednoduchých bugech i low stačí.

💡 Max má hodnotu pro multi-aspect bugy

sympy-22840 má 2 nezávislé části: medium/high opraví 1 ze 2 (67%), max 60% runů opraví obě (100%). Jediný případ kde max přesně rozhoduje.

⚠️ Debug bez testu = nulový efekt

03_debug_order Bug B nebyl nalezen ani max effortem. Failing testy jsou nezbytný guidance.

⚠️ Python 3.14 limituje SWE-bench

Pre-2022 projekty nekompatibilní (odstraněny moduly cgi, distutils). Použijte Django 4.2+ / Sympy 1.11+.

Doporučení

Use case	Effort	Proč

\n\n\n\n`;\n return new Response(html, { headers: {'content-type': 'text/html;charset=utf-8'} });\n}\n","// Claude Code Effort Benchmark — Phase 1+SWE\n// Auto-generated by scripts/generate_report.py\nexport default async function(req: Request): Promise {\n const html = String.raw`\n\n\n\n\nClaude Code Effort Benchmark\n\n\n\n\n

Claude Code Effort Benchmark Phase 1+SWE

Empirické měření dopadu parametru --effort · claude-opus-4-7

145

celkem runů

úlohy

98/145

verify pass

$115.3

celkem náklady

Přehled výsledků

Úloha	low	medium	high	max	Effort gradient

Pass rate per effort

Průměrná cena per effort

Tokeny per effort

Iterace (turns) per effort

Effort	n	Pass%	Avg score	Avg cost	Avg wall	Avg tools	Avg iter	Tokens	Tool breakdown

Efektivita: cena vs. kvalita

Každý bod = jeden run. Osy: cena (USD) vs. skóre (0–1). Ideální bod = levý horní roh (levné a kvalitní). Pareto frontier = zelená čára.

Tokeny a iterace napříč úlohami

Token consumption per effort (stacked)

Průměrné iterace per effort

SWE-bench výsledky

Reálné bugy z open-source projektů.

Instance	low	medium	high	max	Finding

Syntetické úlohy

Na dobře specifikovaných úlohách jsou všechny effort úrovně funkčně ekvivalentní.

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

Na reálných bugech (django-16910) low/medium konzistentně selávají, high uspívá. Na jednoduchých bugech i low stačí.

💡 Max ≠ lepší výsledek

Max effort stojí 2–4× více než high, ale výsledek je stejný. High je sweet spot pro reálné bugy.

⚠️ Debug bez testu = nulový efekt

03_debug_order Bug B nebyl nalezen ani max effortem. Failing testy jsou nezbytný guidance.

⚠️ Python 3.14 limituje SWE-bench

Pre-2022 projekty jsou nekompatibilní (odstraňeny moduly cgi, distutils). Používejte Django 4.2+ / Sympy 1.11+.

Doporučení

Use case	Effort	Proč

\n\n\n`;\n return new Response(html, { headers: {'content-type': 'text/html;charset=utf-8'} });\n}\n","createdAt",["D",1779011195490],"branchVersion","branchId","7683ffee-501f-11f1-afc4-ee650bb23af1","id","76834068-501f-11f1-afc4-ee650bb23af1","effortBenchmark",["D",1778823148771],"deletedAt","legacyEnvVars","privacy","public","httpPrivacy","description","","imageUrl","7241cb49-222a-4e9a-af99-25b2615eda07","pinnedFileId","pinPreview","pinMigrated","screenshotVersion","likeCount",0,"orgId","019e0b3d-7f60-7428-8330-1beb489e3416","tursoName","tursoHostname","tursoMeasuredAt","pinned","userPerm","r","valPerm","blobPerm","rw","emailPerm","sqlitePerm","projectPerm","telemetryPerm","user",{"_95":94,"_103":99},"org",{"_95":94,"_96":97,"_98":99,"_18":55,"_100":101,"_102":25},"branch",{"_13":84,"_32":85,"_43":42,"_86":44,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":-5,"_92":45,"_93":94},"isOwned",23,"main","projectId","forkedBranchId","forkedBranchVersion","forkedProjectPrivacy","forkedProjectId","forkedProjectOrgId","projectName","userHandle","patrickzandl","handle","displayName","Patrick Zandl","avatar","https://lh3.googleusercontent.com/a/ACg8ocK47-vrp0IIPUC8DK3BcjE57lcyPd28xqviX34gk6fkTziKkbep=s96-c","tier","free","canRestrictVals","profileImageUrl","branchName","adoptOrgId",{"_43":44,"_32":45,"_38":150,"_47":-5,"_48":25,"_49":50,"_51":50,"_52":53,"_54":-5,"_18":55,"_56":-5,"_57":25,"_58":-5,"_59":-5,"_60":61,"_62":63,"_64":-5,"_65":-5,"_66":-5,"_67":-5,"_68":69,"_70":69,"_71":72,"_73":72,"_74":72,"_75":69,"_76":69,"_77":151,"_79":152,"_81":153,"_83":25},"tree",[145],"files",[145],"counts",{"_134":61,"_135":136,"_137":61,"_138":139,"_140":141},"contents",[126],"upstreamProject","upstreamBranch","emptyEnvVars","flags",[],"hasHttpFiles",true,"grants","orgs","customDomains",{},{"_43":127,"_32":28,"_30":31,"_128":-5,"_41":42,"_40":14,"_129":25,"_38":130,"_13":131,"_33":34,"_35":37,"_24":25,"_132":-5,"_27":133},"4b195ae0-51d5-11f1-b9fb-ee650bb23af1","parentId","latest",["D",1779011195490],4,"label",[28],"pulls","branches",1,"forks","envVars",{"_142":61,"_144":61},"connections",{"_142":61,"_143":61},"count","unfilledCount","emptyCount",{"_43":127,"_32":28,"_30":31,"_128":-5,"_41":42,"_40":14,"_129":25,"_38":146,"_13":131,"_33":34,"_35":37,"_24":25,"_132":-5,"_147":148,"_27":149},["D",1779011195490],"children",[],[28],["D",1778823148771],{"_95":94,"_103":99},{"_95":94,"_96":97,"_98":99,"_18":55,"_100":101,"_102":25},{"_13":84,"_32":85,"_43":42,"_86":44,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":-5,"_92":45,"_93":94},"ctx",{"_77":-5,"_123":187,"_188":189,"_190":191,"_192":25,"_193":25,"_194":25,"_195":121,"_196":-5,"_197":-7,"_198":-7,"_199":61,"_200":25,"_201":25,"_202":-5,"_203":25,"_204":25,"_205":25,"_206":-5},"ENV",{"_158":159,"_160":161,"_162":85,"_163":164,"_165":166,"_167":168,"_169":170,"_171":172,"_173":174,"_175":176,"_177":178,"_179":180,"_181":182,"_183":184,"_185":186},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","5015b8df46402d8bba4f485ecc37aec202b0e06b","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]

Přehled výsledků

SWE-bench výsledky

Django-16910: detail effort gradientu

Pass rate per effort

Průměrná cena per effort

Syntetické úlohy

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

💡 Max má hodnotu pro multi-aspect bugy

⚠️ Debug bez testu = nulový efekt

⚠️ Python 3.14 limituje SWE-bench

Doporučení

Přehled výsledků

Pass rate per effort

Průměrná cena per effort

Tokeny per effort

Iterace (turns) per effort

Per-check breakdown

Efektivita: cena vs. kvalita

Tokeny a iterace napříč úlohami

Token consumption per effort (stacked)

Průměrné iterace per effort

SWE-bench výsledky

Syntetické úlohy

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

💡 Max ≠ lepší výsledek

⚠️ Debug bez testu = nulový efekt

⚠️ Python 3.14 limituje SWE-bench

Doporučení

Diff · v5