[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8,"_9":10},"actionData","errors","root",{"_153":154,"_155":156},"routes/_app.x.$handle.$name",{"_95":94,"_32":45,"_104":85,"_27":53,"_13":14,"_105":-5,"_11":106,"_107":108,"_109":110,"_111":112,"_113":114,"_20":-5,"_115":-5,"_116":-5,"_117":25,"_118":119,"_120":121,"_122":-5,"_123":-5,"_124":125},"routes/_app.x.$handle.$name.branch.$branch.version.$version.diff",{"_11":12,"_13":14,"_15":16,"_17":-5,"_18":-5},"project",{"_43":44,"_32":45,"_38":46,"_47":-5,"_48":25,"_49":50,"_51":50,"_52":53,"_54":-5,"_18":55,"_56":-5,"_57":25,"_58":-5,"_59":-5,"_60":61,"_62":63,"_64":-5,"_65":-5,"_66":-5,"_67":-5,"_68":69,"_70":69,"_71":72,"_73":72,"_74":72,"_75":69,"_76":69,"_77":78,"_79":80,"_81":82,"_83":25},"version",2,"changes",[19],"aiChatInfo","userId",{"_20":21,"_22":23,"_24":25,"_26":25,"_27":28,"_29":28},"file",{"_30":31,"_32":28,"_33":34,"_35":37,"_38":39,"_40":14,"_41":42,"_24":25},"previous",{"_30":31,"_32":28,"_33":34,"_35":36},"deleted",false,"deletedHere","path","main.http.ts","latestPath","fileId","7768c20a-501f-11f1-afc4-ee650bb23af1","name","fileType","http","content","// Claude Code Effort Benchmark — Phase 1\n// https://github.com/tangero/cc-effort-test\n\nexport default async function(req: Request): Promise {\n const html = `\n\n\n\n\nClaude Code Effort Benchmark\n\n\n\n\n\n

\n \n \n \n

Claude Code Effort Benchmark Phase 1

Empirické měření dopadu parametru --effort na výkon Claude Code

\n\n

\n\n\n

O projektu

\n Benchmark měří dopad parametru --effort na spotřebu tokenů, čas, náklady a kvalitu výsledku modelu Claude Opus 4.7. Testovány úrovně low, medium, high a max, 3 repetice na cell.\n

Tři typy úloh: mechanická rename, specifikovaná implementace a debugging se dvěma záměrnými bugy.

runů

úlohy

—

verify pass

—

náklady

\n\n\n

Testované úlohy

01_rename mechanical

Přejmenovat userId → accountId v TypeScript projektu. 57 výskytů, 3 záměrné pasti. Verify: 7 objective checks.

02_implement_ico specified implementation

Implementovat validateICO() — validace českého IČO (mod 11 checksum). Skrytá sada 9 test cases. Verify: 5 checks.

03_debug_order bounded debugging

Dva záměrné bugy v e-commerce API. Bug A způsobuje padající test (viditelný). Bug B je skrytá business logic chyba (discount aplikován na shipping) — odhalitelná pouze čtením kódu. Verify: 5 checks.

\n\n\n

01_rename — Výsledky

Průměrný čas (s)

Průměrná cena (USD)

Tool calls per run

Detailní tabulka

Effort	Run	Čas (s)	Cena ($)	Tools	Breakdown

\n\n

02_implement_ico — Výsledky

Průměrný čas (s)

Median u medium je 55s; 2 pomalé instance zvyšují průměr.

Průměrná cena (USD)

Tool calls per run

Detailní tabulka

Effort	Run	Čas (s)	Cena ($)	Tools	Breakdown

\n\n

03_debug_order — Výsledky

\n Klíčový výsledek: Všechny runy (low i max) dosáhly skóre 0.6/1.0 — opravily Bug A (padající test), ale Bug B (shipping v discount base) nepřišly. Model bez explicitního failing testu business logic chybu nedetekoval ani na max effort.\n

Verify score (0–1)

Průměrná cena (USD)

Detailní tabulka

\n \n \n \n

Effort	Run	Čas (s)	Cena ($)	Tools	Score	Bug A	Bug B

\n\n\n

Klíčová zjištění

\n\n\n

Interpretace a doporučení

Mechanické úlohy

Pro rename a specifikovanou implementaci jsou všechny effort levely funkčně ekvivalentní. low je nejlevnější volba. max je kontraproduktivní — výrazně vyšší cena a čas, přičemž u 01_rename použil nástroje mimo povolený seznam (TodoWrite).

Debugging bez failing testu

Ani max effort nenašel Bug B, který neměl žádný viditelný failing test. Modely bez explicitního signálu selhání neprovádí proaktivní business logic review. Pro komplexní debugging je nezbytná dobrá testovací sada jako guidance.

Doporučení pro vývojáře

\n \n \n \n \n \n \n \n \n

Use case	Effort	Proč
Refaktoring, rename, bulk změny	low	Nejlevnější, 100% úspěšnost, rychlý
Implementace dle specifikace	medium high	high je nejkonzistentnější, medium levnější
Debugging s dobrými testy	high	Testy jako guidance stačí; max nepřidává hodnotu
Debugging bez testů / open-ended	high max	Netestováno dostatečně — Phase 2 téma

\n\n\n

Limity studie

n=3 nedává statistickou signifikanci
Pouze 3 úlohy; chybí open-ended design a analýza
CLAUDE.md byl aktivní a mohl ovlivnit chování
Anthropic-side variabilita: caching, latence

Bug B u 03_debug_order je možná příliš implicitní — žádný hint
Pouze claude-opus-4-7; Sonnet 4.6 plánován
xhigh nebyl testován (Sonnet ho nepodporuje)
Judge pipeline (LLM-as-judge) zatím nespuštěna

\n\n

\n\n\n\n\n\n`;\n return new Response(html, {\n headers: { \"content-type\": \"text/html; charset=utf-8\" },\n });\n}\n","// Claude Code Effort Benchmark — Phase 1+SWE\n// Auto-generated by scripts/generate_report.py\nexport default async function(req: Request): Promise {\n const html = String.raw`\n\n\n\n\nClaude Code Effort Benchmark\n\n\n\n\n

Claude Code Effort Benchmark Phase 1+SWE

Empirické měření dopadu parametru --effort · claude-opus-4-7

100

celkem runů

úlohy

74/100

verify pass

$57.6

celkem náklady

Přehled výsledků

Úloha	low	medium	high	max	Effort gradient

SWE-bench výsledky

Reálné bugy z open-source projektů.

Instance	low	medium	high	max	Finding

Django-16910: detail effort gradientu

only()+select_related() na reverse OneToOneField. Nejsilnější effort gradient v benchmarku.

Pass rate per effort

Průměrná cena per effort

Effort	n	Pass%	Avg score	Avg cost	Avg wall	Avg tools

Syntetické úlohy

Na dobře specifikovaných úlohách jsou všechny effort úrovně funkčně ekvivalentní.

Pass rate — 06_strict_scope

Průměrná cena — 01_rename

Klíčová zjištění

💡 SWE-bench diferenciuje effort

Na reálných bugech (django-16910) low/medium konzistentně selávají, high uspívá. Na jednoduchých bugech i low stačí.

💡 Max ≠ lepší výsledek

Max effort stojí 2–4× více než high, ale výsledek je stejný. High je sweet spot pro reálné bugy.

⚠️ Debug bez testu = nulový efekt

03_debug_order Bug B nebyl nalezen ani max effortem. Failing testy jsou nezbytný guidance.

⚠️ Python 3.14 limituje SWE-bench

Pre-2022 projekty jsou nekompatibilní (odstraňeny moduly cgi, distutils). Používejte Django 4.2+ / Sympy 1.11+.

Doporučení

Use case	Effort	Proč

\n\n\n`;\n return new Response(html, { headers: {'content-type': 'text/html;charset=utf-8'} });\n}\n","createdAt",["D",1778949757311],"branchVersion","branchId","7683ffee-501f-11f1-afc4-ee650bb23af1","id","76834068-501f-11f1-afc4-ee650bb23af1","effortBenchmark",["D",1778823148771],"deletedAt","legacyEnvVars","privacy","public","httpPrivacy","description","","imageUrl","7241cb49-222a-4e9a-af99-25b2615eda07","pinnedFileId","pinPreview","pinMigrated","screenshotVersion","likeCount",0,"orgId","019e0b3d-7f60-7428-8330-1beb489e3416","tursoName","tursoHostname","tursoMeasuredAt","pinned","userPerm","r","valPerm","blobPerm","rw","emailPerm","sqlitePerm","projectPerm","telemetryPerm","user",{"_95":94,"_103":99},"org",{"_95":94,"_96":97,"_98":99,"_18":55,"_100":101,"_102":25},"branch",{"_13":84,"_32":85,"_43":42,"_86":44,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":-5,"_92":45,"_93":94},"isOwned",23,"main","projectId","forkedBranchId","forkedBranchVersion","forkedProjectPrivacy","forkedProjectId","forkedProjectOrgId","projectName","userHandle","patrickzandl","handle","displayName","Patrick Zandl","avatar","https://lh3.googleusercontent.com/a/ACg8ocK47-vrp0IIPUC8DK3BcjE57lcyPd28xqviX34gk6fkTziKkbep=s96-c","tier","free","canRestrictVals","profileImageUrl","branchName","adoptOrgId",{"_43":44,"_32":45,"_38":149,"_47":-5,"_48":25,"_49":50,"_51":50,"_52":53,"_54":-5,"_18":55,"_56":-5,"_57":25,"_58":-5,"_59":-5,"_60":61,"_62":63,"_64":-5,"_65":-5,"_66":-5,"_67":-5,"_68":69,"_70":69,"_71":72,"_73":72,"_74":72,"_75":69,"_76":69,"_77":150,"_79":151,"_81":152,"_83":25},"tree",[144],"files",[144],"counts",{"_134":61,"_135":131,"_136":61,"_137":138,"_139":140},"contents",[126],"upstreamProject","upstreamBranch","emptyEnvVars","flags",[],"hasHttpFiles",true,"grants","orgs","customDomains",{},{"_43":127,"_32":28,"_30":31,"_128":-5,"_41":42,"_40":14,"_129":25,"_38":130,"_13":131,"_33":34,"_35":37,"_24":25,"_132":-5,"_27":133},"3f1a207e-5146-11f1-8a13-ee650bb23af1","parentId","latest",["D",1778949757311],1,"label",[28],"pulls","branches","forks","envVars",{"_141":61,"_143":61},"connections",{"_141":61,"_142":61},"count","unfilledCount","emptyCount",{"_43":127,"_32":28,"_30":31,"_128":-5,"_41":42,"_40":14,"_129":25,"_38":145,"_13":131,"_33":34,"_35":37,"_24":25,"_132":-5,"_146":147,"_27":148},["D",1778949757311],"children",[],[28],["D",1778823148771],{"_95":94,"_103":99},{"_95":94,"_96":97,"_98":99,"_18":55,"_100":101,"_102":25},{"_13":84,"_32":85,"_43":42,"_86":44,"_87":-5,"_88":-5,"_89":-5,"_90":-5,"_91":-5,"_92":45,"_93":94},"ctx",{"_77":-5,"_123":186,"_187":188,"_189":190,"_191":25,"_192":25,"_193":25,"_194":121,"_195":-5,"_196":-7,"_197":-7,"_198":61,"_199":25,"_200":25,"_201":-5,"_202":25,"_203":25,"_204":25,"_205":-5},"ENV",{"_157":158,"_159":160,"_161":85,"_162":163,"_164":165,"_166":167,"_168":169,"_170":171,"_172":173,"_174":175,"_176":177,"_178":179,"_180":181,"_182":183,"_184":185},"REACT_APP_SENTRY_DSN","https://586020d86f05489db63a554690acf1e9@o1398359.ingest.sentry.io/6724499","RENDER_GIT_COMMIT","e87eb04e707cf64e7e532c8e8ec395f209e17f15","RENDER_GIT_BRANCH","POSTHOG_API_TOKEN","phc_nmoyuAkStEZuUp69rYlg1DZgrIt19Gx0Y4Hwf8tGgJP","POSTHOG_API_HOST","https://wart.val.town","WEB_API_TEMPLATE","https://{val}.web.val.run","APP_URL","https://www.val.town","MODULE_URL","https://esm.town","VANITY_WEB_API_TEMPLATE","https://{label}.val.run/","DENOLS_URL","https://lsp.val.town","NODE_ENV","production","CLOUDFLARE_TURNSTILE_SITE_KEY","0x4AAAAAACtaIZJH7s9rBMZ6","GOOGLE_CLIENT_ID","875989696899-jtsqkhlbdv5jrlsk8631p7ipjg3c6o62.apps.googleusercontent.com","STRIPE_PUBLISHABLE_KEY","pk_live_51N2zvWEpZTTLipIkVEwXrUG9AT0yeOOEYxszarZT1jpsZV58Ly6wlg3SltZpHN26CUalOBmUSY7adQRQiSkCZz2500qSXNkf8Y","API_URL","https://api.val.town",[],"preferencesKeybindings","default","preferencesTheme","light","preferencesFixers","preferencesCopilot","preferencesSelectionAi","preferencesCloseBrackets","preferencesChatPanelState","preferencesDprint","preferencesDenofmt","notificationCount","flagDenols","isBusinessOrgMember","businessOrgHandle","isBusinessOrgAdmin","isHigherTownieTier","navCollapsed","currentOrgId"]

Diff · v2