agent-eval — results

24 arms in 7 cohorts over 26 tasks — 1126 trials, $230.29 spent, 1122 with a built report. Every delta is against its own cohort's none arm, so it measures the servers and not the model. A thumbnail is the component that trial actually produced; click it for the full report. Generated 2026-09-30 15:07 UTC.

overview

variant	tasks	trials	quality	Δ	pass@1	$/trial	cost ×	spend
deepseek-default 5 arms · 23 tasks
none baseline	22	66	0.90	—	44%	$0.05	1.00×	$2.99
both	23	69	0.94	+0.04	75%	$0.11	2.45×	$7.77
component-builder	20	60	0.94	+0.04	80%	$0.04	0.92×	$2.51
contracts	23	69	0.93	+0.05	72%	$0.11	2.32×	$7.26
design-tokens	20	60	0.91	+0.01	51%	$0.06	1.33×	$3.62
glm-default 4 arms · 20 tasks
none baseline	20	60	0.88	—	33%	$0.06	1.00×	$3.74
both	20	60	0.94	+0.06	68%	$0.10	1.56×	$5.85
component-builder	20	60	0.95	+0.06	55%	$0.07	1.10×	$4.11
design-tokens	20	60	0.88	+0.00	38%	$0.10	1.54×	$5.76
haiku-high 4 arms · 20 tasks
none baseline	20	60	0.82	—	42%	$0.19	1.00×	$11.14
both	20	60	0.92	+0.10	64%	$0.20	1.06×	$11.56
component-builder	20	60	0.92	+0.09	63%	$0.18	0.96×	$10.64
design-tokens	20	60	0.85	+0.02	46%	$0.21	1.13×	$12.43
haiku-low 4 arms · 20 tasks
none baseline	20	60	0.85	—	42%	$0.17	1.00×	$10.24
both	20	60	0.92	+0.07	68%	$0.21	1.22×	$12.47
component-builder	20	60	0.93	+0.08	65%	$0.18	1.07×	$11.00
design-tokens	20	60	0.88	+0.03	50%	$0.22	1.28×	$13.14
sonnet-high 4 arms · 5 tasks
none baseline	5	15	0.72	—	20%	$0.87	1.00×	$13.08
both	5	15	0.94	+0.23	73%	$2.16	2.47×	$32.35
component-builder	5	15	0.94	+0.23	86%	$1.38	1.58×	$19.99
design-tokens	5	15	0.74	+0.04	23%	$1.66	1.90×	$25.45
other 1 arms · 4 tasks
spike-novita-both	4	4	0.97	—	50%	$0.09	—	$0.37
haiku-paste 2 arms · 3 tasksfixture ships no `src/token/` — the pasted-snippet context of D-159, where there is nothing to grep. Absolute scores are not comparable to the repo cohorts above; only the within-cohort Δ is.
none baseline	3	9	0.95	—	0%	$0.12	1.00×	$1.06
design-tokens	3	9	0.98	+0.03	89%	$0.20	1.66×	$1.76

Rows are grouped by cohort — one model in one context. Δ and cost × are against that cohort's own none arm, never across cohorts, so a delta here is the servers' effect and not the model's or the fixture's. Δ is the mean of the per-task deltas, paired over the tasks both arms ran, so it is not the difference of the two quality columns whenever coverage differs.

models compared

cohort	none	both	component-builder	contracts	design-tokens	spike-novita-both	best	Δ	pass@1	$/trial none	$/trial best	s/trial
deepseek-default	0.87	0.97	0.98	0.97	0.91	—	component-builder	+0.11	100%	$0.07	$0.07	304s
glm-default	0.78	0.98	0.97	—	0.79	—	both	+0.20	56%	$0.12	$0.16	1430s
haiku-low	0.72	0.91	0.95	—	0.77	—	component-builder	+0.23	22%	$0.39	$0.42	417s
haiku-high	0.71	0.96	0.94	—	0.76	—	both	+0.25	56%	$0.48	$0.44	399s
sonnet-high	0.70	0.97	0.98	—	0.76	—	component-builder	+0.28	89%	$1.00	$1.45	513s
other	—	—	—	—	—	0.97	spike-novita-both	—	67%	—	$0.11	406s

Every figure is the mean over the 3 tasks run by all 6 cohorts, unweighted by trial count, so the columns answer the same question for each row even where total coverage differs. Quality columns are absolute; bold marks the best variant in a row. Δ is that variant's quality minus the cohort's own none arm — the servers' contribution at that model and effort, and the only cell here that is a controlled comparison. A larger Δ on a weaker baseline is the servers substituting for model capability. 1 cohort is held out for running a different fixture set; see the note on paste above.

quality deltas by task

deepseek-default 23 tasks · 5 arms

task	none baseline	both	component-builder	contracts	design-tokens
802-composite-from-two	0.91	0.00	0.00	0.00	0.00
804-story-conventions	0.96	0.00	0.00	0.00	0.00
806-inverted-context	0.97	0.00	0.00	0.00	0.00
810-atom-from-schema	0.77	+0.19	+0.18	+0.20	+0.02
811-token-intent	0.96	0.00	0.00	0.00	0.00
812-restyle-with-tokens	0.97	+0.01	0.00	−0.01	−0.01
816-typography-pairing	0.95	0.00	0.00	+0.01	0.00
817-responsive-tokens	0.97	0.00	0.00	+0.02	0.00
818-component-token-layer	0.93	+0.07	+0.07	+0.07	+0.04
820-extend-schema-safely	0.90	−0.01	0.00	0.00	0.00
824-api-from-behaviour	0.81	+0.17	+0.18	+0.19	−0.02
832-client-behaviour	0.77	+0.18	+0.22	+0.20	+0.11
836-behaviour-bugfix	0.97	0.00	0.00	0.00	0.00
840-reuse-over-native	0.86	+0.12	+0.12	+0.12	+0.03
842-reuse-edit	0.89	0.00	0.00	0.00	0.00
850-focus-return	1.00	0.00	0.00	0.00	0.00
852-a11y-repair	0.82	−0.03	−0.03	−0.03	0.00
860-restraint	0.87	+0.02	0.00	0.00	+0.02
861-token-restraint	0.95	0.00	0.00	0.00	0.00
862-api-freeze	0.85	+0.02	+0.02	+0.02	0.00
880-contract-lookup	—	0.42	—	0.53	—
890-pricing-plans	0.86	+0.12	—	+0.11	—
891-comparison-table	0.87	+0.12	—	+0.13	—
mean Δ	—	+0.04	+0.04	+0.05	+0.01

glm-default 20 tasks · 4 arms

task	none baseline	both	component-builder	design-tokens
802-composite-from-two	0.86	+0.05	+0.05	+0.01
804-story-conventions	0.96	0.00	0.00	0.00
806-inverted-context	0.97	+0.01	+0.03	0.00
810-atom-from-schema	0.68	+0.31	+0.30	−0.02
811-token-intent	0.96	0.00	+0.04	0.00
812-restyle-with-tokens	0.98	+0.01	0.00	0.00
816-typography-pairing	0.95	0.00	0.00	−0.01
817-responsive-tokens	0.97	+0.02	+0.03	0.00
818-component-token-layer	0.97	+0.03	+0.03	+0.01
820-extend-schema-safely	0.90	0.00	+0.01	0.00
824-api-from-behaviour	0.79	+0.17	+0.21	−0.01
832-client-behaviour	0.63	+0.35	+0.33	+0.03
836-behaviour-bugfix	0.97	−0.01	−0.01	−0.01
840-reuse-over-native	0.73	+0.25	+0.24	0.00
842-reuse-edit	0.89	0.00	0.00	0.00
850-focus-return	0.99	−0.01	0.00	−0.01
852-a11y-repair	0.84	0.00	0.00	+0.02
860-restraint	0.87	+0.05	+0.03	0.00
861-token-restraint	0.95	0.00	0.00	0.00
862-api-freeze	0.80	0.00	0.00	0.00
mean Δ	—	+0.06	+0.06	0.00

haiku-high 20 tasks · 4 arms

task	none baseline	both	component-builder	design-tokens
802-composite-from-two	0.91	+0.01	−0.16	0.00
804-story-conventions	0.81	+0.15	+0.15	0.00
806-inverted-context	0.95	+0.02	+0.02	+0.02
810-atom-from-schema	0.57	+0.35	+0.41	+0.10
811-token-intent	0.89	+0.10	+0.11	+0.08
812-restyle-with-tokens	0.86	+0.12	+0.12	+0.05
816-typography-pairing	0.92	+0.05	+0.01	+0.02
817-responsive-tokens	0.96	+0.03	+0.02	+0.02
818-component-token-layer	0.85	+0.10	+0.15	+0.08
820-extend-schema-safely	0.89	+0.01	+0.01	+0.01
824-api-from-behaviour	0.68	+0.23	+0.21	+0.02
832-client-behaviour	0.60	+0.34	+0.33	+0.05
836-behaviour-bugfix	0.96	+0.01	+0.01	+0.01
840-reuse-over-native	0.68	+0.28	+0.23	+0.07
842-reuse-edit	0.60	−0.01	+0.13	−0.17
850-focus-return	0.90	+0.10	+0.10	+0.10
852-a11y-repair	0.80	+0.04	+0.01	+0.03
860-restraint	0.87	0.00	0.00	0.00
861-token-restraint	0.94	+0.01	+0.01	+0.01
862-api-freeze	0.82	−0.02	−0.02	−0.02
mean Δ	—	+0.10	+0.09	+0.02

haiku-low 20 tasks · 4 arms

task	none baseline	both	component-builder	design-tokens
802-composite-from-two	0.86	+0.05	+0.05	+0.05
804-story-conventions	0.86	+0.10	+0.10	−0.05
806-inverted-context	0.97	0.00	0.00	0.00
810-atom-from-schema	0.60	+0.38	+0.34	+0.05
811-token-intent	0.96	+0.01	+0.04	0.00
812-restyle-with-tokens	0.97	0.00	−0.01	−0.01
816-typography-pairing	0.92	+0.04	0.00	+0.03
817-responsive-tokens	0.97	0.00	0.00	0.00
818-component-token-layer	0.93	+0.07	+0.07	+0.04
820-extend-schema-safely	0.90	0.00	0.00	0.00
824-api-from-behaviour	0.77	+0.02	+0.12	0.00
832-client-behaviour	0.61	+0.30	+0.37	+0.07
836-behaviour-bugfix	0.96	+0.01	+0.01	+0.01
840-reuse-over-native	0.59	+0.27	+0.32	+0.07
842-reuse-edit	0.71	+0.14	+0.18	+0.18
850-focus-return	1.00	0.00	0.00	0.00
852-a11y-repair	0.79	+0.05	+0.05	+0.03
860-restraint	0.87	0.00	0.00	0.00
861-token-restraint	0.95	0.00	0.00	0.00
862-api-freeze	0.80	0.00	−0.05	+0.07
mean Δ	—	+0.07	+0.08	+0.03

sonnet-high 5 tasks · 4 arms

task	none baseline	both	component-builder	design-tokens
810-atom-from-schema	0.62	+0.33	+0.31	+0.02
812-restyle-with-tokens	0.84	+0.14	+0.14	+0.04
832-client-behaviour	0.60	+0.37	+0.38	+0.07
840-reuse-over-native	0.66	+0.29	+0.31	+0.07
860-restraint	0.87	0.00	0.00	0.00
mean Δ	—	+0.23	+0.23	+0.04

other 4 tasks · 1 arms

task	spike-novita-both
812-restyle-with-tokens	0.96
816-typography-pairing	0.95
832-client-behaviour	0.98
840-reuse-over-native	0.97
mean Δ	—

haiku-paste 3 tasks · 2 arms

fixture ships no src/token/ — the pasted-snippet context of D-159, where there is nothing to grep. Absolute scores are not comparable to the repo cohorts above; only the within-cohort Δ is.

task	none baseline	design-tokens
871-paste-token-intent	0.96	0.00
872-paste-component-token-layer	0.97	+0.02
873-paste-responsive-tokens	0.92	+0.05
mean Δ	—	+0.03

One matrix per cohort, so every column is the same model in the same context and every row is a task that cohort ran. The baseline column carries absolute quality; every other cell is that variant's quality on that task minus the baseline's. Tint saturates at ±0.25. Cells in italics are absolute quality where no baseline exists to subtract. Task names link to the run below.

802-composite-from-two

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.91 ±0.00	—	67%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — failed
both	0.91 ±0.00	+0.00	100%	$0.10	3.71×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.91 ±0.00	+0.00	100%	$0.03	1.20×	0.000	run-1 — passed run-2 — passed run-3 — passed
contracts	0.91 ±0.00	+0.00	100%	$0.05	1.96×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens run invalid	0.91 ±0.00	+0.00	100%	$0.06	2.09×	0.000	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.86 ±0.07	—	0%	$0.07	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.91 ±0.00	+0.05	100%	$0.10	1.49×	1.516	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.91 ±0.00	+0.05	67%	$0.06	0.93×	—	run-1 — passed run-2 — passed run-3 — failed
design-tokens	0.87 ±0.05	+0.01	0%	$0.11	1.59×	0.302	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.91 ±0.00	—	67%	$0.26	—	—	run-1 — passed run-2 — failed run-3 — passed
both	0.92 ±0.02	+0.01	67%	$0.29	1.12×	0.332	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.74 ±0.23	−0.16	67%	$0.21	0.81×	—	run-1 — passed not built run-2 — failed run-3 — passed
design-tokens	0.90 ±0.00	−0.00	67%	$0.32	1.23×	-0.039	run-1 — passed run-2 — failed run-3 — passed
haiku-low
none	0.86 ±0.07	—	67%	$0.16	—	—	run-1 — passed run-2 — passed run-3 — failed
both	0.90 ±0.01	+0.05	33%	$0.36	2.18×	0.246	run-1 — passed run-2 — failed run-3 — failed
component-builder	0.90 ±0.00	+0.05	100%	$0.19	1.19×	1.547	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.91 ±0.00	+0.05	100%	$0.17	1.03×	9.225	run-1 — passed run-2 — passed run-3 — passed

804-story-conventions

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.96 ±0.00	—	0%	$0.04	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.00	+0.00	67%	$0.05	1.39×	0.000	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.96 ±0.00	+0.00	33%	$0.03	0.92×	—	run-1 — failed run-2 — failed run-3 — passed
contracts	0.96 ±0.00	+0.00	100%	$0.07	1.94×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.96 ±0.00	+0.00	0%	$0.07	1.95×	0.000	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.96 ±0.00	—	0%	$0.08	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.00	+0.00	100%	$0.07	0.95×	—	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.96 ±0.00	+0.00	100%	$0.04	0.55×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.96 ±0.00	+0.00	0%	$0.09	1.14×	0.000	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.81 ±0.00	—	0%	$0.13	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.00	+0.15	67%	$0.15	1.13×	8.579	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.96 ±0.00	+0.15	67%	$0.14	1.08×	13.979	run-1 — passed run-2 — failed run-3 — passed
design-tokens	0.81 ±0.00	+0.00	0%	$0.17	1.34×	0.000	run-1 — failed run-2 — failed run-3 — failed
haiku-low
none	0.86 ±0.07	—	0%	$0.14	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.00	+0.10	67%	$0.17	1.18×	3.842	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.96 ±0.00	+0.10	67%	$0.13	0.90×	—	run-1 — passed run-2 — failed run-3 — passed
design-tokens	0.81 ±0.00	−0.05	0%	$0.20	1.36×	-0.955	run-1 — failed run-2 — failed run-3 — failed

806-inverted-context

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.97 ±0.00	—	100%	$0.04	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.97 ±0.00	+0.00	100%	$0.12	3.01×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.00	+0.00	100%	$0.06	1.37×	0.000	run-1 — passed run-2 — passed run-3 — passed
contracts	0.97 ±0.00	+0.00	100%	$0.16	3.81×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.00	100%	$0.07	1.75×	0.000	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.97 ±0.00	—	67%	$0.09	—	—	run-1 — passed run-2 — failed run-3 — passed
both	0.98 ±0.01	+0.01	100%	$0.13	1.42×	0.246	run-1 — passed run-2 — passed run-3 — passed
component-builder	1.00 ±0.00	+0.03	33%	$0.12	1.25×	1.221	run-1 — failed run-2 — passed run-3 — failed
design-tokens	0.97 ±0.00	+0.00	100%	$0.11	1.14×	0.000	run-1 — passed run-2 — passed run-3 — passed
haiku-high
none	0.95 ±0.00	—	67%	$0.12	—	—	run-1 — passed run-2 — failed run-3 — passed
both	0.97 ±0.00	+0.02	100%	$0.10	0.79×	—	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.00	+0.02	100%	$0.12	0.97×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.02	100%	$0.13	1.05×	2.999	run-1 — passed run-2 — passed run-3 — passed
haiku-low
none	0.97 ±0.00	—	100%	$0.08	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.97 ±0.00	+0.00	100%	$0.15	1.80×	0.018	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.00	+0.00	100%	$0.13	1.58×	0.024	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.00	100%	$0.15	1.80×	0.018	run-1 — passed run-2 — passed run-3 — passed

810-atom-from-schema

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.77 ±0.01	—	0%	$0.07	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.01	+0.19	67%	$0.16	2.31×	2.140	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.96 ±0.01	+0.18	67%	$0.10	1.56×	4.893	run-1 — passed run-2 — failed run-3 — passed
contracts	0.97 ±0.02	+0.20	67%	$0.15	2.30×	2.290	run-1 — failed run-2 — passed run-3 — passed
design-tokens	0.80 ±0.03	+0.02	0%	$0.12	1.74×	0.473	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.68 ±0.01	—	0%	$0.11	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.01	+0.31	100%	$0.20	1.81×	3.368	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.98 ±0.01	+0.30	100%	$0.19	1.70×	3.796	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.65 ±0.01	−0.02	0%	$0.19	1.66×	-0.332	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.57 ±0.06	—	0%	$0.39	—	—	run-1 — failed run-2 — failed run-3 — failed
both run invalid	0.92 ±0.07	+0.35	50%	$0.61	1.58×	1.566	run-1 — failed run-2 — failed run-3 — passed
component-builder	0.98 ±0.01	+0.41	100%	$0.48	1.25×	4.263	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.66 ±0.03	+0.10	0%	$0.50	1.29×	0.863	run-1 — failed run-2 — failed run-3 — failed
haiku-low
none	0.60 ±0.07	—	0%	$0.46	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.99 ±0.01	+0.38	67%	$0.60	1.32×	2.611	run-1 — failed run-2 — passed run-3 — passed
component-builder	0.95 ±0.07	+0.34	67%	$0.47	1.02×	31.737	run-1 — passed run-2 — passed run-3 — failed
design-tokens	0.66 ±0.03	+0.05	0%	$0.56	1.23×	0.504	run-1 — failed run-2 — failed run-3 — failed
sonnet-high
none	0.62 ±0.01	—	0%	$1.27	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.95 ±0.02	+0.33	67%	$3.51	2.76×	0.148	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.93 ±0.05	+0.31	67%	$2.20	1.74×	0.333	run-1 — failed run-2 — passed run-3 — passed
design-tokens	0.64 ±0.01	+0.02	0%	$2.21	1.74×	0.023	run-1 — failed run-2 — failed run-3 — failed

811-token-intent

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.96 ±0.00	—	0%	$0.06	—	—	run-1 — failed run-2 — failed run-3 — failed
both run invalid	0.96 ±0.00	+0.00	100%	$0.16	2.80×	0.000	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.96 ±0.00	+0.00	33%	$0.05	0.85×	—	run-1 — passed run-2 — failed run-3 — failed
contracts	0.96 ±0.00	+0.00	100%	$0.16	2.86×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.96 ±0.00	+0.00	100%	$0.07	1.27×	0.000	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.96 ±0.00	—	0%	$0.07	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.00	+0.00	33%	$0.06	0.87×	—	run-1 — passed run-2 — failed run-3 — failed
component-builder	1.00 ±0.00	+0.04	0%	$0.08	1.03×	17.259	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.96 ±0.00	+0.00	100%	$0.06	0.86×	—	run-1 — passed run-2 — passed run-3 — passed
haiku-high
none	0.89 ±0.04	—	33%	$0.13	—	—	run-1 — passed run-2 — failed run-3 — failed
both	0.98 ±0.02	+0.10	0%	$0.23	1.77×	0.973	run-1 — failed run-2 — failed run-3 — failed
component-builder	1.00 ±0.00	+0.11	0%	$0.15	1.10×	8.532	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.96 ±0.00	+0.08	67%	$0.20	1.50×	1.173	run-1 — passed run-2 — failed run-3 — passed
haiku-low
none	0.96 ±0.00	—	33%	$0.15	—	—	run-1 — failed run-2 — failed run-3 — passed
both	0.97 ±0.02	+0.01	67%	$0.15	1.04×	2.050	run-1 — failed run-2 — passed run-3 — passed
component-builder	1.00 ±0.00	+0.04	0%	$0.15	1.04×	6.563	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.96 ±0.00	+0.00	100%	$0.22	1.49×	0.000	run-1 — passed run-2 — passed run-3 — passed

812-restyle-with-tokens

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.97 ±0.02	—	0%	$0.08	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.99 ±0.02	+0.01	100%	$0.26	3.40×	0.071	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.02	−0.00	100%	$0.07	0.94×	—	run-1 — passed run-2 — passed run-3 — passed
contracts	0.96 ±0.00	−0.01	100%	$0.16	2.12×	-0.151	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.03	−0.01	0%	$0.14	1.77×	-0.087	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.98 ±0.01	—	0%	$0.08	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.99 ±0.02	+0.01	100%	$0.16	2.05×	0.118	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.02	−0.00	100%	$0.09	1.14×	-0.302	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.01	−0.00	0%	$0.12	1.53×	-0.102	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.86 ±0.00	—	0%	$0.33	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.01	+0.12	100%	$0.23	0.70×	—	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.98 ±0.01	+0.12	100%	$0.22	0.67×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.90 ±0.04	+0.05	0%	$0.26	0.78×	—	run-1 — failed run-2 — failed not built run-3 — failed
haiku-low
none	0.97 ±0.02	—	0%	$0.32	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.02	+0.00	67%	$0.33	1.04×	0.226	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.97 ±0.01	−0.01	0%	$0.21	0.66×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.96 ±0.02	−0.01	0%	$0.45	1.39×	-0.077	run-1 — failed run-2 — failed run-3 — failed
sonnet-high
none	0.84 ±0.00	—	0%	$0.44	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.01	+0.14	100%	$1.79	4.12×	0.101	run-1 — passed run-2 — passed run-3 — passed
component-builder run invalid	0.98 ±0.00	+0.14	100%	$0.69	1.59×	0.532	run-1 — passed run-2 — passed run-3 — passed
design-tokens run invalid	0.88 ±0.00	+0.04	0%	$1.28	2.93×	0.053	run-1 — failed run-2 — failed run-3 — failed
other
spike-novita-both	0.96 ±0.00	—	100%	$0.08	—	—	not built run-1 — passed

816-typography-pairing

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.95 ±0.00	—	0%	$0.03	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.95 ±0.00	+0.00	0%	$0.04	1.34×	0.000	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.95 ±0.00	+0.00	0%	$0.03	1.05×	0.000	run-1 — failed run-2 — failed run-3 — failed
contracts	0.96 ±0.02	+0.01	33%	$0.09	3.41×	0.201	run-1 — passed run-2 — failed run-3 — failed
design-tokens	0.95 ±0.00	+0.00	0%	$0.06	2.10×	0.000	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.95 ±0.00	—	0%	$0.03	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.95 ±0.03	+0.00	33%	$0.08	2.94×	0.000	run-1 — failed run-2 — passed run-3 — failed
component-builder	0.95 ±0.00	+0.00	0%	$0.04	1.34×	0.000	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.94 ±0.02	−0.01	0%	$0.06	2.18×	-0.407	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.92 ±0.01	—	0%	$0.09	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.02	+0.05	67%	$0.09	1.07×	7.994	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.94 ±0.02	+0.01	0%	$0.08	0.93×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.95 ±0.03	+0.02	33%	$0.10	1.13×	2.120	run-1 — passed run-2 — failed run-3 — failed
haiku-low
none	0.92 ±0.02	—	0%	$0.07	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.04	+0.04	67%	$0.08	1.02×	22.417	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.92 ±0.02	+0.00	0%	$0.11	1.52×	0.000	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.95 ±0.03	+0.03	33%	$0.11	1.47×	0.750	run-1 — failed run-2 — passed run-3 — failed
other
spike-novita-both	0.95 ±0.00	—	0%	$0.05	—	—	not built run-1 — failed

817-responsive-tokens

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.97 ±0.00	—	67%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — failed
both	0.97 ±0.00	+0.00	100%	$0.05	1.77×	0.173	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.00	+0.00	100%	$0.03	1.34×	0.397	run-1 — passed run-2 — passed run-3 — passed
contracts	0.99 ±0.01	+0.02	100%	$0.07	2.66×	0.524	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.00	100%	$0.04	1.65×	0.207	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.97 ±0.00	—	67%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — failed
both	0.99 ±0.01	+0.02	100%	$0.09	2.63×	0.428	run-1 — passed run-2 — passed run-3 — passed
component-builder	1.00 ±0.00	+0.03	100%	$0.03	0.94×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.00	100%	$0.06	1.86×	0.125	run-1 — passed run-2 — passed run-3 — passed
haiku-high
none	0.96 ±0.00	—	67%	$0.06	—	—	run-1 — passed run-2 — failed run-3 — passed
both	0.98 ±0.01	+0.03	67%	$0.09	1.47×	0.873	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.97 ±0.00	+0.02	33%	$0.06	1.05×	4.870	run-1 — failed run-2 — passed run-3 — failed
design-tokens	0.97 ±0.00	+0.02	100%	$0.10	1.63×	0.402	run-1 — passed run-2 — passed run-3 — passed
haiku-low
none	0.97 ±0.00	—	67%	$0.04	—	—	run-1 — failed run-2 — passed run-3 — passed
both	0.97 ±0.00	+0.00	67%	$0.11	2.38×	0.000	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.97 ±0.00	+0.00	100%	$0.06	1.38×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.00	67%	$0.09	1.96×	0.000	run-1 — passed run-2 — failed run-3 — passed

818-component-token-layer

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.93 ±0.00	—	0%	$0.05	—	—	run-1 — failed run-2 — failed run-3 — failed
both	1.00 ±0.00	+0.07	100%	$0.11	2.36×	1.012	run-1 — passed run-2 — passed run-3 — passed
component-builder	1.00 ±0.00	+0.07	100%	$0.03	0.56×	—	run-1 — passed run-2 — passed run-3 — passed
contracts	1.00 ±0.00	+0.07	33%	$0.12	2.54×	0.894	run-1 — failed run-2 — passed run-3 — failed
design-tokens	0.97 ±0.00	+0.04	0%	$0.07	1.35×	2.228	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.97 ±0.01	—	0%	$0.05	—	—	run-1 — failed run-2 — failed run-3 — failed
both	1.00 ±0.00	+0.03	100%	$0.09	1.88×	0.660	run-1 — passed run-2 — passed run-3 — passed
component-builder	1.00 ±0.00	+0.03	0%	$0.04	0.77×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.98 ±0.01	+0.01	33%	$0.11	2.17×	0.131	run-1 — failed run-2 — passed run-3 — failed
haiku-high
none	0.85 ±0.01	—	0%	$0.19	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.95 ±0.07	+0.10	33%	$0.14	0.71×	—	run-1 — passed not built run-2 — failed run-3 — failed
component-builder	1.00 ±0.00	+0.15	100%	$0.10	0.54×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.94 ±0.05	+0.08	0%	$0.34	1.78×	0.569	run-1 — failed run-2 — failed run-3 — failed
haiku-low
none	0.93 ±0.05	—	0%	$0.16	—	—	run-1 — failed run-2 — failed run-3 — failed
both	1.00 ±0.00	+0.07	100%	$0.20	1.28×	1.582	run-1 — passed run-2 — passed run-3 — passed
component-builder	1.00 ±0.00	+0.07	100%	$0.11	0.73×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.04	0%	$0.30	1.95×	0.246	run-1 — failed run-2 — failed run-3 — failed

820-extend-schema-safely

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.90 ±0.00	—	67%	$0.03	—	—	run-1 — passed run-2 — failed run-3 — passed
both	0.90 ±0.00	−0.01	67%	$0.06	2.22×	-0.186	run-1 — failed run-2 — passed run-3 — passed
component-builder	0.90 ±0.00	−0.00	33%	$0.03	1.03×	-3.742	run-1 — failed run-2 — failed run-3 — passed
contracts	0.91 ±0.01	+0.00	67%	$0.08	3.16×	0.057	run-1 — passed run-2 — failed run-3 — passed
design-tokens	0.90 ±0.00	−0.00	67%	$0.05	1.97×	-0.152	run-1 — passed run-2 — passed run-3 — failed
glm-default
none	0.90 ±0.00	—	33%	$0.07	—	—	run-1 — failed run-2 — failed run-3 — passed
both	0.90 ±0.00	+0.00	67%	$0.07	0.90×	—	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.91 ±0.01	+0.01	0%	$0.04	0.59×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.90 ±0.00	+0.00	33%	$0.07	0.89×	—	run-1 — passed run-2 — failed run-3 — failed
haiku-high
none	0.89 ±0.00	—	100%	$0.10	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.90 ±0.00	+0.01	100%	$0.11	1.16×	0.518	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.90 ±0.00	+0.01	100%	$0.09	0.94×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.90 ±0.00	+0.01	67%	$0.09	0.98×	—	run-1 — passed run-2 — passed run-3 — failed
haiku-low
none	0.90 ±0.00	—	67%	$0.14	—	—	run-1 — failed run-2 — passed run-3 — passed
both	0.90 ±0.00	+0.00	100%	$0.12	0.83×	—	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.90 ±0.00	−0.00	100%	$0.08	0.54×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.90 ±0.00	+0.00	33%	$0.10	0.69×	—	run-1 — failed run-2 — passed run-3 — failed

824-api-from-behaviour

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.81 ±0.02	—	0%	$0.05	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.01	+0.17	100%	$0.20	4.05×	1.123	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.99 ±0.01	+0.18	100%	$0.07	1.31×	11.683	run-1 — passed run-2 — passed run-3 — passed
contracts	0.99 ±0.00	+0.19	100%	$0.17	3.43×	1.550	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.79 ±0.00	−0.02	0%	$0.09	1.84×	-0.472	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.79 ±0.02	—	0%	$0.09	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.00	+0.17	100%	$0.18	1.96×	1.894	run-1 — passed run-2 — passed run-3 — passed
component-builder	1.00 ±0.00	+0.21	100%	$0.13	1.43×	5.254	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.78 ±0.01	−0.01	0%	$0.17	1.89×	-0.119	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.68 ±0.06	—	0%	$0.48	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.91 ±0.07	+0.23	67%	$0.46	0.96×	—	run-1 — failed run-2 — passed run-3 — passed
component-builder	0.89 ±0.07	+0.21	33%	$0.39	0.81×	—	run-1 — passed run-2 — failed run-3 — failed
design-tokens	0.70 ±0.08	+0.02	0%	$0.68	1.41×	0.099	run-1 — failed run-2 — failed run-3 — failed
haiku-low
none	0.77 ±0.03	—	0%	$0.42	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.79 ±0.13	+0.02	67%	$0.43	1.02×	1.944	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.90 ±0.06	+0.12	33%	$0.48	1.13×	2.288	not built run-1 — failed run-2 — failed run-3 — passed
design-tokens	0.77 ±0.05	−0.00	0%	$0.53	1.24×	-0.019	run-1 — failed run-2 — failed run-3 — failed

832-client-behaviour

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.77 ±0.03	—	0%	$0.06	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.94 ±0.02	+0.18	67%	$0.13	2.14×	2.446	run-1 — passed run-2 — failed run-3 — passed
component-builder	0.99 ±0.02	+0.22	100%	$0.07	1.15×	24.089	run-1 — passed run-2 — passed run-3 — passed
contracts	0.97 ±0.02	+0.20	100%	$0.18	2.82×	1.787	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.88 ±0.02	+0.11	0%	$0.12	1.90×	1.909	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.63 ±0.03	—	0%	$0.15	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.02	+0.35	67%	$0.14	0.90×	—	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.96 ±0.01	+0.33	100%	$0.12	0.79×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.66 ±0.01	+0.03	0%	$0.20	1.29×	0.678	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.60 ±0.01	—	0%	$0.62	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.94 ±0.06	+0.34	67%	$0.57	0.91×	—	run-1 — passed not built run-2 — failed run-3 — passed
component-builder	0.93 ±0.07	+0.33	67%	$0.57	0.91×	—	run-1 — passed run-2 — passed not built run-3 — failed
design-tokens	0.65 ±0.02	+0.05	0%	$0.55	0.88×	—	run-1 — failed run-2 — failed run-3 — failed
haiku-low
none	0.61 ±0.09	—	0%	$0.42	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.91 ±0.07	+0.30	67%	$0.50	1.21×	3.473	run-1 — failed run-2 — passed run-3 — passed
component-builder	0.98 ±0.01	+0.37	67%	$0.58	1.38×	2.336	run-1 — passed run-2 — passed run-3 — failed
design-tokens	0.68 ±0.02	+0.07	0%	$0.56	1.33×	0.494	run-1 — failed run-2 — failed run-3 — failed
sonnet-high
none	0.60 ±0.01	—	0%	$1.24	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.97 ±0.02	+0.37	100%	$2.26	1.81×	0.361	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.98 ±0.00	+0.38	100%	$1.81	1.46×	0.669	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.67 ±0.05	+0.07	0%	$2.04	1.64×	0.086	run-1 — failed run-2 — failed run-3 — failed
other
spike-novita-both	0.98 ±0.00	—	0%	$0.13	—	—	not built run-1 — failed

836-behaviour-bugfix

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.97 ±0.00	—	100%	$0.01	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.97 ±0.00	+0.00	100%	$0.03	2.04×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.00	+0.00	100%	$0.02	1.19×	0.000	run-1 — passed run-2 — passed run-3 — passed
contracts	0.97 ±0.00	+0.00	100%	$0.04	2.48×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.00	100%	$0.02	1.09×	0.000	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.97 ±0.00	—	100%	$0.02	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.96 ±0.01	−0.01	33%	$0.05	2.63×	-0.370	run-1 — failed run-2 — passed run-3 — failed
component-builder	0.96 ±0.01	−0.01	67%	$0.03	1.70×	-0.431	run-1 — failed run-2 — passed run-3 — passed
design-tokens	0.96 ±0.01	−0.01	33%	$0.03	1.65×	-0.921	run-1 — passed run-2 — failed run-3 — failed
haiku-high
none	0.96 ±0.01	—	100%	$0.05	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.97 ±0.00	+0.01	100%	$0.10	2.11×	0.141	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.97 ±0.00	+0.01	100%	$0.05	1.06×	2.433	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.01	100%	$0.05	0.96×	—	run-1 — passed run-2 — passed run-3 — passed
haiku-low
none	0.96 ±0.01	—	67%	$0.06	—	—	run-1 — failed run-2 — passed run-3 — passed
both	0.97 ±0.00	+0.01	67%	$0.09	1.57×	0.164	run-1 — passed run-2 — passed run-3 — failed
component-builder	0.97 ±0.00	+0.01	100%	$0.06	1.08×	1.207	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.97 ±0.00	+0.01	100%	$0.06	1.01×	9.153	run-1 — passed run-2 — passed run-3 — passed

840-reuse-over-native

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.86 ±0.01	—	0%	$0.08	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.01	+0.12	0%	$0.23	2.90×	0.818	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.98 ±0.01	+0.12	100%	$0.08	0.97×	—	run-1 — passed run-2 — passed run-3 — passed
contracts	0.98 ±0.02	+0.12	0%	$0.28	3.64×	0.562	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.89 ±0.02	+0.03	0%	$0.10	1.27×	1.327	run-1 — failed run-2 — failed run-3 — failed
glm-default
none	0.73 ±0.08	—	0%	$0.12	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.98 ±0.01	+0.25	0%	$0.18	1.50×	4.121	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.98 ±0.01	+0.24	67%	$0.14	1.15×	13.517	run-1 — failed run-2 — passed run-3 — passed
design-tokens	0.74 ±0.08	+0.00	0%	$0.35	2.90×	0.021	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.68 ±0.01	—	0%	$0.48	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.96 ±0.01	+0.28	0%	$0.51	1.07×	8.584	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.91 ±0.06	+0.23	0%	$0.45	0.93×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens run invalid	0.74 ±0.02	+0.07	0%	$0.48	0.99×	—	run-1 — failed run-2 — failed run-3 — failed
haiku-low
none	0.59 ±0.13	—	0%	$0.44	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.86 ±0.08	+0.27	0%	$0.49	1.10×	5.872	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.91 ±0.07	+0.32	0%	$0.48	1.08×	9.021	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.67 ±0.12	+0.07	0%	$0.50	1.12×	1.410	run-1 — failed run-2 — failed run-3 — failed
sonnet-high
none	0.66 ±0.02	—	0%	$1.32	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.95 ±0.03	+0.29	0%	$3.10	2.35×	0.165	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.97 ±0.01	+0.31	67%	$1.85	1.40×	0.588	run-1 — failed run-2 — passed run-3 — passed
design-tokens	0.73 ±0.06	+0.07	0%	$2.39	1.81×	0.069	run-1 — failed run-2 — failed run-3 — failed
other
spike-novita-both	0.97 ±0.00	—	100%	$0.11	—	—	not built run-1 — passed

842-reuse-edit

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.89 ±0.00	—	100%	$0.02	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.89 ±0.00	+0.00	100%	$0.05	2.44×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.89 ±0.00	+0.00	100%	$0.02	1.11×	0.000	run-1 — passed run-2 — passed run-3 — passed
contracts	0.89 ±0.00	+0.00	100%	$0.06	3.32×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.89 ±0.00	+0.00	100%	$0.03	1.35×	0.000	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.89 ±0.00	—	100%	$0.05	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.89 ±0.00	+0.00	100%	$0.10	2.02×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.89 ±0.00	+0.00	33%	$0.05	1.06×	0.000	run-1 — failed run-2 — failed run-3 — passed
design-tokens	0.89 ±0.00	+0.00	100%	$0.06	1.20×	0.000	run-1 — passed run-2 — passed run-3 — passed
haiku-high
none	0.60 ±0.17	—	0%	$0.06	—	—	not built run-1 — failed not built run-2 — failed run-3 — failed
both	0.59 ±0.21	−0.01	0%	$0.09	1.41×	-0.479	run-1 — failed not built run-2 — failed not built run-3 — failed
component-builder	0.73 ±0.23	+0.13	0%	$0.12	1.93×	2.116	run-1 — failed run-2 — failed not built run-3 — failed
design-tokens	0.42 ±0.03	−0.17	0%	$0.09	1.44×	-6.150	not built run-1 — failed not built run-2 — failed not built run-3 — failed
haiku-low
none	0.71 ±0.22	—	33%	$0.11	—	—	run-1 — failed run-2 — passed not built run-3 — failed
both	0.85 ±0.05	+0.14	33%	$0.16	1.40×	3.139	run-1 — failed run-2 — failed run-3 — passed
component-builder	0.89 ±0.00	+0.18	100%	$0.19	1.68×	2.324	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.89 ±0.00	+0.18	67%	$0.18	1.60×	2.619	run-1 — passed run-2 — failed run-3 — passed

850-focus-return

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	1.00 ±0.00	—	0%	$0.03	—	—	run-1 — failed run-2 — failed run-3 — failed
both	1.00 ±0.00	+0.00	33%	$0.05	2.07×	0.000	run-1 — passed run-2 — failed run-3 — failed
component-builder	1.00 ±0.00	+0.00	33%	$0.04	1.50×	0.000	run-1 — failed run-2 — failed run-3 — passed
contracts	1.00 ±0.00	+0.00	0%	$0.04	1.58×	0.000	run-1 — failed run-2 — failed run-3 — failed
design-tokens	1.00 ±0.00	+0.00	33%	$0.03	1.11×	0.000	run-1 — failed run-2 — passed run-3 — failed
glm-default
none	0.99 ±0.01	—	0%	$0.04	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.99 ±0.01	−0.01	0%	$0.04	0.87×	—	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.99 ±0.01	+0.00	0%	$0.04	0.86×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.99 ±0.01	−0.01	0%	$0.03	0.71×	—	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.90 ±0.00	—	0%	$0.05	—	—	run-1 — failed run-2 — failed run-3 — failed
both	1.00 ±0.00	+0.10	0%	$0.07	1.37×	5.360	run-1 — failed run-2 — failed run-3 — failed
component-builder	1.00 ±0.00	+0.10	0%	$0.05	0.95×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	1.00 ±0.00	+0.10	0%	$0.06	1.23×	8.659	run-1 — failed run-2 — failed run-3 — failed
haiku-low
none	1.00 ±0.00	—	0%	$0.08	—	—	run-1 — failed run-2 — failed run-3 — failed
both	1.00 ±0.00	+0.00	0%	$0.05	0.55×	—	run-1 — failed run-2 — failed run-3 — failed
component-builder	1.00 ±0.00	+0.00	0%	$0.05	0.56×	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	1.00 ±0.00	+0.00	0%	$0.04	0.50×	—	run-1 — failed run-2 — failed run-3 — failed

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.82 ±0.04	—	100%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.79 ±0.00	−0.03	100%	$0.07	2.42×	-0.655	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.79 ±0.00	−0.03	100%	$0.03	1.26×	-3.535	run-1 — passed run-2 — passed run-3 — passed
contracts	0.79 ±0.00	−0.03	67%	$0.06	2.22×	-0.827	run-1 — failed run-2 — passed run-3 — passed
design-tokens	0.81 ±0.04	−0.00	67%	$0.04	1.41×	-0.193	run-1 — failed run-2 — passed run-3 — passed
glm-default
none	0.84 ±0.04	—	0%	$0.04	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.84 ±0.03	−0.00	33%	$0.10	2.56×	-0.059	run-1 — failed run-2 — failed run-3 — passed
component-builder	0.84 ±0.04	−0.00	33%	$0.06	1.50×	-0.237	run-1 — failed run-2 — failed run-3 — passed
design-tokens	0.86 ±0.00	+0.02	0%	$0.07	1.69×	0.754	run-1 — failed run-2 — failed run-3 — failed
haiku-high
none	0.80 ±0.00	—	100%	$0.05	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.84 ±0.04	+0.04	100%	$0.06	1.15×	5.633	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.82 ±0.04	+0.01	100%	$0.06	1.18×	1.652	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.84 ±0.03	+0.03	67%	$0.06	1.16×	4.460	run-1 — failed run-2 — passed run-3 — passed
haiku-low
none	0.79 ±0.00	—	100%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.84 ±0.04	+0.05	100%	$0.06	1.72×	2.213	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.84 ±0.04	+0.05	100%	$0.05	1.65×	2.426	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.82 ±0.04	+0.03	100%	$0.06	1.76×	1.044	run-1 — passed run-2 — passed run-3 — passed

860-restraint

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.87 ±0.00	—	100%	$0.01	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.89 ±0.02	+0.02	67%	$0.04	2.65×	0.743	run-1 — failed run-2 — passed run-3 — passed
component-builder	0.87 ±0.00	+0.00	100%	$0.02	1.38×	0.000	run-1 — passed run-2 — passed run-3 — passed
contracts	0.87 ±0.00	+0.00	100%	$0.03	2.05×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.89 ±0.02	+0.02	67%	$0.02	1.10×	12.463	run-1 — passed run-2 — passed run-3 — failed
glm-default
none	0.87 ±0.00	—	100%	$0.02	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.92 ±0.00	+0.05	0%	$0.03	2.02×	3.142	run-1 — failed run-2 — failed run-3 — failed
component-builder	0.90 ±0.02	+0.03	33%	$0.02	1.51×	4.189	run-1 — passed run-2 — failed run-3 — failed
design-tokens	0.87 ±0.00	+0.00	100%	$0.02	1.23×	0.000	run-1 — passed run-2 — passed run-3 — passed
haiku-high
none	0.87 ±0.00	—	100%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.87 ±0.00	−0.00	100%	$0.04	1.26×	-0.233	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.87 ±0.00	−0.00	100%	$0.04	1.25×	-0.243	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.87 ±0.00	−0.00	100%	$0.04	1.22×	-0.279	run-1 — passed run-2 — passed run-3 — passed
haiku-low
none	0.87 ±0.00	—	100%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.87 ±0.00	+0.00	100%	$0.04	1.63×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.87 ±0.00	+0.00	100%	$0.03	1.34×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.87 ±0.00	+0.00	100%	$0.05	1.87×	0.000	run-1 — passed run-2 — passed run-3 — passed
sonnet-high
none	0.87 ±0.00	—	100%	$0.09	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.87 ±0.00	+0.00	100%	$0.12	1.35×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.87 ±0.00	+0.00	100%	$0.12	1.28×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.87 ±0.00	+0.00	100%	$0.12	1.27×	0.000	run-1 — passed run-2 — passed run-3 — passed

861-token-restraint

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.95 ±0.00	—	100%	$0.01	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.95 ±0.00	+0.00	100%	$0.02	2.21×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.95 ±0.00	+0.00	100%	$0.01	1.08×	0.000	run-1 — passed run-2 — passed run-3 — passed
contracts	0.95 ±0.00	+0.00	100%	$0.02	2.15×	0.000	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.95 ±0.00	+0.00	100%	$0.02	1.59×	0.000	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.95 ±0.00	—	100%	$0.01	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.95 ±0.00	+0.00	100%	$0.02	1.90×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.95 ±0.00	+0.00	100%	$0.01	0.84×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.95 ±0.00	+0.00	100%	$0.01	0.98×	—	run-1 — passed run-2 — passed run-3 — passed
haiku-high
none	0.94 ±0.00	—	100%	$0.04	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.95 ±0.00	+0.01	100%	$0.04	0.99×	—	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.95 ±0.00	+0.01	100%	$0.05	1.28×	0.793	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.95 ±0.00	+0.01	100%	$0.03	0.79×	—	run-1 — passed run-2 — passed run-3 — passed
haiku-low
none	0.95 ±0.00	—	100%	$0.05	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.95 ±0.00	+0.00	100%	$0.03	0.73×	—	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.95 ±0.00	+0.00	100%	$0.03	0.59×	—	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.95 ±0.00	+0.00	100%	$0.03	0.70×	—	run-1 — passed run-2 — passed run-3 — passed

862-api-freeze

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.85 ±0.03	—	100%	$0.02	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.87 ±0.00	+0.02	100%	$0.03	1.44×	2.514	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.87 ±0.00	+0.02	100%	$0.02	0.81×	—	run-1 — passed run-2 — passed run-3 — passed
contracts	0.87 ±0.00	+0.02	100%	$0.03	1.50×	2.225	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.85 ±0.03	+0.00	100%	$0.02	0.84×	—	run-1 — passed run-2 — passed run-3 — passed
glm-default
none	0.80 ±0.00	—	100%	$0.02	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.80 ±0.00	+0.00	100%	$0.05	2.55×	0.000	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.80 ±0.00	+0.00	67%	$0.04	1.91×	0.000	run-1 — passed run-2 — passed run-3 — failed
design-tokens	0.80 ±0.00	+0.00	67%	$0.02	1.02×	0.000	run-1 — passed run-2 — failed run-3 — passed
haiku-high
none	0.82 ±0.01	—	100%	$0.05	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.80 ±0.00	−0.02	100%	$0.07	1.52×	-0.831	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.80 ±0.00	−0.02	100%	$0.11	2.36×	-0.316	run-1 — passed run-2 — passed run-3 — passed
design-tokens	0.80 ±0.00	−0.02	100%	$0.06	1.27×	-1.590	run-1 — passed run-2 — passed run-3 — passed
haiku-low
none	0.80 ±0.00	—	100%	$0.04	—	—	run-1 — passed run-2 — passed run-3 — passed
both	0.80 ±0.00	+0.00	100%	$0.04	0.93×	—	run-1 — passed run-2 — passed run-3 — passed
component-builder	0.75 ±0.07	−0.05	67%	$0.07	1.75×	-1.679	run-1 — passed run-2 — failed run-3 — passed
design-tokens	0.87 ±0.09	+0.07	100%	$0.04	1.07×	22.140	run-1 — passed run-2 — passed run-3 — passed

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
haiku-paste
none	0.96 ±0.00	—	0%	$0.12	—	—	run-1 — failed run-2 — failed run-3 — failed
design-tokens	0.96 ±0.00	+0.00	100%	$0.16	1.27×	0.117	run-1 — passed run-2 — passed run-3 — passed

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
both run invalid	0.42 ±0.00	—	100%	$0.47	—	—	run-1 — passed run-2 — failed run-3 — passed
contracts	0.53 ±0.03	—	100%	$0.03	—	—	run-1 — passed run-2 — passed run-3 — passed

variant	quality	Δ	pass@1	$/trial	cost ×	quality/extra $	runs
deepseek-default
none	0.87 ±0.01	—	0%	$0.11	—	—	run-1 — failed run-2 — failed run-3 — failed
both	0.99 ±0.01	+0.12	0%	$0.18	1.64×	1.659	run-1 — failed run-2 — failed run-3 — failed
contracts	1.00 ±0.00	+0.13	0%	$0.18	1.63×	1.793	run-1 — failed run-2 — failed run-3 — failed

UI generation eval — current results