Evaluation

Cost per task vs cost per solved PCB task

PCBBench editorial · Published October 14, 2026 · Editorial policy

Cost per task measures spending across attempts. Cost per solved task measures that spending relative to accepted outcomes. The second metric exposes cheap attempts that rarely produce usable results, provided the evaluation uses a consistent task and retry policy.

Use transparent denominators

For a fixed evaluated set, divide total included cost by attempted tasks for one metric and solved tasks for the other. State how multiple attempts, interruptions and partially completed tasks are counted. Do not silently switch between task-level and attempt-level denominators.

Keep the acceptance bar fixed

A solved task should meet the same requirements for every entrant. Otherwise a low-cost model can appear efficient merely because incomplete artifacts are accepted. Preserve hard gates and show partial progress separately.

Account for deployment reality

Human review, repair time and external tool charges can change the economics. Keep those values visible alongside inference cost. When no task is solved, report zero completions and an undefined cost-per-solved value rather than inventing a finite success cost.

Practical checklist

Publish total included cost.
Show attempted and solved counts.
State retry and interruption policy.
Use identical acceptance criteria.
Keep human effort visible.

A concrete example

Two systems spend $10 and $20 across ten tasks and solve one and five tasks respectively.

Evidence to accept: The report shows $10 versus $4 per solved task under that stated example.
Evidence that is insufficient: The first system is declared cheaper solely because its total spend is lower.

Common questions

Is total cost divided by pass rate always valid?

Only when the cost and pass-rate denominators align with the same task and attempt policy. Publish the underlying counts.

Does this include fabrication cost?

Only if the cost boundary explicitly includes fabrication. Keep inference and manufacturing expenses distinct.

Primary references and further reading

Use the matching device documentation and project configuration when applying this checklist. These references explain the underlying tools and evaluation approaches.

How this fits PCBBench

The End-to-end AI PCB design benchmark describes the proposed inputs, outputs and checks for this capability. The pilot snapshot contains partial measured attempts; it is not a finished ranking.