Evaluation

Pass rate, pass@k and reliability in PCB benchmarks

PCBBench editorial · Published October 14, 2026 · Editorial policy

First-attempt pass rate, success within k attempts and success on every repeat measure different behavior. State the retry policy and denominator before comparing them, particularly when a PCB failure carries review and fabrication consequences.

Start with the task-level event

Define what counts as a pass: all required gates, a partial score threshold or a manual decision. Keep that event consistent across attempts. A model that produces one acceptable board among several failures is different from one that completes every repeat successfully.

Declare selection and feedback policies

Say whether later attempts receive grader feedback or are independent samples. Explain how the best output is selected. Success within k feedback-guided attempts is an agent workflow result; it should not be presented as the same statistic as independent sampling without feedback.

Report uncertainty at the right level

Tasks from one circuit family may have correlated failure modes. Preserve task-level results and avoid treating every repeated run as an unrelated design. Publish sample counts alongside intervals and explain when a task set is too small for a stable ordering.

Practical checklist

Define the pass event.
Publish k and the retry policy.
Record feedback and output selection.
Separate at-least-one and every-repeat success.
Show task counts and uncertainty.

A concrete example

One model passes one of three runs; another passes all three.

Evidence to accept: The report distinguishes eventual success from repeat reliability.
Evidence that is insufficient: Both models are described as equally reliable because each produced an acceptable board once.

Common questions

Is pass@k always better than pass rate?

Neither is universally better. Choose the statistic that matches whether the real workflow permits multiple attempts and reliable selection.

Can partial scores replace a pass rate?

They show progress, but keep hard functional failure visible rather than averaging it away.

Primary references and further reading

Use the matching device documentation and project configuration when applying this checklist. These references explain the underlying tools and evaluation approaches.

How this fits PCBBench

The Electronics reasoning benchmark describes the proposed inputs, outputs and checks for this capability. The pilot snapshot contains partial measured attempts; it is not a finished ranking.