Proposed Season 1 protocol
How PCBBench evaluates AI PCB design
The public site provides a dated pilot snapshot, technical guides and a proposed evaluation plan. Comparable Season 1 rankings are not published. Placeholder scores from the interface prototype are excluded from public rankings.
Model and agent tracks
A reference-harness track aims to compare models with the same tools, instructions and budgets. An agent track should evaluate the complete workflow. Human intervention, repair loops and extra tools must be recorded so improvements are attributed to the system that produced them.
Task coverage
These counts and weights are planning targets, not completed evaluations. The pilot uses a smaller public set.
| Suite | Proposed weight | Expected output |
|---|---|---|
| Electronics reasoning | 5% | Numeric answer with units |
| Datasheet grounding | 8% | Values, pin tables and page citations |
| Component selection | 8% | Manufacturer part number(s) |
| Symbols and footprints | 8% | KiCad .kicad_sym + .kicad_mod |
| Schematic from spec | 18% | KiCad schematic |
| Design review | 10% | Structured findings (refdes, net, category) |
| Design changes (ECO) | 5% | Edited schematic + PCB |
| Placement and routing | 18% | Placed and routed .kicad_pcb |
| End-to-end board | 15% | KiCad project, Gerbers, drill, BOM, CPL |
| Bring-up and debug | 5% | Root cause (refdes/net) + fix as an ECO |
Checks before quality scores
Validation begins with the native artifact and task requirements. Relevant gates include file validity, exact component and pin identity, intended connectivity, schematic-board agreement and frozen ERC or DRC settings. Simulation tests modeled behavior under specified conditions. Physical bring-up supplies a separate evidence level.
Compare routing quality after required connectivity and rule gates pass. A missing connection should not be offset by shorter traces.
Aggregation and uncertainty
Publish suite results and task counts before an aggregate. State weights, failure treatment, retry policy and missing-data policy. Repeated attempts on similar designs should not automatically be counted as independent tasks.
Models in the current pilot attempted different subsets. Their partial averages describe the snapshot and do not establish a ranking.
Cost accounting
Record all included attempts. Distinguish actual billed cost from a list-price equivalent calculated from tokens. Preserve pricing date, route and accounting assumptions. Include required review and repair when measuring the cost of a complete workflow.
When nothing is solved, cost per solved task is undefined. Retries and output selection belong inside the stated cost boundary.
Provenance and holdouts
Public examples explain the format. Held-out tasks stay outside public pages, repositories and search indexes. Track related circuit families, upstream provenance and licenses. Protect reference answers and acceptance tests from the candidate workspace.
Canaries can expose some leakage paths but cannot prove that a benchmark is uncontaminated. Public and held-out evidence should remain distinguishable.
Reproduction and source availability
A released public harness repository and complete Season 1 archive are not available through this site yet. We will link reviewed artifacts when ready. The pilot snapshot states its date and caveats.
For your own evaluation, retain inputs, submitted artifacts, exact tool versions, rule settings, run configuration and independent reports. See the reproducibility guide.
Limitations and corrections
Automated checks cover the requirements they encode. A clean report does not establish every electrical, thermal, mechanical or assembly requirement. Valid alternative designs should not fail for differing cosmetically from a reference.
Record task and grader changes, distinguish regrading from fresh attempts, and explain material corrections in the changelog.
Maintainer disclosure
PCBBench is maintained by ProtoFlow, which develops PCB design software. See the editorial policy and related research when interpreting future comparisons.