Proposed Season 1 protocol
AI PCB design review benchmark
Can the system locate real defects without overwhelming engineers with false alarms?
What the task receives
A native project, its requirements, and a controlled set of defective and clean examples.
What counts as evidence
Structured findings with a location, defect class, supporting evidence and expected consequence.
Include clean controls and define how findings match the defect key. Score precision and recall together, and retain disputed findings for engineering review. Treat an exhaustive checklist as a method of investigation rather than evidence that a particular board has every listed defect.
Failure cases the evaluation should catch
- Returning generic advice without identifying the actual defect.
- Flagging correct circuits to inflate the number of findings.
- Reporting a design preference as a confirmed requirement violation.
Illustrative task brief
Review this schematic before fab. List every defect that would stop the board from working or fail certification.
This example describes the intended task format. It is not a published model result, and the exact task acceptance contract must accompany its frozen inputs.
Metrics to report
- Pooled defect F1
- False alarms on clean designs
Limits of this evidence
A benchmark review score cannot certify a new design; the specific findings still need verification.
Read the proposed methodology and dated pilot snapshot before interpreting any result. Partial pilot averages cover different task mixes and do not establish a model ordering.