Proposed Season 1 protocol
End-to-end AI PCB design benchmark
Can the system turn a product brief into a coherent native project and manufacturing package?
What the task receives
A product-level brief with interfaces, power, mechanical bounds, assembly assumptions and deliverables.
What counts as evidence
Schematic and board consistency, applicable functional and design checks, and a reconciled fabrication and assembly package.
Preserve intermediate and final artifacts, then apply all required schematic and layout gates. Inspect exported manufacturing files and assembly data independently. When boards are physically built, report board revision, firmware, substitutions and rework alongside observations.
Failure cases the evaluation should catch
- Submitting Gerbers that do not match the checked native board.
- Meeting routing rules while omitting a required product function.
- Claiming physical success from file-level checks alone.
Illustrative task brief
USB-C powered, 2-layer, ≤ 50×50 mm: ESP32-C3, BME280, status LED, two buttons, JLCPCB assembly-ready, BOM under $8 at qty 100.
This example describes the intended task format. It is not a published model result, and the exact task acceptance contract must accompany its frozen inputs.
Metrics to report
- Boards passing every gate
- BOM cost
- Fab-ready rate
- First-power-up pass (fab track)
Limits of this evidence
Fabrication readiness and successful physical operation are different evidence levels and should be labeled separately.
Read the proposed methodology and dated pilot snapshot before interpreting any result. Partial pilot averages cover different task mixes and do not establish a model ordering.