Proposed Season 1 protocol
AI schematic generation benchmark
Can the system implement a written electrical brief in a valid native schematic?
What the task receives
A functional brief, supply and load conditions, allowed parts or libraries, and explicit output requirements.
What counts as evidence
Native schematic files that satisfy required roles, pin-level connections, device requirements and applicable behavioral checks.
Apply structural checks before behavioral tests and report the failed layer. Allow alternative topologies that satisfy the brief. Keep the acceptance contract independent of the candidate and avoid requiring a cosmetic match to one reference schematic.
Failure cases the evaluation should catch
- Producing a clean drawing with a missing protection or programming function.
- Passing ERC with an incorrect physical pin mapping.
- Satisfying a nominal simulation while missing an operating corner.
Illustrative task brief
USB-C sink (5 V only), single-cell LiPo charger at 500 mA with status LED, 3.3 V / 300 mA LDO, JST-PH battery connector, reverse-polarity protection.
This example describes the intended task format. It is not a published model result, and the exact task acceptance contract must accompany its frozen inputs.
Metrics to report
- Layered pass rate
- ERC errors
- Datasheet rules met
- SPICE checks at corners
Limits of this evidence
Schematic success does not establish layout, manufacturing readiness or physical bring-up.
Read the proposed methodology and dated pilot snapshot before interpreting any result. Partial pilot averages cover different task mixes and do not establish a model ordering.