Proposed Season 1 protocol
PCB engineering change benchmark
Can the system make a requested edit while preserving unrelated design intent?
What the task receives
A baseline schematic and board, a bounded change request, and an explicit allowed-edit policy.
What counts as evidence
A revised native project with the requested electrical change, correct board synchronization and no unexplained regressions.
Compare before and after using electrical structure and relevant mechanical state. Save baseline findings, then apply regression checks to the changed artifact. Minimal edits are useful evidence, but do not penalize necessary rerouting that follows from a legitimate component change.
Failure cases the evaluation should catch
- Changing the schematic without updating footprint mapping.
- Rewiring unrelated nets while completing the requested edit.
- Relaxing project rules to hide new physical violations.
Illustrative task brief
The AP2112K-3.3 on this board is going EOL. Replace it with an in-stock alternative and update the layout.
This example describes the intended task format. It is not a published model result, and the exact task acceptance contract must accompany its frozen inputs.
Metrics to report
- Change verified
- Collateral edits
- ERC/DRC regressions
Limits of this evidence
A passing edit applies to the specified contract and does not automatically validate unrelated portions of the original design.
Read the proposed methodology and dated pilot snapshot before interpreting any result. Partial pilot averages cover different task mixes and do not establish a model ordering.