Proposed Season 1 protocol
PCB bring-up and debug benchmark
Can the system use observations to distinguish likely causes and propose a discriminating next test?
What the task receives
A design, symptoms, permitted measurements and a controlled fault or verified diagnosis.
What counts as evidence
A diagnosis supported by the observations and a test plan that meaningfully distinguishes competing explanations.
Reveal measurements according to a fixed interaction policy and record each requested test. Grade diagnosis and evidence gathering separately. If physical hardware is involved, keep the fixture, firmware and assembly state traceable so the fault does not shift between entrants.
Failure cases the evaluation should catch
- Guessing a component failure without connecting it to measurements.
- Repeating generic advice that cannot distinguish hypotheses.
- Treating missing observations as confirmed facts.
Illustrative task brief
The 3V3 rail sits at 2.1 V and U4 is hot to the touch. Scope capture and rail measurements attached. Find the cause and fix it.
This example describes the intended task format. It is not a published model result, and the exact task acceptance contract must accompany its frozen inputs.
Metrics to report
- Top-1 root cause
- Fix verified
Limits of this evidence
A successful diagnosis on one fault does not establish unrestricted ability to repair or safely operate other hardware.
Read the proposed methodology and dated pilot snapshot before interpreting any result. Partial pilot averages cover different task mixes and do not establish a model ordering.