Proposed Season 1 protocol
Datasheet interpretation benchmark
Can the system find the correct requirement in the exact device documentation?
What the task receives
A frozen datasheet or manufacturer artifact and a question that requires its package tables, operating conditions or plots.
What counts as evidence
An answer tied to the relevant page, table or figure, including conditions and device variant.
Version the source document and keep citations machine-checkable where possible. Include values that differ across package variants and operating conditions. Grade the answer and the supporting evidence separately so a lucky guess cannot hide a failed document lookup.
Failure cases the evaluation should catch
- Citing a page that does not support the answer.
- Confusing absolute limits with recommended operation.
- Using a familiar family pinout instead of the supplied package table.
Illustrative task brief
From the attached Murata curves, which of these six 0805 capacitors still gives ≥ 10 µF effective at 4.5 V DC bias? Cite the page.
This example describes the intended task format. It is not a published model result, and the exact task acceptance contract must accompany its frozen inputs.
Metrics to report
- Value accuracy
- Pin-table F1
- Citation validity
Limits of this evidence
A supported answer establishes one document interpretation, not complete circuit suitability.
Read the proposed methodology and dated pilot snapshot before interpreting any result. Partial pilot averages cover different task mixes and do not establish a model ordering.