Proposed Season 1 protocol
AI component selection benchmark
Can the system choose a real component that satisfies every mandatory constraint?
What the task receives
A requirement table and a dated catalog snapshot with unambiguous part identifiers and procurement conditions.
What counts as evidence
An exact orderable part number with verifiable electrical, package and snapshot procurement properties.
Check hard constraints before optimizing price or preferences. Preserve the catalog snapshot and quantity assumptions. Distinguish a missing-data outcome from a confirmed violation, and permit multiple feasible answers where the task does not require one unique part.
Failure cases the evaluation should catch
- Inventing a plausible but nonexistent part suffix.
- Meeting a current rating while missing dropout or temperature requirements.
- Reporting snapshot stock as a live availability guarantee.
Illustrative task brief
3.0 V rail from a 1S LiPo (3.4 to 4.2 V), 150 mA peak, Iq < 5 µA, SOT-23-5, in stock at LCSC, under $0.40 at 1k.
This example describes the intended task format. It is not a published model result, and the exact task acceptance contract must accompany its frozen inputs.
Metrics to report
- Constraint pass rate
- Hallucinated MPN rate
- Price vs cheapest feasible
Limits of this evidence
Catalog feasibility does not establish a drop-in substitution or a complete circuit design.
Read the proposed methodology and dated pilot snapshot before interpreting any result. Partial pilot averages cover different task mixes and do not establish a model ordering.