Task suites

The proposed Season 1 design covers 300 tasks across 10 suites. These are planning targets, not completed-run counts. The dated pilot uses a smaller public set; see its coverage and caveats on the pilot page. The pages below describe intended task formats and checks.

Electronics reasoning

Planned: 60 tasks · 5% of proposed indexEvaluation protocol

Closed-form calculations a board designer does every day: feedback dividers, LDO dissipation, IPC-2152 trace width, filter corners, crystal load capacitance, impedance, thermal rise. Variant-grouping (from CIRCUIT) means a lucky guess or a memorized answer does not score.

Example task · Buck feedback divider

TPS62130 (adjustable), VFB = 0.8 V. Pick E24 resistors for a 3.3 V output with R2 ≤ 100 kΩ. Report R1, R2 and the resulting Vout.

Passes when: Vout within ±1% of 3.3 V, both values on E24, R2 ≤ 100 kΩ, on all four variants (1.8 V, 2.5 V, 3.3 V, 5.0 V).

Model produces: Numeric answer with units
Graded by: Numeric match within per-question tolerance, units parsed. Each question ships as 4 parametric variants and only counts if all 4 pass
Reported metrics: Variant-group accuracy · Single-question accuracy
Builds on: CIRCUIT (unit-test grouping) · CircuChain (trap variants)

Datasheet grounding

Planned: 30 tasks · 8% of proposed indexEvaluation protocol

Read the real PDF, not a pre-digested JSON. Absolute-maximum versus recommended ratings, pin-mux tables, EN thresholds, MLCC DC-bias derating curves. Trap variants use parts that break the usual defaults, like an active-low enable or a nonstandard I²C address strap.

Example task · Usable capacitance under bias

From the attached Murata curves, which of these six 0805 capacitors still gives ≥ 10 µF effective at 4.5 V DC bias? Cite the page.

Passes when: Exactly the two qualifying parts, with a citation to the page holding the DC-bias curve.

Model produces: Values, pin tables and page citations
Graded by: Numeric tolerance vs curated key; pin-table F1 over (pin, name, function) triples; cited page must contain the value
Reported metrics: Value accuracy · Pin-table F1 · Citation validity
Builds on: HWE-Bench · PCB-QA · EEBench (DC-bias derating)

Component selection

Planned: 30 tasks · 8% of proposed indexEvaluation protocol

Choose real, purchasable parts that meet every constraint: electrical ratings, package, temperature, lifecycle, JLCPCB basic-part preference and a cost ceiling. Part numbers that are not in the catalog score zero and are counted as hallucinations.

Example task · LDO for a battery sensor node

3.0 V rail from a 1S LiPo (3.4 to 4.2 V), 150 mA peak, Iq < 5 µA, SOT-23-5, in stock at LCSC, under $0.40 at 1k.

Passes when: MPN exists, dropout at 150 mA keeps regulation at 3.4 V, and Iq, package, stock and price all hold. Score scales with price vs the cheapest feasible part.

Model produces: Manufacturer part number(s)
Graded by: Parametric check against a frozen, versioned catalog snapshot: ratings, package, lifecycle, stock and price at snapshot date
Reported metrics: Constraint pass rate · Hallucinated MPN rate · Price vs cheapest feasible
Builds on: EEBench (cost in score) · JITX 2024 tests

Symbols and footprints

Planned: 30 tasks · 8% of proposed indexEvaluation protocol

Create a symbol and land pattern from a datasheet drawing. Footprint mistakes are the classic first-spin killer, and footprint geometry from drawings is a measured gap: general models scored under 20% pad IoU on ICGeo8K.

Example task · QFN-24 with exposed pad

From the attached datasheet, create the KiCad symbol and footprint for the 4×4 mm QFN-24 package, including the exposed pad.

Passes when: All 25 pads present, pad IoU ≥ 0.9 against the IPC reference, EP paste reduced, pin numbers and electrical types match the datasheet.

Model produces: KiCad .kicad_sym + .kicad_mod
Graded by: Exact pin-number map; pad IoU vs IPC-7351B nominal land pattern; pitch, drill and courtyard tolerance. KiCad's stock libraries are hidden for these tasks, so a footprint cannot be copied
Reported metrics: Pin-map accuracy · Pad IoU · KLC violations
Builds on: ICGeo8K / LLM4-IC8K · SFgen

Schematic from spec

Planned: 30 tasks · 18% of proposed indexEvaluation protocol

Turn a written requirement into a complete schematic with real parts: power tree, decoupling, protection, programming. Graded on electrical intent, not resemblance to one reference drawing, so any topology that meets the spec can pass. Analog and power blocks are simulated at component-tolerance corners with MLCC derating.

Example task · USB-C LiPo charger + 3.3 V rail

USB-C sink (5 V only), single-cell LiPo charger at 500 mA with status LED, 3.3 V / 300 mA LDO, JST-PH battery connector, reverse-polarity protection.

Passes when: ERC clean, 5.1 kΩ Rd on each CC pin, PROG resistor sets 500 mA ±10%, LDO stable with the chosen caps at corners, every IC has datasheet-required decoupling.

Model produces: KiCad schematic
Graded by: Five layers, each reported: kicad-cli ERC (frozen config) → required roles → datasheet rule library → topology signatures (VF2) → ngspice at worst-case corners
Reported metrics: Layered pass rate · ERC errors · Datasheet rules met · SPICE checks at corners
Builds on: PCBSchemaGen verifier · HWE-Bench · EEBench · AnalogCoder

Design review

Planned: 40 tasks · 10% of proposed indexEvaluation protocol

Find the bugs in a working-looking schematic: swapped UART lines, a missing pull-up, an LDO in dropout, a SOT-23 pinout variant, a cap rated below its rail, a strapping pin left floating. One in five designs is clean, so crying wolf costs points.

Example task · ESP32-S3 USB-C dev board

Review this schematic before fab. List every defect that would stop the board from working or fail certification.

Passes when: Finds the single shared CC resistor (works in one cable orientation only) and the missing EN RC delay; raises no issue on the correct ESD array.

Model produces: Structured findings (refdes, net, category)
Graded by: One F1 score pooled over every finding in the suite, matched to seeded defects by class and location. Silence and shotgun answers both score near zero; false alarms on clean designs cost precision
Reported metrics: Pooled defect F1 · False alarms on clean designs
Builds on: SWE-bench (real defects) · Traceformer-style review

Design changes (ECO)

Planned: 20 tasks · 5% of proposed indexEvaluation protocol

Real engineering is mostly edits. Swap a discontinued regulator, add reverse-polarity protection, move a sensor to another I²C bus. The grader checks that the change landed and that nothing else moved.

Example task · Swap an EOL regulator

The AP2112K-3.3 on this board is going EOL. Replace it with an in-stock alternative and update the layout.

Passes when: New part meets the old one's specs, layout updated with zero new DRC errors, no unrelated nets changed.

Model produces: Edited schematic + PCB
Graded by: Canonical netlist diff vs golden post-change netlist; minimal-diff penalty; ERC/DRC must not regress; schematic↔PCB parity
Reported metrics: Change verified · Collateral edits · ERC/DRC regressions
Builds on: Cua-Bench KiCad edits · SWE-bench Verified

Placement and routing

Planned: 30 tasks · 18% of proposed indexEvaluation protocol

Place and route a manufacturable board from a netlist, outline and stackup, across a scale ladder from 20 to 400+ parts. A 99%-routed board does not work, so gates come first. Then physics rule checks DRC cannot see: decoupling distance, switcher hot-loop area, return-path continuity under high-speed nets, diff-pair impedance, IPC-2152 current capacity.

Example task · 2-layer STM32 sensor board

Place and route on the given 40×30 mm outline, 2 layers, JLCPCB 6/6 mil rules. Crystal within 5 mm of the MCU, decoupling within 2 mm of each supply pin, USB pair at 90 Ω.

Passes when: 100% connectivity, zero DRC errors with the frozen rule deck, placement rules met; partial credit for vias, wirelength and loop area vs the reference.

Model produces: Placed and routed .kicad_pcb
Graded by: Hard gates (connectivity, zero DRC at fab rules, rule-deck hash unchanged) then placement and physics checks normalized to a human reference
Reported metrics: Clean pass · Routed % · Physics rule checks · Vias and wirelength vs reference
Builds on: PCBWorld (Clean Pass) · Quilter physics rule checks · DAC 2020 routing set · tscircuit SRJ18

End-to-end board

Planned: 15 tasks · 15% of proposed indexEvaluation protocol

One-paragraph product brief to fabrication files. The model makes every decision, from parts to stackup to silkscreen. Each quarter the top submissions on three tasks are fabricated, assembled and bench-tested, and the photos and logs are published.

Example task · BLE environmental sensor

USB-C powered, 2-layer, ≤ 50×50 mm: ESP32-C3, BME280, status LED, two buttons, JLCPCB assembly-ready, BOM under $8 at qty 100.

Passes when: All files export, every gate passes, every part orderable, BOM under budget. Fab track: the board enumerates on USB and reads the sensor on first power-up.

Model produces: KiCad project, Gerbers, drill, BOM, CPL
Graded by: Every Schematic and Layout gate, plus fab-profile DRC, Gerber re-import vs board copper, BOM orderability and cost, CPL rotation sanity, outline vs DXF
Reported metrics: Boards passing every gate · BOM cost · Fab-ready rate · First-power-up pass (fab track)
Builds on: GDPval (real deliverables) · AMS-IO-Bench (taped-out output)

Bring-up and debug

Planned: 15 tasks · 5% of proposed indexEvaluation protocol

Design files plus a symptom and some measurements from a real board with an injected fault. Name the cause and fix it. A list of ten possibilities scores nothing.

Example task · Hot regulator, sagging rail

The 3V3 rail sits at 2.1 V and U4 is hot to the touch. Scope capture and rail measurements attached. Find the cause and fix it.

Passes when: Identifies the reversed tantalum C17 on 3V3, proposes rotating it, and the fixed design passes ERC and the rail simulation.

Model produces: Root cause (refdes/net) + fix as an ECO
Graded by: Root cause matched to the injected fault within top-1; the proposed fix is applied and verified by the ECO grader
Reported metrics: Top-1 root cause · Fix verified
Builds on: METR (task horizons) · Field failure reports

How tasks are made

Acceptance before scoring. Each task needs an explicit contract, a passing reference and tests that reject known wrong outputs.
Variants stay together. Related circuits should be grouped when constructing public and held-out splits.
Clean controls. Review tasks need clean designs as well as defects to expose false alarms.
Provenance. Source designs, licenses and transformations need to be recorded before public release.
Published scope. The current pilot snapshot and the proposed Season 1 task inventory are different datasets.

Want to contribute tasks or donate a board? Get in touch.