Season 0 pilot · public tasks · 1 attempt
Pilot results snapshot
The pilot protocol runs each model once through the 120 public tasks to shake out broken tasks, graders and transports. 1 of 13 models were recorded as complete in this snapshot. The numbers below are real, generated from the run records (2026-10-03 21:25 UTC), but they are not a leaderboard: models have finished different suites, so their averages are not comparable yet.
Recorded progress by model
| Model | Status | Public tasks done | Passed | Mean | Cost so far | Out tok |
|---|---|---|---|---|---|---|
GPT-6.1 Sol via Codex CLI · effort max | Complete Full public pass finished. | 120/120 | 88/120 | 0.81 | $30.00 | 9k |
GPT-6 Astra via Codex CLI · effort max | Running Public pass in progress. | 104/120 2 interrupted | 87/104 | 0.90 | $67.14 | 5k |
Claude Sonnet 5.5 via Claude Code · effort max | Paused Provider usage limit; resumes after the weekly reset. | 77/120 1 interrupted | 65/77 | 0.89 | $85.43 | 53k |
Gemini 3.8 Flash via Cursor CLI · effort high | Paused Provider usage limit until Oct 26; a submit-format issue is being fixed. | 20/120 7 to rerun (task changed) · 81 interrupted | 11/20 | 0.56 | $30.27 | 72k |
Grok 4.7 via Grok CLI · effort xhigh | Paused Provider usage limit. | 9/120 4 interrupted | 8/9 | 0.89 | $1.91 | 15k |
Claude Opus 5.5 via Claude Code | Queued Next on Claude Code after Sonnet 5.5. | 0/120 | – | – | – | – |
Gemini 3.1 Pro via Cursor CLI | Queued Provider usage limit until Oct 26. | 0/120 | – | – | – | – |
GPT-5.3 Codex via Cursor CLI | Queued Provider usage limit until Oct 26. | 0/120 | – | – | – | – |
Muse Spark 1.3 via Cursor CLI | Queued Provider usage limit until Oct 26. | 0/120 | – | – | – | – |
Kimi K2.7 Code via Cursor CLI | Queued Provider usage limit until Oct 26. | 0/120 | – | – | – | – |
Claude Haiku 4.5 via Claude Code | Queued After Opus 5.5. | 0/120 | – | – | – | – |
GPT-6 Luna via Codex CLI | Queued Starts after GPT-6 Astra. | 0/120 | – | – | – | – |
Claude Fable 5.1 via Claude Code | Not scheduled Not scheduled for the pilot. | 0/120 | – | – | – | – |
Interrupted attempts (network outages, provider usage limits) never count as failures; they are rerun. Attempts on a task that changed after the model saw it are rerun too. Cost is the list-price equivalent of the tokens each CLI reported.
Pilot results by suite
Tasks passed out of valid attempts, shaded by mean score. Empty cells have not run yet.
| Model | Reason 24 public | Datasheet 12 public | Select 12 public | Library 12 public | Schematic 12 public | Review 16 public | ECO 8 public | Layout 12 public | Board 6 public | Debug 6 public |
|---|---|---|---|---|---|---|---|---|---|---|
| GPT-6.1 Sol | 23/24 | 11/12 | 12/12 | 12/12 | 12/12 | 3/16 | 7/8 | 2/12 | 1/6 | 5/6 |
| GPT-6 Astra | 23/24 | 11/12 | 12/12 | 12/12 | 12/12 | 4/16 | 8/8 | – | 0/2 | 5/6 |
| Claude Sonnet 5.5 | 18/24 | 12/12 | 12/12 | 12/12 | – | 11/16 | 0/1 | – | – | – |
| Gemini 3.8 Flash | 11/11 | 0/9 | – | – | – | – | – | – | – | – |
| Grok 4.7 | 8/9 | – | – | – | – | – | – | – | – | – |
- Gemini 3.8 Flash's 0/9 on datasheets is a transport failure: its CLI could not submit nested answers. Those attempts will be rerun once the submit format is fixed.
- GPT-6.1 Sol's layout and board results include attempts zeroed by a layout integrity false positive and by prompts that did not say which existing DRC findings are ignored; both are being fixed and rerun.
What the pilot has shown so far
- Board-level work is where models fail
- GPT-6.1 Sol, the first model through every public task, passed 23 of 24 reasoning tasks and all 12 schematics, but only 1 of 6 end-to-end boards and 2 of 12 placement-and-routing tasks.
- Two suites are close to saturated
- Every model passed every component-selection task it attempted, and most reasoning tasks fall in one calculation. Season 1 makes both harder (constraints the catalog does not index, datasheet-only parameters) or weights them down.
- The pilot found real task bugs, so it did its job
- Five task or grader problems mis-scored correct work: a unit written as mA/LSB that the parser rejected, a solder jumper dropped from a netlist, layout prompts that did not say which pre-existing DRC findings are ignored, a review prompt that invited findings the key did not count, and a BOM check that failed equivalent part numbers. Each is being fixed and the affected attempts rerun before anything is ranked.
- Isolation held under pressure
- One model spent most of its tokens searching the sandbox for the grader. No grader file, reference answer or hidden marker ever appeared in any trace.
- The same task can cost 15× more on one model
- List-price equivalent of the measured tokens: a footprint task cost $2.17 on Claude Sonnet 5.5 at max effort and $0.15 on GPT-6.1 Sol; a design review $2.68 against $0.22, though Sonnet scored higher on review (0.87 vs 0.58). Averaged over the public tasks each model finished: $1.07 against $0.25.
What happens next
- Fix the task and grader bugs the pilot found, freeze every task by content hash, and rerun the affected attempts.
- Finish one public pass for every model, then add the second and third attempts.
- Run the held-out tasks only through providers that do not retain or train on the data.
- Replace the preview leaderboard with measured results and log it in the changelog.
Models are reached through their vendors' agent CLIs with PCBBench's prompt and tools; see methodology. Claude Fable 5.1 is not part of the pilot.