Chip-Bench is an open benchmark for AI chip design. It asks AI coding agents to design a working RISC-V CPU in synthesizable SystemVerilog, and then to make that CPU faster without breaking it. A submission only scores if it computes the right answer on every clock cycle and fits inside the chip area it was given. Version 0.3 records 16 runs across 3 levels, collected 1–10 September 2026 by ARA Labs, and every run ships with its RTL and verification evidence.

What is Level 3, the chip-optimization level?

Level 3 hands every agent the same working reference core, the same tests and a frozen harness, then asks it to make the CPU faster. Importing a CPU built outside the run is banned, and correctness cannot be traded for speed — breaking a gate scores zero. Six designs have entered. The highest public score is 5.8416 (cc-fuzz), though one of its proofs never finished; the fastest design that passed every check is cc-fable, at 5.6896 and 41.03 MHz.

What does PPA mean, and how does Chip-Bench score it?

PPA is power, performance and area — the three properties chip designers trade against each other. Chip-Bench scores performance as clock frequency and cycle count after Yosys synthesis and nextpnr place-and-route, and area as logic cells, with a hard budget per level: 10,000 LUTs on Level 1 and 20,000 on Level 2. Area is a pass/fail gate, not a soft penalty, so over the line is a rejection. The trade-off shows up directly in the results: cc-fuzz is 2.7% faster than cc-fable while using 21.5% less chip area, 13,766 logic cells against 17,543.

Is Chip-Bench a test of recursive self-improvement (RSI)?

Not in the strong sense, and it is worth being precise. Level 3 is self-improvement of an artifact: the agent measures its own design, changes it, measures again, and climbs. Recursive self-improvement means a system improving the thing that does the improving, and Chip-Bench does not test that. What it does provide is a hard, cheap-to-check signal for studying optimization loops, because a CPU either computes the right answer on every cycle or it does not. cc-fuzz ran inside a fuzzing-style exploration loop and finished 2.7% faster on 21.5% less area than cc-fable, which runs the same model — but with one run each and unequal prior knowledge, that is not evidence the loop caused the gain.

Which AI models and agents have been benchmarked?

Eight configurations across seventeen CPU runs: Claude Code (Fable 5 on Level 1, Opus 5 on Level 3), cc-fable (Claude Code running Claude Fable 5.1, on Levels 2 and 3), cc-fuzz (the same model inside a fuzzing-style exploration loop, on Level 3), Codex running GPT-5.6 Sol (Levels 1 and 3), Codex running GPT-6 Astra, Grok 4.6, Gemini 3.8 Flash through the agy CLI, and GLM 5.3 through the pi coding agent (Levels 1 and 2).

How is a submission verified?

Every design is checked against an independent architectural interpreter, delayed request and response schedules, directed assembly and compiled C, random differential programs, protocol assertions, Yosys synthesis, nextpnr place-and-route, and bounded formal jobs. cc-fable cleared all 97 directed tests, all 350 random differential programs and the proof. Agent-supplied test harnesses are left alone and reported separately.

What is the Chip-Bench hill-climb suite?

Twenty frozen optimization tasks across six layers of the chip stack: logic and synthesis, microarchitecture, memory system, physical design, verification and formal, and co-design including reliability, security and analog. Each task hands the agent a working design, a baseline number, a frozen expert bar that scores 1.0, and a frozen grader, then asks for one number to get better without breaking a single correctness gate. All twenty tasks have now been graded: ten by five frontier models, the other ten by Gemini 3.8 Flash and Grok 4.6, plus six GLM 5.3 tasks still in progress on the rest of the suite, in seventy-six runs; the score is log(baseline/agent) divided by log(baseline/bar) under hc-0.3.0, uncapped, so matching the author's expert solution scores 1.00 and beating it scores above 1.00.

What is the Chip-Bench network track?

Three PCIe build tasks. n1-pcie-ep is a Gen1-class x1 endpoint controller: flow-control credits, sequence numbers, LCRC, ACK/NAK and retry, configuration space, BAR decode, completer and requester paths and ordering. n2-pcie-ltssm is the physical-layer logical sub-block and link training state machine over a PIPE x4 interface. n3-pcie-switch is a three-port switch fabric core above the data link layer, with a shared-storage ceiling and a cut-through latency bound. All three are graded by an independently written link partner or hidden protocol suite that misbehaves on purpose. In the 2026-09-07 pilot all three agents passed every public case on n1 and n2 and were separated only by the hidden suites: 94, 86 and 86 of 95 on the endpoint, and 110, 103 and 86 of 113 on link training. On n3's 2026-09-08 revision-4 run, the same three-way field scored 100, 94 and 71 of 100 -- the first run on this network track to clear a hidden suite exactly, on cc-opus5's third attempt.

Who grades a Chip-Bench submission — is there an LLM judge?

No model grades anything. Yosys reports logic area, nextpnr places and routes and reports the clock, OpenSTA and OpenROAD close timing, SymbiYosys with yices proves the formal properties, and Verilator or g++ runs the cycle-accurate simulations. Every score on this site is a number one of those tools printed, on a harness the agent is forbidden to modify — the submission is re-graded from a pristine copy of the harness on hidden workloads the agent never saw.

Where are the run traces, the RTL and the data?

Every run is published as an agent-native research artifact — the trajectory, the generated RTL and the evidence, not just the score — on Hugging Face at huggingface.co/datasets/AgentNativeResearchLab/cpu-design-benchmark. Read the known limits before citing any number: results are provisional at v0.2, there is one run per configuration, and every score shown is a public score, not the official hidden-workload result.

Chip-Bench — the open benchmark for AI chip design

Can AI design a chip? Three tracks: build a CPU from a written spec, make a working circuit faster without breaking it, and build a link controller that stays correct while the other side misbehaves. Every run gets the same frozen harness and a few hours. Yosys, nextpnr, OpenSTA, SymbiYosys and hidden compliance suites produce the number — no model grades anything.

20 hill-climb tasks across 6 layers, 3 CPU-design levels, 3 PCIe core tasks. 76 graded hill-climb runs (ten of the tasks so far run by two agents, Gemini 3.8 Flash and Grok 4.6, against the task author's expert bar; GLM 5.3 has six graded tasks so far) and 9 network runs across 6 frontier models. The hill-climb score is log(baseline/agent) / log(baseline/bar) under hc-0.3.0, uncapped: matching the author's expert solution scores 1.00, beating it scores above 1.00. The network tasks are scored as cases passing, public and hidden.

The 20 hill-climb tasks

Logic & synthesis

c1_approx_area — Shrink a 16-bit reciprocal circuit, never off by more than 4 LSB on any of its 65,536 inputs. Scores: cc-fable 1.07, cc-opus 1.14, codex-astra 1.31, grok 0.48, gemini-flash 0.98, glm-5.3 0.80. Bar 338 LUT4-eq.
c1_pipeline_fmax — Cut a slow arithmetic datapath into pipeline stages so the clock can run faster. Scores: cc-fable 1.06, cc-opus 1.04, codex-astra 1.04, grok 1.02, gemini-flash 0.97, glm-5.3 0.89. Bar 125.19 MHz.
c1_recipe_pareto — Write synthesis recipes that beat the default flow on area and logic depth — on circuits you never see. Scores: cc-fable 1.37, cc-opus 1.33, codex-astra 1.50, grok 0.66, gemini-flash 0.54, glm-5.3 0.52. Bar — HV ratio 1.0 by construction.

Microarchitecture

c2_branch_pred — Guess which way each branch goes, using 8192 bits of memory and nothing else. Scores: cc-fable 1.98, cc-opus 1.34, codex-astra 1.94, grok 1.83, gemini-flash 1.19, glm-5.3 1.88. Bar 75.5 MPKB.
c2_cache_ctrl — Design the data-cache controller that spends the fewest cycles waiting for memory. Scores: cc-fable 1.81, cc-opus 1.55, codex-astra 1.58, grok 0.84, gemini-flash 0.71, glm-5.3 1.24. Bar 540194 cycles.
c2_chacha_tput — Build a ChaCha20 core with the best area x time per block on an ECP5 FPGA. Scores: cc-fable 1.39, cc-opus 1.35, codex-astra 1.40, grok 1.30, gemini-flash 1.36. Bar 726 LUT4-eq·µs.
c2_divider_al — Build an RV32M divider that is both small and quick on a hidden mix of operands. Scores: cc-fable 1.75, cc-opus 1.53, codex-astra 1.37, grok 1.30, gemini-flash 1.23. Bar 7972 LUT4-eq·cyc.

Memory system

c3_mem_sched — Order the DRAM request queue so more accesses land on a row that is already open. Scores: cc-fable 2.35, cc-opus 1.69, codex-astra 1.42, grok 1.68, gemini-flash 1.64. Bar 812916 cycles.
c3_prefetcher — Fetch the next address before the program asks for it, in 2 KB of C++ state. Scores: cc-fable 1.41, cc-opus 1.03, codex-astra 1.05, grok 1.15, gemini-flash 0.92. Bar 5.788 cyc/access.
c3_replacement — Choose which cache line to evict, with 4 bits of metadata per line. Scores: cc-fable 1.06, cc-opus 1.04, codex-astra 0.22, grok 1.06, gemini-flash 0.61. Bar 241.17 MPKA.

Physical design

c4_clock_gating — Gate the clocks to cut power, with a proof the circuit still behaves the same. Scores: gemini-flash 1.31, grok 1.84.
c4_flow_tune — Turn the physical-design flow knobs to shrink area x delay on sky130. Scores: gemini-flash 0.94, grok 1.11.
c4_sdc_refine — Prove which timing paths can never happen, and let the clock go faster. Scores: gemini-flash 1.00, grok 1.00 — both matched the author's design.

Verification & formal

c5_induction — Write the invariants a bounded proof needs to close on a credit-based link. Scores: gemini-flash 1.07, grok 1.12 — both closed all 8 properties; grok’s solver was faster, so it scores higher.
c5_mutation_eff — Write the testbench that kills the most injected bugs per simulated kilocycle. Scores: gemini-flash 1.17, grok 1.23.
c5_testset_min — Catch every injected bug, run the fewest cycles: pick the subset of 200 tests. Scores: gemini-flash 0.68, grok 0.35 (grok's subset ran fewer cycles than the record but lost one mutant in two of the three hidden sets, and each set that loses a kill scores zero).

Co-design, reliability, security, analog

c6_const_time — Make modular exponentiation take the same time no matter what the secret is. Scores: gemini-flash 1.00, grok 1.00 — both reached zero measured leak.
c6_custom_isa — Add your own RISC-V instructions, then rewrite the kernels to spend fewer cycles. Scores: gemini-flash 1.31, grok 1.39, glm-5.3 1.64.
c6_opamp_fom — Size a two-stage Miller op-amp for the most gain-bandwidth per microamp, in 60 SPICE runs. Scores: gemini-flash 0.00, grok 0.00 — both beat the record on the figure of merit and both missed the 60° phase-margin spec at the fast-cold corner, which zeroes the score.
c6_seu_harden — Harden the flip-flops that matter against particle strikes, inside an area budget. Scores: gemini-flash 1.32, grok 1.00.

Build a CPU — Levels 1 to 3

Level 1: write a TinyRV-14 core from the spec. GPT-6 Astra passed all 37 directed programs and reached 39,040 median cycles at 2,763 routed LUT4. GLM 5.3 also passed every public Level 1 check, including eight formal jobs, at 4,146 routed LUT4 and 95,588 median cycles. Level 2: build an RV32IM pipeline. Since 14 September 2026 it must also run CoreMark at 0.6 instructions per cycle or better. In that rerun GPT-6 Astra in Codex passed every public check, formal proofs included, at CoreMark IPC 0.771 on 8,911 LUT4-equivalents, and sealed in 2 hours 3 minutes. Claude Fable 5.1 in Claude Code also passed every public check, with the highest CoreMark IPC (0.837) on the smallest design (6,007 LUT4-equivalents), and sealed in 1 hour 9 minutes. GLM 5.3 in the pi coding agent sealed a design that passes every functional and performance check, CoreMark included at IPC 0.739, but no formal job finished inside the two-hour budget, so it is proof pending; its provider rate-limited the run throughout. Gemini 3.8 Flash passed every test case but failed CoreMark (IPC 0.530) and its interface proof, which found a data request withdrawn before memory accepted it. These are public, provisional results; no sealed hidden result is available. Level 3: take a working reference core and make it faster without breaking it — best public score 5.84.

Network — PCIe cores

A CPU that is wrong computes a wrong number. A controller that is wrong deadlocks, drops a completion, reorders a posted write past a read, or overruns a credit — failures that only appear when a second, independently written agent is talking back to it.

n1-pcie-ep — a Gen1-class x1 PCI Express endpoint controller: flow-control credits, 12-bit sequence numbers, LCRC, ACK/NAK and retry, Type 0 configuration space, BAR decode, completer and requester paths, ordering and error reporting. 46 public cases, 95 hidden. Hidden cases passing: cc-opus5 (Claude Opus 5) 94, codex-sol (GPT-5.6 Sol) 86, grok46 (Grok 4.6) 86.
n2-pcie-ltssm — the physical-layer logical sub-block and link training state machine over PIPE x4: ordered-set framing, descrambling, lane deskew, polarity and reversal, 17 LTSSM states, speed change, hot reset. 28 public cases, 113 hidden. Hidden cases passing: codex-sol 110, cc-opus5 103, grok46 86.
n3-pcie-switch — a three-port switch fabric core above the data link layer: routing by address, ID and bus number, three Type 1 bridge configuration spaces, shared per-class buffering under a hard storage ceiling, cut-through forwarding latency, and ordering and deadlock avoidance across independent ingress-egress pairs. 38 public cases, 128 hidden. Hidden cases passing: cc-opus5 (Claude Opus 5) 128 — exact-correct, on its third attempt — codex-astra (GPT-5.6 Astra) 127, grok46 (Grok 4.6) 125.

These are completion tasks, not optimization tasks: no baseline, no record, no metric to climb. The bar is exact-correct — every case in the hidden suite. n1 and n2 have not been cleared; n3 has, once, on 2026-09-08. All nine runs pass every public case on their own task. The public suite ships in the workspace by design, so passing it is necessary and not sufficient; the hidden suites are what separate the field, and they separate it differently on each task. n1 and n2's pilot ran 2026-09-07: one run per agent, a 150-minute cap against the 6 hours the task specifies, still blocked on reference calibration and tool pins. n3's revision-4 run ran 2026-09-08 under the full 360-minute budget, after an earlier revision that did not discriminate between the three agents at all.

Accelerator — datapath blocks

Nine tasks. Correct rounding is a gate. The score is 10000 × (fmax / latency) / area. Grok 4.6 ran all eight in 60 minutes on 18 September 2026: 7 passed, 1 failed a resource gate.

a1-fixedexp — Grok 4.6 53.11 PASSED. Record: Fable 5.1 904.66.
a2-bf16-rsqrt — Grok 4.6 3,971.35 PASSED. Record: Codex Astra 32,776.97.
a3-bf16-exp — Grok 4.6 83.56 PASSED. Record: Codex Astra 3,322.28.
a4-bf16-gelu — Grok 4.6 407.97 PASSED. Record: Codex Astra 6,216.72.
a5-bf16-sigmoid — Grok 4.6 4,187.98 PASSED. First agent run; author 367.74.
a6-bf16-ln — Opus 5.5 26.73 PASSED, Fable 5.1 20.80 PASSED (30 September 2026). Author's reference fails the clock.
alu1-vreduce-128b — Grok 4.6 41.12 PASSED. Author 15.10.
v1-fixedexp-64b — Grok 4.6 42.12 PASSED (64 DSPs). Author 425.16.
v2-fixedexp-128b — Grok 4.6 FAILED (0 ULP, 105,537 LUT over 64,000 cap). Author 428.23.

This page is an interactive desktop; enable JavaScript to open the leaderboard, the task diagrams, the metric-over-time curves and the run traces. Data: Hugging Face, GitHub, built by ARA Labs.

CHIPBENCH.AI