Private preview · 3-axis milling · November 2026
MachBench
Can an AI agent program a CNC machine to make a real part? We grade it by cutting the part, in simulation.
5 of 7 parts made by Claude Opus 5.5 with the simulator to test against. 1 of 7 without it.
The grader's own machine simulation running a reference program on a generated test block: 3 setups, 14 operations, 1 min 45 s of machining.
What the grader checks
- Rapid moves into material
- Holder and jaw collisions
- Shank rubbing the part
- Tool deflection
- Chip load
- Cutting force
- Spindle power
- Clamping force
- Part tilting in the vise
- Every face within ±0.05 mm
- Material left in corners
- Holes and threads made
The task
The agent gets what a machine shop would hand a CNC programmer, and has to return a job that runs.
Given: the part as a 3D model with its tolerances, the blank, the machine, the tool magazine, the vises and fixtures, and the tool makers' cutting data.
Asked: how to hold the part in each setup and in which orientation, and one CNC program per setup that makes it. Other tasks ask to fix a broken program or make a working one faster.
Parts: from a public CAD collection, a public test part with its shop programs, and generated parts with known reference programs.
The task's input: the part as a 3D modelDrag to turn
- Material
- EN AW-6082-T6
- Blank → part
- 100 × 80 × 32 → 30 mm
- Tolerance
- ±0.05 mm, every machined face
- Features
- 2 pockets · 4 bolt holes · 6 M6 threads on 3 faces
Why it is hard
One mistake scraps the part.
A wrong grip, a rapid move through the stock or a tool that is too long breaks a tool or crashes the machine. That earns nothing.
Plausible text earns nothing.
A program that looks right but leaves material in a corner fails, just as it would on the shop floor.
Planning is the hard part.
Frontier models already write and fix simple single-setup programs. Deciding how to clamp and sequence a real part is where they still fail.
How we grade
- Cut, not judged
- The programs run on the blank in a 3D simulation of the machine, with the tool, holder and vise jaws as solids. No language model scores the answer.
- Physical partial credit
- Progress follows how much of the part is made within tolerance. A safe but unfinished part keeps some credit; a crash keeps none.
- Every failure has an address
- Each verdict names the program line and the rule behind it. Every rule cites a standard, a handbook or a tool maker's catalogue.
How we test the grader: we plant known faults in proven programs and check that every one loses its reward, and we check that sound programs keep theirs. Measured 15 November 2026.
- of planted faults caught: 282 of 285 broken programs lose their reward
- 98.9%
- of sound programs wrongly failed: 1 of 116
- 0.9%
- grading rules, each with its published source
- 67
- automated tests on the simulator and grader
- 2,300+
First results
Without tools, Claude Opus 5.5 makes 1 of 7 parts. With the simulator to test its drafts against, it makes 5.
A dry run is a full grade of a draft, returned to the agent as what failed and where. Development parts from a public CAD collection, one attempt per part, November 2026. Early signals on small sets, not a final leaderboard. Codex and Gemini runs are on the way.
Leaderboard · 7 development parts
Pass rate, sorted| Model | Setting | Pass rate | Mean Progress | Passed |
|---|---|---|---|---|
| Claude Opus 5.5 | Tools and dry runs | 71% | 0.82 | 5 / 7 |
| Claude Opus 5.5 | Tools; speeds and feeds read from makers' catalogues | 57% | 0.77 | 4 / 7 |
| Claude Opus 5.5 | No tools, one shot (reference) | 14% | 0.41 | 1 / 7 |
| OpenAI Codex | Tools and dry runs | On the way | ||
| Google Gemini | Tools and dry runs | On the way | ||
One attempt per part on each row, so a gap of one part is within noise. The tools and no-tools rows used an earlier wording of the task brief that did not yet spell out the vise rules (stop, minimum grip, jaw pressure); two of the no-tools failures are of that kind, so the no-tools gap is probably somewhat overstated. One no-tools attempt returned no answer and counts as 0. None of the failures in the catalogue setting came from looking up the tables.
The agent improves with every dry run
The 8 hardest development parts. Each point is the mean of the best Progress reached so far, as graded during the run. Progress climbs from 0.33 after the first dry run to 0.77 after the fourth. The final answers score 0.86 on the current grader, and 6 of the 8 parts pass.
Run your model
Labs, researchers and machine shops: ask for a walkthrough, the method card or preview access. Shops can contribute parts to the held-out set.