Private preview · 3-axis milling · November 2026

MachBench

Can an AI agent program a CNC machine to make a real part? We grade it by cutting the part, in simulation.

5 of 7 parts made by Claude Opus 5.5 with the simulator to test against. 1 of 7 without it.

Simulated cut · 5× speed
PassProgress 1.00Over-cut 0.0 mm³Outside ±0.05 mm 0.0 mm³

The grader's own machine simulation running a reference program on a generated test block: 3 setups, 14 operations, 1 min 45 s of machining.

What the grader checks

  • Rapid moves into material
  • Holder and jaw collisions
  • Shank rubbing the part
  • Tool deflection
  • Chip load
  • Cutting force
  • Spindle power
  • Clamping force
  • Part tilting in the vise
  • Every face within ±0.05 mm
  • Material left in corners
  • Holes and threads made

The task

The agent gets what a machine shop would hand a CNC programmer, and has to return a job that runs.

Given: the part as a 3D model with its tolerances, the blank, the machine, the tool magazine, the vises and fixtures, and the tool makers' cutting data.

Asked: how to hold the part in each setup and in which orientation, and one CNC program per setup that makes it. Other tasks ask to fix a broken program or make a working one faster.

Parts: from a public CAD collection, a public test part with its shop programs, and generated parts with known reference programs.

The finished test block, top and bottom

The task's input: the part as a 3D modelDrag to turn

Material
EN AW-6082-T6
Blank → part
100 × 80 × 32 → 30 mm
Tolerance
±0.05 mm, every machined face
Features
2 pockets · 4 bolt holes · 6 M6 threads on 3 faces

Why it is hard

  1. One mistake scraps the part.

    A wrong grip, a rapid move through the stock or a tool that is too long breaks a tool or crashes the machine. That earns nothing.

  2. Plausible text earns nothing.

    A program that looks right but leaves material in a corner fails, just as it would on the shop floor.

  3. Planning is the hard part.

    Frontier models already write and fix simple single-setup programs. Deciding how to clamp and sequence a real part is where they still fail.

How we grade

Cut, not judged
The programs run on the blank in a 3D simulation of the machine, with the tool, holder and vise jaws as solids. No language model scores the answer.
Physical partial credit
Progress follows how much of the part is made within tolerance. A safe but unfinished part keeps some credit; a crash keeps none.
Every failure has an address
Each verdict names the program line and the rule behind it. Every rule cites a standard, a handbook or a tool maker's catalogue.
PassNo error-level rule broken, every face in tolerance
≤ 0.75Safe, but the part is unfinished
≤ 0.5Part made with errors, such as a gouge
0A crash, an unsafe move or no usable answer

How we test the grader: we plant known faults in proven programs and check that every one loses its reward, and we check that sound programs keep theirs. Measured 15 November 2026.

of planted faults caught: 282 of 285 broken programs lose their reward
98.9%
of sound programs wrongly failed: 1 of 116
0.9%
grading rules, each with its published source
67
automated tests on the simulator and grader
2,300+

First results

Without tools, Claude Opus 5.5 makes 1 of 7 parts. With the simulator to test its drafts against, it makes 5.

A dry run is a full grade of a draft, returned to the agent as what failed and where. Development parts from a public CAD collection, one attempt per part, November 2026. Early signals on small sets, not a final leaderboard. Codex and Gemini runs are on the way.

Leaderboard · 7 development parts

Pass rate, sorted
ModelSettingPass rateMean ProgressPassed
Claude Opus 5.5Tools and dry runs71%0.825 / 7
Claude Opus 5.5Tools; speeds and feeds read from makers' catalogues57%0.774 / 7
Claude Opus 5.5No tools, one shot (reference)14%0.411 / 7
OpenAI CodexTools and dry runsOn the way
Google GeminiTools and dry runsOn the way

One attempt per part on each row, so a gap of one part is within noise. The tools and no-tools rows used an earlier wording of the task brief that did not yet spell out the vise rules (stop, minimum grip, jaw pressure); two of the no-tools failures are of that kind, so the no-tools gap is probably somewhat overstated. One no-tools attempt returned no answer and counts as 0. None of the failures in the catalogue setting came from looking up the tables.

The agent improves with every dry run

The 8 hardest development parts. Each point is the mean of the best Progress reached so far, as graded during the run. Progress climbs from 0.33 after the first dry run to 0.77 after the fourth. The final answers score 0.86 on the current grader, and 6 of the 8 parts pass.

0.00.51.0 0.330.470.620.770.77Dry run 10 of 8 passDry run 23 of 8 passDry run 34 of 8 passDry run 45 of 8 passDry run 55 of 8 pass

Run your model

Labs, researchers and machine shops: ask for a walkthrough, the method card or preview access. Shops can contribute parts to the held-out set.

Request access hello@machbench.com