Public-repository study / Ramp CLI
Every Agent Change Passed the Tests. Seven Were Still Wrong.
We independently reconstructed three historical failure patterns in Ramp’s public CLI. Two coding-agent deployments produced 18 reviewed repairs. The repository tests and every check disclosed to the agents approved all 18. Held-out business rules rejected seven.
The business decision was different for each workflow
| Reconstructed workflow | Held-out result | Operating decision |
|---|---|---|
| Respect a user’s deliberately deleted setting | 6/6 passed | delegate within this lane |
| Match recovery guidance to the actual failure | 5/6 passed | require supervision |
| Distinguish user-owned from managed configuration | 0/6 passed | do not delegate |
What the ordinary checks missed
Six repairs correctly protected a user’s custom setting, but failed to recognize existing Ramp-managed state. They would leave managed configuration stale instead of safely refreshing it. One other repair could tell a user to fix request fields after a server-side 500 error.
The meaningful result is not a generic failure rate. It is the separation of one apparently uniform set of green changes into three different operating boundaries: one lane the deployment could handle, one that still needed review, and one it should not own.
How the study was run
For each workflow, one Claude model repaired the pinned public repository and the other reviewed the exact frozen result. We tested both role pairings three times per workflow. The plans and held-out rules were fixed before the agents ran, and failures were retained rather than retried away.
Neither complete deployment qualified for unsupervised use across all three reconstructed workflows. The sample does not establish that one model is generally better than the other.
Inspect the basis
Public source repository · plans and checks · Veyl methodology
This is an independent reconstruction from public ramp-public/ramp-cli history at pinned commits. It is not a Ramp engagement, customer result, production assessment, or statement about Ramp’s private systems. A real deployment decision requires company-selected workflows, private evidence, and company approval.