Research and evidence
Evidence should decide what work agents may own.
Veyl studies the gap between an agent appearing successful and producing an outcome an organization can actually trust. The current public library begins with coding agents because open repositories make the work independently reproducible. Broader enterprise-agent research is in development.
These studies are not leaderboards or customer results. Each one demonstrates a different part of the operating decision: finding hidden business failures, granting a bounded yes, and discovering which checks ordinary CI is missing.
Every Agent Change Passed the Tests. Seven Were Still Wrong.
Veyl independently reconstructed three historical workflows from Ramp’s public CLI repository at pinned commits. Two exact agent-and-reviewer deployments attempted each workflow three times, producing 18 final repairs.
All 18 passed the repository and disclosed checks. Seven still violated held-out business rules written before the agents ran.
| Reconstructed workflow | Held-out result | Veyl decision |
|---|---|---|
| Respect a user’s deliberately deleted setting | 6/6 passed | bounded delegation |
| Match recovery guidance to the real failure | 5/6 passed | supervise |
| Distinguish user-owned from managed configuration | 0/6 passed | do not delegate |
The sharpest failure was configuration ownership. Six repairs protected a user’s custom setting but failed to recognize existing managed state, leaving that state stale. Another repair could recommend fixing request fields for a server-side 500 error. Both exact deployments therefore failed to qualify for unsupervised use across all three workflows.
This is the commercial job: not producing a generic model score, but showing where one exact deployment can act independently, where it needs review, and where it must stop.
This was an independent reconstruction from public ramp-public/ramp-cli history, not a Ramp engagement, customer result, production assessment, or statement about Ramp’s private systems.
Read the full Ramp CLI study →
A Narrow Yes Is More Useful Than an Agent Score.
Veyl also reconstructed three public Brex Substation workflow families: copying large financial identifiers exactly, routing records with missing fields, and scanning large records without truncation. Both exact deployments passed all 18 held-out runs.
Veyl approved those three reconstructed lanes for bounded delegation. It did not claim that either deployment was globally better, and it did not extend the approval to other workflows. A useful control system must be able to say a narrow yes as well as a defensible no.
This was an independent reconstruction from public brexhq/substation history, not a Brex engagement, customer result, production assessment, or statement about Brex’s private systems.
Read the full Brex Substation study →
Twenty Payment Bugs Passed a 71-Package Test Suite.
Before a deployment can be qualified, Veyl needs scenarios that represent real failure conditions. We seeded 32 single-identifier defects into the public ACH payments library and ran the repository’s own suite — 71 packages — against each one. Twenty survived. The suite stayed green while the library was materially broken.
| Seeded defect — every one genuinely breaks behaviour | Repository’s own suite |
|---|---|
| isDebitTransactionCode − LoanDebit | missed |
| isDebitTransactionCode − SavingsReturnNOCDebit | missed |
| isCreditTransactionCode − GLReturnNOCCredit | missed |
| calculateBatchAmounts − SavingsCredit | missed |
| isDebitTransactionCode − CheckingDebit | caught |
| isCreditTransactionCode − SavingsCredit | caught |
Six of thirty-two shown. A single dropped identifier in a transaction-code classifier passed every test the project has, and broke loan-credit reversals at runtime.
What the green suite could not see
Those twenty survivors reveal behaviours the software depends on but its visible checks do not defend. An agent could break any one of them while every ordinary signal remained green.
Inside a company, Veyl finds scenarios of this shape in approved pull requests, incidents, tests, rollbacks, and agent runs. An accountable engineer confirms what matters. Veyl then turns that judgment into private executable evidence and tests the complete agent deployment against it.
The result is not a score about coding agents in general. It is evidence about what one exact deployment can be trusted to do inside one real engineering workflow.
moov-io/ach is a public Apache-2.0 repository. This is an internal demonstration, not a customer engagement. Customer scenarios remain private: never published, resold, or used for training.