Build Log / AI-Assisted Development

Generating Code Was Never the Bottleneck: What Two Codebases Taught Me About Auditing AI

July 2026aiagentscode-reviewverificationmethodology
I did not set out to invent an audit methodology. I set out to build software with AI without shipping something that breaks in production. The first few months taught me that the model writing the code was the easy part, and that reading a green test run and calling it done was how you end up with a codebase nobody can trust. What follows is the method I kept arriving at: adversarial, iterative, and built entirely around the one thing an AI cannot fake, which is a test that failed before the fix and passed after. I arrived at it twice, on two projects with nothing in common. Then I searched and found other people had arrived at the same place. That convergence is the whole point.

The Two Ways It Goes Wrong

There are two failure modes I keep watching play out, and they look like opposites.

The first is a project moving too fast. Unlimited budget, a team, good tooling, and a merge rate no human could review. Every change passes its checks in isolation. The assembled product does not launch half the time. The stability trend goes negative: the best release is in the past, and each new one scores worse than the last. Users start pinning old versions and talking about forks.

The second is a mature, feature-complete tool. It has been stable for years and is wired into millions of scripts and cron jobs nobody has opened in a long time. Someone decides to revive it, accelerates the change rate, and a couple of patch releases that were supposed to be routine bugfixes quietly break existing behavior. The blast radius is enormous and the discovery is late.

These are the same disease. Change without coherence. In the first case the velocity is too high for anyone to hold the system in their head. In the second the tool was done, and any nontrivial change was almost pure downside risk. Both are what happens when generation is cheap and verification is an afterthought.

I have spent the last stretch building the opposite of both. Not because I am disciplined by temperament. Because I watched where the other road goes.

I Did Not Invent This

The method in this post is not novel, and that is the argument, not a disclaimer.

I first put it together on an earlier project, a financial and accounting backend in a completely different language and framework from anything I work in now. The domain forced it: in a system that posts double-entry ledgers and moves money, a subtly wrong calculation is not a cosmetic bug, and “the tests pass” is not the same as “the arithmetic is correct.” So I built a repeatable audit that scanned the whole codebase, confirmed each suspected violation against the actual source, fixed it, and repeated until several consecutive passes came back with nothing.

Later I built Rakkr, an open-source Linux audio-recording platform in an entirely unrelated domain, and found myself reaching for the same shape without thinking about it. Parallel readers, confirm before you trust, fix with proof, repeat until it goes quiet. The method had ported itself.

Then I went looking, and found academic papers describing “adversarial convergence” development loops and iterative audit convergence with clean-pass stopping rules, and off-the-shelf agent skills that spawn parallel auditors and re-audit until clean. People I have never met, working from different starting points, had landed in the same place.

I think this is just what you converge on if you write a real amount of code with an AI and actually pay attention to what it produces. Not the demos. The diffs. The method is a fixed point. It is the thing caring turns into.

Generating Was Never the Hard Part

The pitch for coding with AI is throughput. That framing hides the real problem.

Generation was never the bottleneck. Verification and coherence are. An AI can produce a plausible, well-formatted, test-having change faster than you can read it. Whether that change is correct, and whether it is coherent with the forty other changes around it, is a judgment that does not get faster. Optimizing generation harder is optimizing the resource that was already abundant, and it just moves the bottleneck downstream and makes it worse.

The specific way AI-written code fails is worth naming, because it is not the way human code fails.

It fails plausibly. The output looks right. It compiles, it reads cleanly, and presentation is not a quality signal. A model trained on a planet of code is very good at producing something that resembles a correct solution, which is exactly the property that makes a wrong one hard to catch.

It fails on the unhappy path. Happy-path logic is heavily represented in training data, so the model gets it right. Empty inputs, boundary values, the error cases, the second concurrent writer: these are where the gaps cluster, and they are the parts a fast reviewer skims.

It fails globally. This is the one that eats projects. A gate checks one change in isolation. “Does not launch half the time” is not a per-change failure; every one of those changes went green. It is an emergent failure, a set of individually valid changes that do not cohere. You cannot gate your way to global coherence with local checks, because coherence is a property of the whole and only shows up when you assemble the whole.

The thing that maintains global coherence is judgment holding the system in view. That resource does not parallelize. You can add gates in parallel and generation in parallel. You cannot add coherence in parallel, because a second mind holds its own model of the system, not yours, and now two models drift. That is why more people and more tokens can make a project worse instead of better.

So the method is not about generating better. It is about verifying at a scale that matches how fast the generation happens.

The Method, With the Project Removed

Here is the method stripped of both projects. These are the invariants that showed up in the financial backend and Rakkr alike, which is how I know they are not domain-specific.

Read-only fan-out, single-writer fixes. Launch several independent readers in parallel, one per dimension of the codebase: authorization, core correctness, data-loss paths, input validation, concurrency, and so on. They only read. They never edit. Then one writer, the main loop, fixes findings one at a time. Parallel reading is safe and fast. Parallel writing is how two agents race on the same file and corrupt each other’s work.

A lead is not a finding. This is the single most important rule, and the one everyone skips. A reader’s output is a lead, not a fact. The main loop re-reads the actual source and reproduces the problem before it counts as real. Models over-report. Asked to check their own work, they say it looks fine. Asked to find bugs, they invent plausible ones. The confirm step is the only defense, and it is not optional.

Try to disprove it. For each surviving lead, the verification is adversarial. The question is not “is this a bug” but “prove this is not a bug.” A finding that survives an honest attempt to refute it is worth acting on. One that does not gets discarded rather than padding the ledger.

Proof is red then green. A fix is not “done” until a test fails against the current code and passes after the fix. No failing test, no claim of fixed. This is the load-bearing rule, because it is the only part of the entire method that an AI cannot fake. Everything else rests on model judgment. A test that goes red then green is ground truth. If a confirmed bug cannot be given a failing test in the available harness, because it needs a real database or hardware or wall-clock time, it does not get quietly fixed. It gets written down as confirmed-but-unproven, with the exact fix recorded for later.

Iterate until it goes quiet. One pass is stochastic. It misses things a slightly different pass would catch. So you run passes until some number of them in a row come back completely clean. The count is a proxy for “the surface is quiet,” and consecutive clean runs turn many noisy individual passes into convergent coverage.

Keep an honest ledger. Every finding is recorded with a truthful status. Fixed means a test landed. Catalogued means confirmed but not yet proven-and-fixed. Suspected means a strong lead that did not fully hold up. The status labels are not decoration. They are the difference between a document you can trust and a changelog of vibes.

The two projects also showed me the method maturing, which was the most convincing evidence of all. The earlier version had the skeleton: parallel readers, confirm before trust, repeat until several clean passes. What it did not yet have was the rigor. “Confirm it is not a false positive” grew into “prove it with a failing test.” “Run until clean” grew rules for what resets the count when new code lands underneath you. Flat findings grew the fixed / catalogued / suspected taxonomy. Same fixed point, deeper over time. I was not copying a framework. I was rediscovering the same shape and then hardening it. The current, full version — the operator contract, the clean-run rules, the exact ledger format — is written up in Rakkr’s docs.

The Receipts

Claims about methodology are cheap. Here is the data.

On Rakkr I ran two full audits. Between them they closed 134 confirmed findings, each with a red-then-green test or resolved by another fix. Not 134 suspicions. 134 bugs that had a test which failed before and passed after.

The first audit converged. It took 28 runs plus a warm-up batch and closed 84 findings, and then went genuinely quiet: five consecutive clean runs at the end, nothing left to find on the surface I could test from my machine. The remaining known issues were hardware-gated, and they got closed later on the real rig.

The second audit did not converge, and that turned out to be the more useful result. It opened right after a large new feature landed, closed 50 findings, and never strung together five clean runs. The yield per run, counting confirmed fixes, went like this:

run:    1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
fixed:  5  7  3  1  1  3  1  1  0  0  1  2  2  1  1  1  1  1  1  0  2  0  1  0

Read that curve. It looked converged at runs 9 and 10, two clean passes in a row. If I had stopped there I would have shipped with two critical bugs still live, one found at run 11 and another at run 18, plus high-severity findings at runs 19, 21, and 23. A critical bug at run 18 is not the dregs. The tail was long and it was still paying out.

That is the lesson the data teaches that the theory does not: as a codebase grows, stringing together consecutive clean runs gets exponentially harder, and you often will not converge. But iterating to a fixed budget stays productive well past the point it stops converging, because the value is severity-weighted yield, not a clean streak. A run that surfaces a critical at depth is worth far more than it costs, even though it resets the streak to zero.

And the earlier financial project is the control. Same method, a domain with nothing in common, and it found the class of bug that domain produces: a general-ledger service that had swapped base-currency and foreign-currency fields, native floating-point arithmetic where decimal precision was mandatory, queries missing their tenant scope, multi-step money movements that were not wrapped in a single transaction and could commit half a change. Different bugs, identical machine. That the same loop finds real defects in double-entry accounting and in a Rust audio agent is the strongest evidence I have that the method is not a story I tell about one project.

What This Does Not Do

The parts that do not work are the parts worth being honest about, because everything above depends on this section being true.

It does not prove absence. You cannot prove there are no bugs, not with this and not with anything. It is stochastic search. It accumulates evidence, not proof. A clean run means these particular readers, framed this way, found nothing. It does not mean nothing is there. Security is the worst case: a clean audit is “no one on this pass found a way in,” never “there is no way in,” which is exactly what a human penetration test is too.

It is self-graded, except for one thing. The readers, the verifier, and the ledger author are the same kind of system. The one part that is not self-judgment is the red-then-green test. That is why it is load-bearing. Trust a finding marked fixed far more than one marked verified-clean, because the first has an external oracle and the second is an opinion.

Cost is the real constraint. Running many parallel readers many times is not free, and it scales with codebase size. The lever that helped most was matching the model to the job: cheap, fast models for the read-only fan-out, where you want coverage and the verifier re-checks everything anyway, and the expensive model reserved for the single-writer fix loop where judgment matters. Scoping most runs to what actually changed, rather than re-reading the whole tree every time, keeps the cost tied to churn instead of size.

The orchestrator is one person, and one person fatigues. This sounds like a limitation. It is also a governor. A single person can only drive so much change before running out, which physically caps velocity below the rate at which coherence breaks down. The projects that go negative removed that governor, more people and more budget, without replacing the coherence function, because coherence cannot be staffed or funded. It can only be not-exceeded. The way a solo developer raises that ceiling is not by going faster. It is by moving coherence out of their head and into artifacts: the tests, the design docs, the ledger. Every invariant written down is one you no longer have to hold in memory.

When the Answer Is to Not Touch It

There is a case where the right amount of AI is none, and it is worth stating plainly because the tooling pushes the other way.

For a growing project, change is mostly value-additive. For a feature-complete one, the sign flips. There are almost no features left that users want, so nearly every change is a small-upside bugfix or a large-downside regression risk, and the blast radius is everything the tool is wired into. A stable tool that has not changed in years is not simple code. It is years of encoded implicit correctness, every edge case that ever broke now fixed and load-bearing, documented nowhere except the code’s exact current shape and the silence of the bug tracker. AI is very good at making such code look cleaner and more modern while being blind to why it was shaped that way. That is the wrong objective for code whose entire value is its ugly, battle-tested, specific behavior.

The line I use is not “simple versus complex.” It is verification-completeness. A mechanical change is safe not because it is easy but because you can verify it completely: the diff, the types, a test cover the whole correctness criterion. A behavioral change to code that millions of unseen scripts depend on is unsafe because verification is incomplete by nature, and you cannot enumerate everything you might have broken. AI’s genuine sweet spot is code where a mistake is cheap, or catchable. Load-bearing infrastructure is the exact opposite, and it is the worst possible place to trade human review for machine speed.

Where It Landed

The durable output of all this is not the clean streak. A clean run leaves nothing behind. The thing that ratchets quality upward is the test each fix deposits, because it is deterministic, permanent, and it defends the next pass. You cannot iterate your way to proof, but every test you land converts a stochastically-found bug into a guarded invariant, and over time the guarded surface grows and there is less left to find. That is the real convergence, not the streak: the deterministic guards catching up to the reachable behavior.

Which is also why I bothered to write this down. Most writing about coding with AI is either “ten times your output” or “the slop is killing us.” The honest middle, with actual numbers and the failures included, is rare because it is the unglamorous part to write. I did not invent any of this. If you have built something real with these tools and paid attention, you probably recognize most of it already. That is the point. It is not a framework to adopt. It is where you end up.

Technical Notes

Rakkr is a Hono and React controller with a Rust recorder agent; the earlier project is a Node and Postgres financial backend that stays private. The audit loops are driven by parallel read-only agents with an adversarial verify pass and red-then-green proof for every confirmed fix, tracked in a per-run ledger. Rakkr is open source, so these receipts are checkable: the audit methodology and the raw run ledgers are public. The 134 figure is the count of findings closed across the two Rakkr audits, each with a landed test. The yield curve above is the confirmed-fix count per run of the second, non-converging audit.

Rakkr’s source is at github.com/yashau/Rakkr. The earlier project’s specifics are kept out of this post; the method is what carries over.