We turned off our AI code fixers

Fabricate no longer runs an LLM repair pass over generated code. Two stages used to. realtimeCodeFixer inspected each generated file over 50 lines while a phase was still streaming, whenever auto-fix was on. fastCodeFixer ran over the project just before deploy, once the pre-deploy safety gate had flagged blocking issues, repairing whatever it flagged. Both are disabled today: set to a DISABLED model in worker/agents/inferutils/config.ts and short-circuited at every call site before any inference happens.

What replaced them is deliberately less clever. A deterministic pre-deploy safety gate at worker/agents/validators/preDeploySafetyGate.ts, roughly 3,900 lines of AST walks and source pattern checks with no model anywhere in the loop. A set of deterministic source transforms that rewrite known-bad constructs directly. And a phasic pipeline in worker/agents/execution/phasicStateMachine.ts that plans one deployable milestone, implements it, validates it, and only then plans the next.

This post will not quote you a percentage. We have no clean experiment isolating the fixers from everything else that shipped in the same period, so any number here would be invented. What follows is the architecture, the code, and the reasoning, including the part of the trade that went against us.

realtimeCodeFixer lived at worker/agents/assistants/RealtimeCodeFixer.ts and ran per file. When a file over 50 lines finished streaming during phase implementation, it went to a dedicated model call whose system prompt asked it to simulate that file's runtime behaviour and report critical issues: infinite render loops, setState during render, useEffect without a dependency array, unstable store selectors, import integrity problems, undefined property access, JSX tag mismatches. Output came back as search and replace diff blocks applied with fuzzy matching, looping up to five passes per file by default.

The structural weakness is visible in the prompt itself. It tells the model: "You are only provided with this file to review. Assume all imports are correct and exist." That is a reviewer with no cross-file context, asked to judge whether code is correct, in a codebase where most interesting defects are relational. It could see that a component re-renders; it could not see whether the store it reads from exposes that field.

fastCodeFixer was the whole-project counterpart, implemented as PostPhaseCodeFixerOperation in worker/agents/operations/PostPhaseCodeFixer.ts. It sat between the gate and the deploy: when the pre-deploy safety gate reported blocking issues, the flagged issues plus the project files went to a model and came back as a rewritten set of files, which were overlaid on the phase output and re-checked before anything shipped.

The cost shape matters as much as the behaviour. Fabricate bills in weighted tokens (worker/config/billing.ts): input weighs 1.0, output weighs 5.0, cache reads 0.1, and 10,000 weighted tokens make one credit. A repair pass that rewrites a file is almost entirely output tokens, the expensive class. Multiply by five passes per large file, by every large file in a phase, by up to ten phases for an initial build (MAX_PHASES in worker/agents/core/state.ts). That cost was paid on every generation, including the ones with nothing to fix.

A post-generation repair pass is a bet that a second model call will fix more than it breaks. Three quantities decide whether it pays: the base model's defect rate, the fixer's own damage rate, and what the fixer costs to run. Over time one of those moves in your favour and two do not.

When base error rates were high, a repair pass saw a steady supply of genuine defects, so even a mediocre fixer came out ahead. As base quality improves, the population of files reaching the fixer is mostly correct, and its false-positive rate stops being a rounding error and becomes the dominant term. A model handed a correct file and asked to find problems will find problems, because that is the task it was given. It tightens a dependency array that was intentionally loose. It "fixes" a deliberate escape hatch.

The asymmetry is the part people underrate. A generator bug lands in code nothing has validated yet, and the rest of the pipeline exists to catch it. A fixer bug lands in code that already passed, where the remaining checks are thinner, and it arrives labelled as a repair. Worse, the fixer edits by applying search and replace blocks with fuzzy matching. Every application is a small chance to corrupt a file whose only offence was being over 50 lines.

Setting temperature to zero does not solve this. It reduces variance for one call with one context, but the fixer never sees the same context twice: phase concepts differ, prior files differ, the issue list differs. A defect the fixer introduces is therefore not reliably reproducible, which is a bad property for anything on a shipping path.

Meanwhile the cost never decays. Better base models do not make the repair pass cheaper. They make it more likely to be redundant while it charges the same minutes and the same output-weighted tokens on every generation.

The replacement for stochastic repair is deterministic detection: a static gate that either fires or does not, on the same input, every time. In Fabricate that is runPreDeploySafetyGate in worker/agents/validators/preDeploySafetyGate.ts. Its rule set is a closed TypeScript union, PreDeploySafetyRule, so every finding the system can produce is enumerable by reading one type declaration.

The union currently holds 28 rules, and most of them fall into two families. React runtime rules catch the patterns that turn a preview into a white screen: effect_missing_dependency_array, effect_self_dependency_state_update, top_level_state_update_in_render, unstable_store_selector, module_level_jsx. Backend and wiring rules catch the patterns that turn a deployed app into a 500: worker_routes_not_mounted, missing_api_catchall, invalid_worker_entry_fetch_export, hono_missing_appenv, missing_spa_fallback_delegation, schema_column_drift, unresolved_resource_placeholder, empty_drizzle_config. The remainder are one-offs that fit neither bucket, like tailwind_v4_vite_plugin_with_v3 and framer_motion_props_on_non_motion_element.

Two examples of the precision a rule buys. reduce_without_initial_value flags a reduce() with no initial value, because an empty array is the normal first-render state and the call throws "Reduce of empty array with no initial value" before data arrives. unstable_store_selector matches six selector shapes: an object-literal selector, an array-literal selector, a method-chain selector ending in map, filter, reduce, sort, slice, concat or flatMap, an Object.keys/values/entries selector, an identity selector, and a bare store call with no selector at all. The first four return a fresh reference every render; the last two subscribe to the whole store. Every finding carries a file path, a line, an excerpt and a rule id.

The gate is non-blocking by design. When it finds unresolved blocking issues, phasicStateMachine.ts logs them and deploys anyway. The comment in the code explains why: throwing would crash the phase and leave the preview showing the template. Being correct about a defect is not worth handing the user a blank screen.

Detection without repair only moves the problem. The useful observation is that a large share of the defects worth repairing can be repaired without a model at all, because the diagnosis already localizes the fix.

Fabricate has two deterministic repair paths. The first is applyDeterministicFixes, in the safety gate file. It inserts a missing VALUES keyword into SQL inserts, strips worker app stub declarations, moves top-level worker route registrations into the user routes file where they belong, rewrites unstable store selectors into per-primitive hook calls, replaces ts-ignore directives with ts-expect-error, strips the Tailwind v4 Vite plugin out of a project still on v3, and generates a canonical drizzle.config.ts when a schema declares tables but the config is missing or empty. Each is a source-to-source transform with a named entry in transformDetails, so you can see exactly what was rewritten.

The second path handles type errors. applyDeterministicPostPhaseRepairs in phasicStateMachine.ts runs static analysis over the phase output, filters the results down to errors whose ruleId matches the TypeScript diagnostic pattern, and hands them to fixProjectIssues in worker/services/code-fixer. That module has a dedicated fixer per diagnostic code: ts2304, ts2305, ts2307, ts2322, ts2339, ts2345, ts2551, ts2613, ts2614 and ts2724.

This is the shape to aim for. When the compiler reports TS2307 it has already told you which import specifier failed and where. The hard part was the diagnosis, and the compiler did it for free. Spending a model call to add an import is paying inference prices for a lookup. Reserve model calls for problems where the diagnosis is the hard part.

The deepest reason the repair passes became redundant is that the pipeline stopped producing a finished artifact for them to repair. Fabricate generates phasically: worker/agents/execution/phasicStateMachine.ts plans one deployable milestone, implements its files, saves them, runs the gate and the deterministic transforms, deploys, validates, and repeats. Initial builds get up to MAX_PHASES cycles and follow-ups get FOLLOWUP_PHASES, both defined in worker/agents/core/state.ts.

A repair pass over a finished application is archaeology: it receives an artifact and must reconstruct intent from the output in order to judge the output. A per-phase validation loop never loses the intent. It still holds the phase concept that was planned, the files that phase touched, and static analysis scoped to that delta. Checking work against a stated goal is a far easier problem than inferring the goal from the work.

The practical effect is that defects surface while the surface area is small. A broken route registration in phase two gets caught in phase two, when three files exist, rather than being hunted for in a twenty-file project by a model guessing which of the twenty is wrong.

We did not remove every LLM from the repair path. deep_debug is still live and still routed to a model, but it is invoked deliberately, when there is a concrete failure worth investigating, and it gets a real tool set to work with. The distinction that matters is not model versus no model. It is unconditional versus invoked: a stage that runs on every file of every generation has to justify its cost against the majority case, where nothing is wrong.

A static gate only catches what somebody encoded a rule for. That is the whole cost of this decision, and it is not small. Every rule in preDeploySafetyGate.ts exists because a human saw a failure, characterized it precisely enough to express as a pattern, and wrote it down. Novel failure modes walk straight through. An LLM fixer, for all its faults, can in principle notice something no rule anticipated. We gave that up, knowingly.

The second cost is false positives of a different kind. Some rules are pattern matches over source text rather than full semantic analysis. The unstable store selector detector matches hook calls by name shape, so a hook named useSomethingStore that legitimately returns an array will be flagged even when the reference is stable. Keeping the gate non-blocking is partly an acknowledgement of this. A rule that is right most of the time earns a warning, not a veto.

The third cost is that the fix loop moved from generation time to human time. When a novel failure reaches production now, somebody has to notice it, diagnose it, write a rule and ship it. That is slower for the first occurrence. The compensating property is that once the rule exists it is permanent, free to evaluate, and correct on every subsequent run, whereas the fixer re-rolled the dice on every generation forever at full price.

Summarized honestly: we traded recall for determinism, precision, diagnosability and wall clock. If base model quality regressed sharply, or if we moved to a model family that reintroduced a class of defect the rules do not cover, the right response would be to turn a repair pass back on. That is exactly why the operation code is still in the tree.

Disabling a pipeline stage well is its own small design problem. Deleting the code loses the option. Commenting out the call site leaves the configuration lying about what the system does. Leaving the stage wired but pointing at a dead model turns every generation into a stream of retry-exhausted errors in your error tracker.

Fabricate handles it in three layers. The action config in worker/agents/inferutils/config.ts sets the model name to AIModels.DISABLED, which the inference layer checks directly. The AGENT_CONSTRAINTS map holds an entry whose allowedModels set contains only DISABLED. And a helper, isAgentActionDisabled(action), returns true exactly when a constraint permits nothing but DISABLED, giving every call site one cheap question to ask.

Then the call sites actually ask, before doing any work. PhaseImplementationOperation.ts reads the realtimeCodeFixer constraint inline and skips the branch before it even dynamic-imports the fixer module. repairBlockingSafetyIssues in phasicStateMachine.ts calls isAgentActionDisabled for fastCodeFixer and returns an empty array. PostPhaseCodeFixerOperation.execute repeats the check as a second line of defence and logs at info level rather than error.

That last detail is the one teams get wrong. The comment in the code spells it out: a disabled action that reaches the inference layer returns null, and the caller then logs it as a failure after retries. A stage you turned off on purpose should be silent, not noisy. If your dashboards light up every time you disable something, you will stop disabling things.

Prefer deterministic verification over stochastic repair for every defect class you can name. If you can describe a failure precisely enough to write a prompt about it, you can usually describe it precisely enough to write a rule about it, and the rule is faster, free, reproducible, testable, and improvable by a normal code review rather than by empirical prompt archaeology.

Then re-evaluate every LLM-in-the-loop stage whenever the base model changes. Pipeline stages are almost always added in response to a specific failure mode of a specific model version. The failure mode gets fixed upstream by the vendor. The stage stays, because nobody wants to be the person who removed the safety check, and because proving a negative is hard.

A useful reframe: a stage that costs the same whether or not it finds anything is worse than its hit rate makes it look, because you pay it on the majority case where there is nothing to find. Cheap detection plus expensive conditional repair beats expensive unconditional repair, in essentially every pipeline.

And keep the asymmetry in view. Wherever one stage produces and a later stage edits, the editor works on material that already passed some checks, so its mistakes are later, quieter and more expensive than the producer's. That raises the bar an editing stage has to clear. Being right more often than wrong is not enough when its errors cost more.

If you run an LLM stage that edits the output of another LLM stage, four questions will tell you whether it still earns its place.