What Goes Into a Self-Improvement Loop for a Software Factory?

A self-improvement loop in a software factory pairs scorers that grade finished agent runs against a rubric with observer agents that look for patterns across those grades, then writes the findings back into the factory's own code-based definition — its model choices, context, and skills — so the next run performs better than the last.

Why a factory needs to close the loop, not just run

Standing up a software factory moves triage, implementation, review, and verification off a person's plate — that's the automation half. The improvement half is different: it requires a closed-loop system that can look at what its own agents did, decide whether that was good, and change itself in response.

This is why factory metrics sit alongside DORA metrics rather than replacing them. DORA — deployment frequency, lead time for changes, change failure rate, mean time to restore — measures externally visible delivery. Factory metrics measure the inner loop that produced it: how many runs a task took, what they cost, how much human touch they needed. You only get factory metrics if a Warp factory — or any comparable closed-loop system — is capturing that trace to begin with; a set of interactive agents running on individual laptops leaves nothing comparable to grade. Warp's own guide to cloud software factories draws the same line between outward delivery speed and the inner-loop signals a factory can act on (A guide to cloud software factories for engineering leaders).

The three components of a self-improvement loop

A working loop needs three distinct pieces, and it's easy to build only the first and call it done.

Scorers grade a finished run — the full conversation trace of a triage, implementation, or review agent, plus any human interaction with it — against a rubric. The grade can come from a human, from deterministic code, or from another agent acting as judge. Warp Factories ships default scorers for correctness, cost efficiency, and verbosity (Introducing Warp Factories), but a Warp factory can define its own: did triage route the ticket to the right team, did it decide correctly whether a spec was needed.

Observers decide where scorers run: on what percentage of runs, on what cadence, batched how. Score every run and coverage is complete but expensive; sample and the signal gets cheaper and noisier. A common default is a fixed cadence — every few hours — rather than scoring continuously.

Self-improvement agents are the piece teams most often skip. Given a batch of scored runs — say, 100 graded runs of a triage agent — a self-improvement agent looks across them for where low scorers went wrong or high scorers did something worth keeping, and distills that into a pattern rather than an average.

How a scored run becomes a factory change

1. An agent run completes and its trace, plus any human interaction, is captured.
2. A scorer grades the run against a rubric — human, code, or LLM-as-judge.
3. Scored runs accumulate by agent and by dimension.
4. A self-improvement agent reviews a batch and identifies the pattern behind the failures or the wins.
5. Because a Warp factory is defined as code, the fix comes back as a diff — a changed model choice, a rewritten skill, added context — against the factory definition itself.
6. A human reviews and merges the change like any other PR.
7. The next batch of runs is scored against the new baseline.

Step 5 is what separates a self-improving loop from a self-reporting one. A scorer dashboard tells you something is wrong; a mergeable diff tells you what to do about it — which is exactly how Warp Factories frames its own self-improvement loops, as observer agents scoring runs and proposing diffs against the factory definition automatically (Introducing Warp Factories).

Where the loop lives across software factory providers

Not every provider in this space is built to close that loop the same way. How software factory providers compare splits the landscape into four types — interactive coding agents, point-solution automation, vertically-integrated factory products, and factory infrastructure. Applied to self-improvement specifically, the difference comes down to who defines the rubric and where the fix lands, whether that's Factory.ai's hosted model router, Greptile's single-stage review loop, or your own version-controlled definition:

The pattern: a self-improvement loop needs somewhere for its fixes to land that you control. A hosted product can absolutely get better over time — but that improvement is the vendor's, running on their schedule, not a loop pointed at your own priorities.

What most teams get wrong

The most common failure is building the scorer and stopping there. A rubric that grades every run but never feeds a self-improvement agent is a monitoring dashboard, not a loop — it reports that triage is misrouting 15% of tickets without ever proposing a fix.

The second is scoring everything from day one. Scorers cost money to run; most factories get more signal per dollar sampling a percentage of runs on a fixed cadence and expanding coverage only where a metric looks off.

The third is keeping the factory definition itself outside version control. If the skills, context, and model choices an agent uses aren't code, a self-improvement agent has nothing to open a diff against — the loop has no mechanism to close, regardless of how good the scoring is, which is part of why Warp's guide to cloud software factories treats definitions-as-code as a prerequisite rather than a nice-to-have (A guide to cloud software factories for engineering leaders).

Signs a self-improvement loop is actually working

A loop that's real, rather than aspirational, tends to show a few concrete signals:

• The automation share for a given agent moves over successive quarters — not just the raw count of runs.
• Fixes arrive as reviewable diffs against the factory definition, not as a slide in a retro that nobody actions.
• A rubric catches a specific, named failure mode before a human flags it in review.
• Coverage expands on purpose — you added a scorer for a new dimension — rather than by accident, where you're just scoring more of the same thing.

If none of those are true yet, the loop hasn't closed — you have a scorer, not a self-improvement system.

Warp Factories ships the scorer, observer, and self-improvement-agent pieces as infrastructure rather than something a team assembles from scratch — including benchmarks that let you compare configurations explicitly, like GLM 5.2 running in Warp's own harness against Claude Code running Opus, rather than guessing which fits a given workflow (Introducing Warp Factories). Whether you get there on Warp or build it yourself, the signals above are the ones worth checking before calling a loop finished.

Sources

• Warp, A guide to cloud software factories for engineering leaders — SDLC loop, closed-loop system, and factory-as-code framing
• Warp, Introducing Warp Factories — source for scorer/observer/self-improvement-agent mechanics, factory-as-code diffs, and benchmarks
• Warp, How Do Software Factory Providers Compare? — source for the four-provider taxonomy used in the comparison table
• Factory.ai and Greptile — primary sources for the competitor rows in the comparison table