How Do Evals and Scorers Work in a Cloud Software Factory?

Evals and scorers are how a cloud software factory measures whether its coding agents are actually improving, rather than just producing output. A scorer is a defined rubric — built in or custom — that grades a sample of factory agent runs on dimensions like token cost, code quality, and defects caused. Evals run those scorers repeatedly across configurations so a team can compare, for example, one model or harness against another on the same task.

Why Measurement Is a Distinct Layer from the Workflow Itself

A factory's triage-spec-implement-review-verify pipeline tells you that work moved through the system; it doesn't tell you whether the work was any good, or whether it's getting better or worse over time. Two factories can both report that 80% of PRs merged without human edits and be in very different states — one because agents are genuinely producing clean code, the other because reviewers have stopped catching real problems. Evals and scorers exist to answer the question a workflow can't answer on its own: is the factory actually improving, and on what basis would you know?

What a Scorer Actually Grades

Built-in scorers in a factory typically evaluate three things about a piece of agent work:

• Cost — how many tokens, and how much compute, the run spent to reach a result
• Quality — how good the resulting code is, judged against a rubric or a reference standard
• Defects — whether the change caused downstream issues after it shipped

Teams can also define custom scorers for anything the built-ins don't capture — a scorer for API-contract adherence, one for test-coverage delta, one for adherence to a style guide.

How Evals Connect to Self-Improvement

Evals aren't just a dashboard. Observer agents score a percentage of runs and look for patterns: is the same model getting a worse quality score on frontend tasks than backend ones? Does a longer context window reduce defects but triple cost? Because factory definitions live in code, an observer agent can propose the fix directly — a pull request that swaps the model for a specific task type, or trims a skill's context to cut cost. That's what makes it a self-improvement loop: the eval scores the work, the score changes the configuration, and the changed configuration produces work that gets scored again.

Building a Benchmark from Your Evals

Once scorers are in place, you can go a step further and build a benchmark: a fixed set of representative tasks, each run through multiple model or harness configurations and graded by the same scorer. A benchmark answers a narrower, more decision-oriented question than a general eval — for example, whether a specific open-weight model or Claude Opus scores better on your frontend tasks, controlling for cost.

What Most Teams Get Wrong

Teams often skip evals until something goes wrong, then try to reconstruct quality data after the fact — but a scorer is only useful if it has been grading runs from the start, so there's a trend to compare against. The other common mistake is scoring only for cost. Token spend is the easiest thing to measure, so it's tempting to optimize for it alone; a factory that only tracks cost will quietly get cheaper and worse, because nothing is scoring the defects that a cheaper model or a trimmed context window introduces.

How Warp Fits

Warp Factories builds evals and scorers into the platform rather than treating them as a separate tool a team has to wire up. Every factory run reports queryable metrics on throughput, cost, and quality to the factory's control room, and Warp ships scorers for tokens spent, code quality, and defects out of the box, alongside support for fully custom scorers. Because Warp is natively multi-model and multi-harness, those same scorers double as the basis for a benchmark — comparing, say, an open-weight model in Warp's own harness against Claude Opus running through Claude Code, on the same fixed task set. Warp reports that its own engineering team currently automates 20–30% of its PRs through factories, a figure Warp Factories tracks — and grows — using this same eval infrastructure.

Start with One Scorer

Don't instrument every dimension of quality on day one. Pick the scorer that answers the question you actually have right now — usually cost or defect rate — apply it to one workflow, and let a few weeks of real runs establish a baseline before adding a second scorer or building a formal benchmark.