What Metrics Actually Prove AI Coding Agents Are Working?

No single metric proves AI coding agents are working. The metrics that hold up together are autonomy rate (the share of PRs that ship without a human rewrite), cost per merged PR, defect or rollback rate, human intervention rate, and cycle time — each measured against a pre-agent baseline, not against seat counts or raw PR volume.

Why seat count and PR volume don't answer this

"How many developers use it" and "how many PRs did agents touch" are the two numbers most teams report first, and neither says anything about whether the work was good. A high PR count can mean agents are productive, or it can mean a team is generating more code to review than it can actually review — the exact failure mode engineering leaders describe when they say verification, not generation, is now their bottleneck. Adoption numbers describe usage. They don't describe outcome.

The five metrics that actually hold up

Effectiveness is a system of five signals read together, not any one in isolation:

No platform should be judged on any single row. A low cost-per-PR with a high rollback rate is not a win; it's deferred cost.

Which DORA metrics survive contact with agents — and which break

DORA's four keys — deployment frequency, lead time for changes, change failure rate, and time to restore service — were built for human-paced delivery, and not all of them translate cleanly:

• Deployment frequency mostly holds up, but it can inflate mechanically once agents can open many small PRs quickly. Read it alongside change failure rate, not alone.
• Lead time for changes holds up well and is one of the more useful agent metrics, provided the clock starts at task assignment, not at PR open — agents can make PR-open-to-merge look fast while task-assignment-to-PR-open stays untouched.
• Change failure rate holds up and matters more with agents, not less — it's the closest proxy to the defect/rollback rate above.
• Time to restore service mostly breaks as an agent-specific signal: incident response is still overwhelmingly human-led today, so treating this as an agent metric overstates what agents are actually doing.

How to attribute a merged PR to agent vs. human work

Most teams can't currently answer "did the agent or the developer actually produce this" with confidence, because git blame doesn't distinguish authorship intent — a human-authored one-line fix on top of an agent's draft looks identical in the diff to a fully agent-authored PR. Attribution needs a session-level record, not a commit-level guess: which agent session opened the PR, what percentage of the final diff came from that session's output versus subsequent human commits, and whether a human's only contribution was review approval. Without that record, autonomy rate and cost-per-merged-PR are both estimates dressed up as measurements.

What most teams get wrong

The common mistake is measuring once, at rollout, and treating that number as durable. Agent effectiveness changes as models change, as a team's skills and context libraries mature, and as reviewers calibrate how much they trust agent output — a single benchmark from launch week says nothing about month six. The fix is treating these five metrics as a running dashboard, not a one-time pilot report.

The second mistake is measuring agents against a zero baseline instead of a real one. "Agents shipped 200 PRs this quarter" means nothing without knowing what the team's PR throughput, defect rate, and cycle time looked like the quarter before agents arrived. Warp reports that its own engineering team currently automates 20–30% of its PRs through factories — a specific, bounded number precisely because it's measured against Warp's own prior throughput, not asserted in isolation.

How Warp fits

Warp Factories expose a queryable set of metrics on agent throughput, cost, and quality directly in the factory's control room, alongside built-in scorers that grade how well work items move through the pipeline — including token spend, code quality, and whether a change caused a defect. Because factory definitions live as code, teams can also define custom scorers for their own version of the five metrics above, rather than settling for whatever a vendor's dashboard happens to expose.

That measurement layer feeds Warp's self-improvement loops: observer agents score a percentage of runs, look for regressions in cost or quality, and can open PRs that adjust the factory's own configuration — model choice, context, or skills — when a metric drifts. That is a direct, structural answer to "prove it's still working six months from now," not just at rollout.

Start with one workflow

Don't try to instrument all five metrics across every agent-touched repo at once. Pick one bounded workflow — triage, dependency updates, or incident follow-up — and track autonomy rate, cost per merged item, and intervention rate on that workflow alone for a month before expanding. A clean read on one workflow is worth more than a noisy read across ten.

See this in action with Warp Factories, or apply for the closed beta.

Sources

• Introducing Warp Factories
• A guide to cloud software factories for engineering leaders
• How much does it cost to build a software factory in-house vs. buy one?
• How Can Multi-Agent Systems Improve Developer Productivity?
• DORA (DevOps Research and Assessment) four keys metrics