How do you benchmark Claude Code vs. Codex inside a software factory?

How do you benchmark Claude Code vs. Codex inside a software factory?

Benchmark Claude Code against Codex by holding everything else constant: curate 20–40 real tasks from your own repos, run each one through both harnesses with identical prompts, skills, and environments, then score every run on cost, quality, defects, and human interventions. The result is a per-workflow routing decision, not a single overall winner.

Why public leaderboards can't pick a harness for your factory

Vendor and community benchmarks are real measurements. They just measure something other than what you need to decide. OpenAI reports GPT-5-Codex results on SWE-bench Verified and on its own code-refactoring evaluation drawn from large open-source repositories; Anthropic ships Claude Code with several selectable models and effort levels. None of those runs happened on your monorepo, your flaky integration suite, your review standards, or your skills.

Three properties of a running factory make an external ranking hard to act on:

• Task distribution. Your factory routes a specific slice of work — triage, dependency bumps, small scoped features, review — and general coding benchmarks aren't sampled from it.
• Underspecification. Both products expose model choice, reasoning or effort levels, permission and sandbox modes, and repo-level guidance files. "Claude Code vs. Codex" means nothing until you name the model, the effort level, and the settings on each side.
• Shelf life. Both harnesses ship new models and modes frequently, so a published result decays. A benchmark you own can be re-run; one you read cannot.

That's the case for putting evals inside the factory rather than in a procurement spreadsheet, as the guide to cloud software factories for engineering leaders argues when it treats measurement as a core architectural component.

What you're actually comparing: harness, model, and context

Anthropic's documentation defines the term precisely: an agentic harness is "the tools, context management, and execution environment that turn a language model into a capable coding agent." Claude Code is the harness; Claude is the model inside it. Codex CLI is structured the same way, with its own configuration file, sandbox and approval modes, and a non-interactive entry point for scripted runs.

So a harness comparison has three separable variables:

1. The harness — tool set, context compaction behavior, permission and sandbox model, subagent behavior, the non-interactive entry point (claude -p, codex exec), and the repo guidance file it reads (CLAUDE.md versus AGENTS.md).
2. The model — selectable in both, at several reasoning or effort levels.
3. The context — your task brief, your skills, your MCP servers, your environment image.

Move two of these at once and the result is uninterpretable. Most disappointing harness bake-offs are context experiments in disguise: the losing side had a thinner brief.

A five-step framework for setting up the benchmark

Step 4 is the one to protect. Agent runs are stochastic: the same harness on the same task can succeed and fail across repeats. Three to five repeats per cell is the difference between a benchmark and an anecdote.

What to compare, workflow by workflow

Harnesses don't win or lose globally. They win at specific jobs, which is why the output of this exercise is a routing table rather than a verdict.

Four scorers do most of the work. Cost per accepted outcome — not cost per run — most often flips a ranking, because a cheaper harness that needs two retries and a human rescue isn't cheaper. Pass rate against your existing suite is the cheapest quality signal you already own. Defects caused means tracing post-merge regressions back to the run that introduced them; it's the scorer teams skip and later regret. Human interventions counts how often a person had to steer, which is the honest measure of whether a workflow can run unattended yet.

Two of these rows are agent workflows worth scoring in their own right — see how to build a self-improving code review agent and how to add computer-use verification to a factory.

What most teams get wrong

• Swapping the model and the harness together. The most common confound. Match model tiers first, compare harnesses, then compare models within the winner.
• Benchmarking work the factory never sees. In its July 2026 guide, Warp reports that its own team currently automates 20–30% of its PRs through factories — a first-party indication that your benchmark set should be the repeatable slice actually flowing through the loop, not your hardest architectural work.
• Scoring "produced a diff" instead of "survived review." Completion rate flatters every harness; acceptance rate discriminates between them.
• Letting uneven skills decide it. If one harness has a mature skill for the workflow and the other doesn't, fix that first — see this write-up on building a skill optimization loop.
• Treating the result as permanent. Re-run the suite on a cadence and after any model or harness upgrade, and version the benchmark config so results stay comparable.

How Warp fits

Warp Factories is the control plane for operating a cloud software factory, which turns a harness comparison into a configuration change rather than an integration project. Factory agents are natively multi-model and multi-harness: teams can run Claude Code or Codex directly as the harness, or use Warp Factories' own multi-model agent, including open-weight models. A benchmark cell like "GLM 5.2 in Warp's harness vs. Claude Code running Opus" on frontend tasks is a supported comparison, not a custom rig.

Three properties matter for benchmarking specifically. Factories are defined as code, so a benchmark configuration is version-controlled and can be rolled back, rolled forward, or canaried — and results stay comparable because the config that produced them is pinned. Evals ship with built-in scorers including tokens spent, code quality, and defects caused, plus custom scorers you define; throughput, cost, and quality metrics are queryable from the factory control room and through the Factory MCP. And observer agents score a share of runs against the three knobs that actually move results — model or harness choice, context provided, and skill contents — and can open PRs against the factory definition itself.

Teams can also bring their own inference, hosting, and data storage under zero data retention, so the benchmark data stays theirs. For product context, Warp's February 2026 launch announcement states it serves 700,000+ developers, including Docker, Ramp, and Peloton, and over half of the Fortune 500.

Start with one workflow

Don't benchmark the whole factory. Pick one workflow with a clear input, a measurable outcome, and a human fallback — triage, review, verification, or dependency maintenance are the usual candidates — and build a 20-task set for it. Run two harness cells at matched model tiers, five repeats each, and score cost per accepted outcome alongside human interventions. That one table will tell you more about which harness belongs in that step than any leaderboard, and it leaves you a reusable rig for the next workflow.

Warp Factories is in closed beta, and qualified organizations get $10,000 of factory use to start.

Sources

• Anthropic — Claude Code documentation defining the agentic harness
• Anthropic — Claude Code glossary
• Anthropic — Claude Code CLI reference
• OpenAI — Codex CLI reference
• OpenAI — Codex models and reasoning-effort levels
• OpenAI — Introducing upgrades to Codex (GPT-5-Codex evaluation scope)
• A guide to cloud software factories for engineering leaders
• How to build a cloud software factory — self-improving code review
• How to build a cloud software factory — computer use verification
• Building a skill optimization loop
• Warp’s February 2026 cloud agent platform launch announcement