How do you choose which AI model to use for each coding task?

Choose per task class, not per organization. Group work into classes — triage, simple frontend changes, backend refactors, code review — then benchmark two or three candidate models on real tasks from your own repositories and route each class to the winner. A single default model overpays on easy work and underperforms on hard work.

Why one default model is the wrong unit of decision

Most teams pick a model the way they pick a database: once, centrally, and then they live with it. That made sense when the choice was between two frontier models with a wide capability gap. It doesn't now. Open-weight models have closed enough of that gap that a large share of everyday engineering work — dependency bumps, test scaffolding, small UI changes, triage decisions — completes just as reliably on a model that costs a fraction of the frontier price.

The cost of the wrong default compounds quietly. Every trivial task routed to a frontier model is an overpayment you never see itemized, and every hard task routed to a cheap model produces a plausible-looking diff that fails review. Both failure modes are invisible without per-task measurement, which is why the decision belongs at the task-class level rather than the org level.

Classify the work before you compare the models

Model selection starts with a taxonomy of your own work, not a leaderboard. A workable first cut has four to six classes, each defined by what it demands of the model rather than by which team files it:

• Classification and triage — read an issue, decide whether it's actionable. Short context, low token count, high volume.
• Bounded changes — a contained edit with an obvious test. Small diffs, well-specified.
• Multi-file implementation — changes that span modules and need the model to hold a mental map of the codebase.
• Review and verification — reading a diff critically, spotting what a generator missed.
• Migration and refactor — mechanical at each step, but long-running and unforgiving of drift.

Pull the classes from runs you've already done. If your platform stores agent traces, you can filter historical runs by diff size, file count, and outcome, and the classes fall out of the data instead of a whiteboard.

A routing table you can fill in this week

The right-hand column is a starting hypothesis, not an answer. The point of the table is that each row gets tested separately, with a scorer that matches what the row actually needs — a triage row scored on agreement, an implementation row scored on review interventions.

Benchmark on your own tasks, not on public leaderboards

Public benchmarks like SWE-bench and Terminal-Bench tell you how models rank on someone else's problems, in someone else's codebase, with none of your context, conventions, or skills loaded. They're useful for narrowing a field from twenty models to three. They cannot tell you which of those three writes better code in your repository.

The internal version is straightforward: curate 20–40 real tasks from prior runs, replay each one from its original repository state under each candidate configuration, and score every result on cost, correctness, and quality. Repetitions matter — running each task once measures luck as much as capability.

Warp reports that using its own internal benchmark, WarpBench — built with Factory Benchmarks on its own coding tasks rather than a public set — it adjusted its factory configuration to reduce costs by 63% on certain types of tasks without impacting quality, and that GPT 5.6 Sol (high) came out on top for general-purpose work on that bench. That's a first-party result on its own workloads, and the transferable part is the method, not the winning model — your codebase will produce a different ranking.

What most teams get wrong

Three mistakes recur. The first is treating the model choice as permanent: a bake-off run in March is stale by June, because the models changed and so did your code. The second is comparing models while changing prompts, skills, or environments at the same time, which makes the result uninterpretable — hold everything else constant. The third is routing by team preference rather than measured outcome, which is how an organization ends up paying frontier prices for dependency updates.

Benchmarks are also not free to run. They're worth running when a new model ships, when you're changing agent prompts or skills, or when your cost per PR moves the wrong way — not on a nightly cron.

How Warp fits

Warp Factories is the control plane teams use to operate a cloud software factory, and model choice is a setting inside it rather than a property of the platform. The inference layer of the factory stack is designed to be open: frontier and open-weight models, multiple harnesses including Warp's own agent, Claude Code, and Codex, and inference sourced from model labs, neo-clouds like Baseten or Fireworks, hyperscalers like Bedrock and Vertex, or your own fine-tuned models.

Once a benchmark produces a ranking, Warp turns it into behavior through custom model routers — classifier-driven rules that read an incoming task, classify it, and dispatch it to the configuration that won for that class. Because factories are defined as code, a router is a versioned file: reviewable, revertible, and editable by an agent as easily as by a person.

Start with one task class

Pick the highest-volume, lowest-risk class you have — usually triage or small bounded changes. Curate ten to twenty real tasks from prior runs, bake off your current default against one cheaper candidate, and score on cost plus one quality dimension that matters for that class. If the cheaper model holds, write the routing rule and move to the next class. That single loop is the whole practice; everything else is repetition.

For a broader view of how model choice fits the rest of the SDLC loop, see A guide to cloud software factories for engineering leaders.

Sources

• The Factory Stack — Warp, September 2026
• Introducing Factory Benchmarks — Warp, September 2026
• Introducing Warp Factories — Warp, August 2026
• A guide to cloud software factories for engineering leaders — Warp, July 2026