Reducing your team's cost-per-PR with custom benchmarks

You can reduce your cost-per-PR significantly by optimizing model selections. But how do you decide which models to use across different tasks? Public benchmarks like SWEBench? Manual trial-and-error? The Twitter flavor of the day?

From our testing, the best tool isn’t public benchmarks or recommendations; it's testing on your team's own coding tasks.

In this session, I’ll show how to build a benchmark by replaying your team’s past agent runs using Factory Benchmarks. ​We'll walk through how to find the right sample data, how to score benchmark runs, and how to understand a benchmark report to decide the best model to reduce your team’s cost-per-PR.

We used this approach ourselves to drive down our costs from $80 per PR to $30.