Benchmark method and limits.

The 41% figure is from our own 5-attempt, 300-test SWE-bench Lite run comparing Opus to Suffice (Golem). Here is what it means.

The result

Suffice averaged 263.4/300 and Opus averaged 262.8/300 (87.8% and 87.6% across 5 trials). Average cost per task: only $0.16, down from Opus's $0.27.

Our next goals

We want help from users to branch out beyond one benchmark: more repos, more languages, more real workflows. Join the waitlist if you can contribute a benchmark or a run.

Raw logs

Raw call logs, predictions and evaluation output are retained with the benchmark run. Request access through the waitlist.