OpenRouter Publishes Official Guide on Scientific Evaluation and Benchmarking for LLMs

On November 12, 2026, OpenRouter published a practical guide on how to benchmark LLMs, framing it as a method for scientific evaluation rather than a vendor ranking exercise.

The guide's central advice is to choose models based on evidence drawn from your own application's inputs and output formats, not on public leaderboards. Concretely, it recommends running the identical set of tasks through OpenRouter and comparing three metrics: graded accuracy, billed cost, and request time. It also stresses that evaluation sets for accuracy, cost, and latency should be defined against your own business requirements, that citation counts should be computed separately rather than folded into task scoring, and that prompt tuning should be kept separate from final evaluation so that tuning choices do not contaminate the results.

The guide illustrates the approach with internal numbers from a synthetic suite of 60 tasks. In that setup, GPT-6.1 Sol passed 180/180 trials across three repeats under the tested contracts and settings. For tasks that are code-based checks, the guide says such tests measure correctness and output compliance. It also notes that DeepSeek had the lowest response cost of the models tested, while scoring 157/180 on the same synthetic set.

As of publication, the guide presents this methodology plus OpenRouter's own synthetic-benchmark results. No separate published leaderboard or refreshed figures accompany it.