Developer builds a multi-axis LLM eval harness in about 120 lines of Python

On 2026-10-31, the AI channel of DEV Community published a tutorial describing how to build a multi-axis model evaluation harness in roughly 120 lines of Python. The author says the harness has minimal dependencies and reports per-axis accuracy, bootstrap confidence intervals and cost side by side, instead of collapsing everything into a single overall score.

As a worked example, two models are compared. One scores 0.675 on the eval and the other 0.713. Sorted by that total, the first ranks below the second, and the presentation slides even label the pair as roughly equal. Once the same two models are broken down axis by axis, however, the picture changes: one model is clearly stronger at arithmetic and string manipulation, while the other is clearly stronger at long-context and multi-step work.

The piece does not name the two models or their versions, does not spell out how the capability axes are defined, and does not explain how cost is measured, so those details remain open. The takeaway the author pushes is that a single aggregate score can hide the shape of what a model actually does, and that per-axis reporting, together with confidence intervals and cost, gives a fuller basis for comparison than a leaderboard rank alone.