News · Analysis
Why ±2 points is a tie: reading local LLM agent benchmark scores
Published
If you rerun the same local model on the same agent tasks, the score moves. On LocalAgentBoard it moves by about two points out of 30. So a 30/30 does not beat a 29/30, and the board picks its overall winner by speed instead.
Same setup, different scores
Language models sample their answers, so the same model with the same settings does not take the same path through a task every time. One run might pick a slightly different command, take an extra turn or miss a detail that the checker requires.
We measured this directly. gpt-oss-20b MXFP4 scored 27/30 on Hermes Agent v0.22.4, then 30/30 and 28/30 in two reruns on v0.22.5. Nothing about the model, quant or server changed in a way that would explain a three-point swing. When we reran ten other setups after the Hermes Agent v0.22.5 upgrade, eight of them landed within two points of their earlier score (see the v0.22.5 retest summary).
With 10 tasks and 3 tries each, one run is worth more than three percentage points. A single unlucky run is enough to reorder the top of a table.
The rule: within 2 points means tied
LocalAgentBoard treats setups that score within 2 points of 30 of each other as tied. On the leaderboard, every setup within 2 points of the #1 score gets an ≈ Top group tag. As of 25 October 2026 that group holds nine setups, from 30/30 down to 28/30:
- 30/30: Qwen3.6-27B Q6_K, Qwen3.6-35B-A3B Q8_0, gpt-oss-120b, Gemma 4 26B-A4B Q8_0, Gemma 4 26B-A4B UD-Q4_K_M
- 29/30: Qwen3.6-35B-A3B UD-Q6_K, Qwen3.6-35B-A3B UD-Q4_K_M, Nemotron-Cascade-2-30B-A3B Q4_K_M
- 28/30: Nemotron-Cascade-2-30B-A3B Q8_0
Best overall is the fastest setup in the top group
If the top scores are effectively tied, ranking them by score alone rewards luck. It also produced an odd result: Qwen3.6-27B Q6_K reached 30/30 on its rerun and became the top pick, but it decodes at only 9.4 tokens per second on the DGX Spark. Nemotron-Cascade-2-30B-A3B Q4_K_M scored 29/30, the same within noise, at 71.5 tokens per second.
For an agent that makes many model calls per task, that speed difference changes how the tool feels to use. So Best overall is now the setup with the highest measured decode speed among top-group setups. Two kinds of setup are left out of that choice: setups whose rerun is under review (their score changed by more than 2 points between runs) and setups with no speed measurement yet.
When results move by more than 2 points
If a rerun moves a score by more than 2 points, something beyond normal noise may be going on. In that case we keep the earlier result as the headline, show both runs on the model page and mark the setup “under review”. After the v0.22.5 upgrade, that applied to Nemotron-3-Nano-30B-A3B (25 → 28) and Hermes-4-70B (21 → 24), as well as gpt-oss-20b.
What happens when models get better
When most setups score 30/30, the tie rule keeps the board honest, but the score itself stops separating models. The answer then is a harder test: more tasks, longer contexts and more demanding tool use, so the ceiling sits above what today’s models reach. Until then, read the top group as “all of these work” and choose on speed, memory and the per-task results on each model page.
The full rule is on the methodology page.
Unofficial. Not affiliated with or endorsed by Nous Research. Hermes Agent is a trademark of its owner.