News · Results
Hermes Agent v0.22.5 retest: how 12 local models changed after the upgrade
Published
On 24 October 2026 we upgraded our test install from Hermes Agent v0.22.4 to v0.22.5 and reran every earlier llama.cpp setup, so the whole board runs on one version. Most setups moved by two points or less, which is within normal run-to-run noise.
Every result on LocalAgentBoard records the Hermes Agent version it ran on. Results from 22–24 October used v0.22.4. After the upgrade we reran the same 10 tasks, three tries each, with the same models, quants, llama.cpp build and context rules. Setups added on or after 24 October were tested on v0.22.5 from the start.
Old vs new scores
| Setup | v0.22.4 | v0.22.5 | Published |
|---|---|---|---|
| Gemma 4 26B-A4B Q8_0 | 30/30 | 30/30 | v0.22.5 |
| Qwen3.6-27B Q6_K | 29/30 | 30/30 | v0.22.5 |
| Qwen3.6-35B-A3B UD-Q6_K | 29/30 | 29/30 | v0.22.5 |
| gpt-oss-120b MXFP4 | 30/30 | 30/30 | v0.22.5 |
| Nemotron-Cascade-2-30B-A3B Q4_K_M | 30/30 | 29/30 | v0.22.5 |
| GLM-4.7-Flash Q4_K_M | 28/30 | 27/30 | v0.22.5 |
| Qwopus-GLM-18B Q4_K_M | 27/30 | 26/30 | v0.22.5 |
| Hermes-4-14B Q4_K_M | 12/30 | 13/30 | v0.22.5 |
| Hermes-4.3-36B Q6_K | 21/30 | 22/30 | v0.22.5 |
| gpt-oss-20b MXFP4 | 27/30 | 30/30, 28/30 | v0.22.4, under review |
| Nemotron-3-Nano-30B-A3B Q8_0 | 25/30 | 28/30 | v0.22.4, under review |
| Hermes-4-70B Q4_K_M | 21/30 | 24/30 | v0.22.4, under review |
The Qwen3.6-35B-A3B UD-Q6_K row compares against its most recent v0.22.4 run (a night run on 24 October). It also has an earlier 30/30 run, listed on its model page.
The publishing rule
We decided the rule before the reruns: if the v0.22.5 score is within 2 points of the v0.22.4 score, the new run becomes the published result. If the gap is larger, the old result stays as the headline, both runs are shown, and the setup is marked “under review” until we have checked it. Old runs are never deleted; they stay listed on each model page.
What it means
- No sign that v0.22.5 changes model rankings. Nine of twelve setups landed within ±2 points of their earlier score, and the changes go both up and down. That is what normal sampling noise looks like (see why ±2 points is a tie).
- Three setups improved by 3 points. gpt-oss-20b, Nemotron-3-Nano and Hermes-4-70B all scored higher on v0.22.5. That could be noise at the edge of the band or a real improvement. They stay under review until further runs settle it.
- The top group stayed the top group. The same models still lead. With the new Best overall rule, the fastest of them wins.
Hermes-4.3-36B also got its first speed measurement in this batch: 7.1 tokens per second on the DGX Spark.
The full leaderboard is on the homepage, and the retest policy is on the methodology page.
Unofficial. Not affiliated with or endorsed by Nous Research. Hermes Agent is a trademark of its owner.