News · Press release
LocalAgentBoard launches: a tested leaderboard of local models for Hermes Agent
Published
LocalAgentBoard is a new, independent leaderboard that answers one practical question: which local model, quant and engine should you run Hermes Agent on? Every result comes from real agent runs on an NVIDIA DGX Spark, with the exact config published.
Oslo, 25 October 2026. LocalAgentBoard (localagentboard.com) is now public. It is a leaderboard of open-weight models for people who run AI agents on their own hardware, and it is built on hands-on tests rather than general chat benchmarks.
Most model leaderboards measure knowledge or chat quality. People running a local agent need to know something else: does the model call tools correctly, edit the right files and finish the job inside the agent they actually use? LocalAgentBoard tests exactly that, inside Hermes Agent, and publishes the server and agent settings that produced each result.
What is tested
Each setup is one combination of model, quantization and engine, for example a Q4_K_M GGUF on llama.cpp. Every setup runs the same 10 offline tasks inside the real Hermes Agent, three times each, for a score out of 30. The tasks cover instruction following, single and chained tool calls, file edits across several files, a debugging loop, data transforms, a needle-in-a-long-log task of roughly 86K–130K tokens, and respecting a protected file.
A run only passes if an automatic checker finds the correct result on disk. What the agent says it did does not count. The board also records tool-call success, decode and prefill speed, model-server memory and average turns per task. The full method is on the methodology page.
First results
As of 25 October 2026, 16 of 22 planned setups have been tested on an NVIDIA DGX Spark (GB10, 128 GB unified memory) with llama.cpp. They cover the Qwen3.6, Gemma 4, Nemotron, gpt-oss, Hermes 4 and GLM families, several of them at more than one quant.
- Best overall: Nemotron-Cascade-2-30B-A3B Q4_K_M, 29/30 at 71.5 tokens per second, the fastest setup in the top group.
- Fastest reliable: gpt-oss-20b MXFP4, 27/30 at 79.4 tokens per second.
- Best for small machines: Gemma 4 26B-A4B UD-Q4_K_M, 30/30 with 17.7 GB of model-server memory.
Nine setups score within two points of the top score. Reruns of the same setup move by about ±2 points, so the board treats those nine as tied and picks the overall winner by speed. Why ±2 points is a tie explains the rule.
The live picks on the leaderboard are recomputed from the data every time results change, so they can differ from the snapshot above.
Next
Six larger setups need vLLM with NVFP4 or FP8 weights: Mistral Small 4 119B, Nemotron-3-Super 120B, Qwen3.5-122B-A10B, Qwen3-Coder-Next, Gemma 4 31B and an NVFP4 build of Qwen3.6-35B-A3B. They are listed as “Not tested yet” on the board until they have been run. LocalAgentBoard retests when a new Hermes Agent release, engine version or notable model comes out, and keeps old results with their dates and versions.
Independence
LocalAgentBoard is unofficial and is not affiliated with or endorsed by Nous Research. “Hermes Agent” is used only to describe the software being tested, and it is a trademark of its owner. Sponsors and referral partners never see results early and have no influence on rankings. More on the about page.
Unofficial. Not affiliated with or endorsed by Nous Research. Hermes Agent is a trademark of its owner.