News · Results
Q4 vs Q8 for local agents: does quantization hurt Hermes Agent scores?
Published
Is a Q4 quant good enough for agent work, or do you need Q8? We ran three models at several quants through the same Hermes Agent tasks on a DGX Spark. Q4 scored within run-to-run noise of Q8 every time, with less memory and more speed.
Quantization shrinks a model’s weights so it fits in less memory and runs faster. The usual worry is that it hurts accuracy in exactly the ways agents care about: exact tool-call formats, file paths, multi-step plans. We tested three model families at more than one quant to see whether that shows up in finished tasks.
All runs: 10 Hermes Agent tasks × 3 tries, llama.cpp on an NVIDIA DGX Spark, 64K context. Memory is the model-server memory after loading. Speed is single-stream decode speed.
Results
| Model | Quant | Score | Speed (tok/s) | Memory |
|---|---|---|---|---|
| Gemma 4 26B-A4B | UD-Q4_K_M | 30/30 | 49.8 | 17.7 GB |
| Gemma 4 26B-A4B | Q8_0 | 30/30 | 47.0 | 26.9 GB |
| Qwen3.6-35B-A3B | UD-Q4_K_M | 29/30 | 67.0 | 21.8 GB |
| Qwen3.6-35B-A3B | UD-Q6_K | 29/30 | 60.2 | 28.4 GB |
| Qwen3.6-35B-A3B | Q8_0 | 30/30 | 58.0 | 35.5 GB |
| Nemotron-Cascade-2-30B-A3B | Q4_K_M | 29/30 | 71.5 | 23.6 GB |
| Nemotron-Cascade-2-30B-A3B | Q8_0 | 28/30 | 57.6 | 31.7 GB |
What we see
- Score: no measurable loss at Q4. Every Q4 result is within one point of the same model at Q8, and in one case Q4 scored higher. Runs of the same setup vary by about ±2 points, so none of these gaps is a real difference on this test.
- Memory: Q4 saves 8–14 GB. That is often the difference between fitting on a 24 GB GPU or not.
- Speed: Q4 is faster. Decode speed went up by about 6% for Gemma, 16% for Qwen and 24% for Nemotron-Cascade-2 compared with Q8.
Recommendation
For these mixture-of-experts models, start with a good Q4 quant (Q4_K_M or Unsloth’s UD-Q4_K_M) for agent work. Move up to Q6 or Q8 only if you have memory to spare and see a specific failure on your own tasks. This is one test set with 30 runs per setup, so it cannot rule out small differences, but it does not show any.
The Best for 16/24/128 GB page lists which setups fit which memory size, and the leaderboard shows every quant side by side.
Unofficial. Not affiliated with or endorsed by Nous Research. Hermes Agent is a trademark of its owner.