Menu

← Leaderboard

gpt-oss-120b with Hermes Agent

MXFP4 (official GGUF) on llama.cpp commit a37cc46. Tested

Task completion
100% (30/30)
Tool-call success
77.8%
Decode tok/s
53.6
Prefill tok/s
2470.4
Server memory
61.1 GB
Avg turns
4.9

Setup

Model
gpt-oss-120b
Family / maker
gpt-oss · OpenAI
Parameters
120B
Architecture
MoE
Weights
ggml-org/gpt-oss-120b-GGUF
Quant
MXFP4 (official GGUF)
Engine
llama.cpp commit a37cc46
Hermes Agent
v0.22.4
Tested
2026-10-23
Context
64K (65,536 tokens); long-context task t09 at 128K or 192K
Memory tier
≤128 GB (measured)

Tested in one evening run on 2026-10-23 with the server at 128K context for all 10 tasks (the long-context task t09 needs it). The other setups ran t01–t08 and t10 at 64K and only t09 at 128K/192K; the short tasks use far less than 64K, so this does not change their outcome. Server memory is measured at 64K like the others.

Per-task results

  • t01: 3/3
  • t02: 3/3
  • t03: 3/3
  • t04: 3/3
  • t05: 3/3
  • t06: 3/3
  • t07: 3/3
  • t08: 3/3
  • t09: 3/3
  • t10: 3/3

Copy-paste config

1. Start the model server (llama.cpp)

Exact command used in our test. Download the GGUF from ggml-org/gpt-oss-120b-GGUF into ./models/.

llama-server --model ./models/gpt-oss-120b-MXFP4.gguf --alias gpt-oss-120b --host 127.0.0.1 --port 18080 --ctx-size 131072 --parallel 1 --n-gpu-layers 999 --flash-attn on --jinja

2. Point Hermes Agent at it

hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:18080/v1
hermes config set model.default gpt-oss-120b

Other gpt-oss setups