llm4funAI HARDWARE, EXPLAINED.
THE WORKBENCH
llm4fun / THE WORKBENCH

Put numbers to the test.

Measured performance helps ground the simulation. Explore the evidence or add your own.

SIMULATINGQwen3 8B
Q4 8K context 30 tok/s target

Theoretical vs measured

Calibrated errors hold out one measurement at a time. Other measurements from the same source or machine can remain, so this is not an independent validation of every model.

227 measurements cover 45 of 104 hardware profiles and 12 of 78 models. 163 start with no prior context. 5 include multiple GPUs.

Decode · theory only
±13%
median error · p90 ±34% · n=224
Decode · row held out
±6%
median error · p90 ±26% · n=224
Prefill · theory only
±30%
median error · p90 ±78% · n=193
Prefill · row held out
±8%
median error · p90 ±32% · n=193

Calibration fits separate corrections for dense and mixture-of-experts models, grouped by hardware family × backend, then by hardware profile. Multi-GPU groups also match GPU count, connection and split strategy. Corrections are shrunk toward 1 so sparse data has less influence. Efficiency priors live indata/calibration.json. Confidence also checks the selected model, precision, context and batch against measured coverage. Extrapolations stay LOW; ranges are estimates, not statistical guarantees.

What do PP512, TG128 and FP8 mean?

PP512 measures prompt processing for 512 input tokens. TG128 measures text generation over 128 output tokens. Both report tokens per second, but they time different work: PP512 does not describe the speed at which an answer appears. Neither alone measures the complete response time, and llama-bench excludes tokenization and sampling.

FP8 describes an 8-bit floating-point format, not a benchmark workload. Compare FP8 inference results only when the model, hardware, backend, context, batch and cache settings are comparable. A format supported by a model file does not guarantee a fast kernel on every GPU.

These measurements concern inference speed. For answer-quality evaluations such as MMLU-Pro or SWE-bench, use the model benchmark comparison. See how the performance estimates work for the calculation assumptions.

Test definitions: official llama-bench documentation. Individual rows retain their reported workloads and sources; not every row uses PP512 or TG128.

Fitted correction factors

measured ÷ theoretical, after shrinkage. 1.00 = theory matched reality.

Hardware family · backend · model typeDecode nDecode factorPrefill nPrefill factor
amd-rdna3 · llama.cpp-rocm · dense41.0741.19
amd-rdna3 · llama.cpp-vulkan · dense21.1821.44
amd-rdna4 · llama.cpp-rocm · dense20.9921.03
amd-rdna4 · llama.cpp-vulkan · dense21.2921.42
amd-strix-halo · llama.cpp-rocm · dense31.1130.85
amd-strix-halo · llama.cpp-rocm · moe41.2840.67
amd-strix-halo · llama.cpp-vulkan · dense31.1331.17
amd-strix-halo · llama.cpp-vulkan · moe41.3640.62
apple-m1 · llama.cpp-metal · dense90.9291.15
apple-m1 · llama.cpp-metal · moe11.0111.08
apple-m2 · llama.cpp-metal · dense181.05181.16
apple-m2 · llama.cpp-metal · moe220.6820.93
apple-m3 · llama.cpp-metal · dense121.06121.19
apple-m3 · llama.cpp-metal · moe10.8911.11
apple-m4 · llama.cpp-metal · dense91.1491.24
apple-m4 · llama.cpp-metal · moe11.0811.11
apple-m5 · llama.cpp-metal · dense151.28153.79
intel-xe2 · llama.cpp-vulkan · dense21.0521.59
nvidia-ada · llama.cpp-cuda · dense101.05101.14
nvidia-ada · llama.cpp-cuda · moe21.0320.64
nvidia-ada · llama.cpp-vulkan · dense11.0511.20
nvidia-ampere · llama.cpp-cuda · dense120.91121.03
nvidia-ampere · llama.cpp-cuda · moe10.8810.78
nvidia-ampere · llama.cpp-vulkan · dense11.0311.26
nvidia-blackwell · llama.cpp-cuda · dense141.03141.22
nvidia-blackwell · llama.cpp-cuda · moe100.8690.49
nvidia-blackwell · llama.cpp-vulkan · dense11.0111.18
nvidia-gb10 · llama.cpp-cuda · dense51.0352.88
nvidia-gb10 · llama.cpp-cuda · moe351.19352.27
nvidia-gb10 · llama.cpp-vulkan · dense21.0922.84
nvidia-hopper · llama.cpp-cuda · dense40.8440.46
nvidia-hopper · vllm · dense51.060—
nvidia-hopper · vllm · dense · 2gpu · nvlink · tensor51.140—
nvidia-pascal · llama.cpp-cuda · dense21.0021.78

Add a measured benchmark

Your own llama-bench / LM Studio numbers improve the calibration. Stored only in this browser.

Hardware
Model
Quantization
Backend
Decode (tg) tok/s
Prompt (pp512) tok/s
Context depth
tokens
GPU count
Source (required)
Source URL (optional)

Measured benchmarks (227)

Reported observations with sources. Rows marked chip proxy came from the same chip in a different or unspecified computer configuration.

HardwareModelQuantCtxBackendMeasured tgTheoryCalibrated*Measured ppTheory ppSource
Mac Studio M1 Max (32-core GPU) 32GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)23.032123(-2%)599.53512 tokens · depth 0470llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Max (32-core GPU) 32GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)40.23637(-7%)537.37512 tokens · depth 0470llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Max (32-core GPU) 32GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)61.195254(-12%)530.06512 tokens · depth 0470llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Ultra (48-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)33.924235(+2%)875.81512 tokens · depth 0700llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Ultra (48-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)55.696957(+3%)783.45512 tokens · depth 0700llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Ultra (48-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)74.9310085(+14%)772.24512 tokens · depth 0700llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Ultra (64-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)37.014237(-1%)1168.89512 tokens · depth 0930llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Ultra (64-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)59.876961(+2%)1042.95512 tokens · depth 0930llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M1 Ultra (64-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)83.7310090(+7%)1030.04512 tokens · depth 0930llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Max (38-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)24.652125(+0%)755.67512 tokens · depth 0600llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Max (38-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)41.833641(-2%)677.91512 tokens · depth 0600llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Max (38-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)65.955259(-11%)671.31512 tokens · depth 0600llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Ultra (60-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)39.864240(-1%)1128.59512 tokens · depth 0940llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Ultra (60-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)62.146967(+7%)1003.16512 tokens · depth 0940llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Ultra (60-core GPU) 64GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)88.6410097(+10%)1013.81512 tokens · depth 0940llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)41.024245(+9%)1401.85512 tokens · depth 01,200llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)66.646974(+11%)1248.59512 tokens · depth 01,200llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)94.27100110(+14%)1238.48512 tokens · depth 01,200llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro M3 Max (30-core GPU) 36GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)19.541620(+2%)589.41512 tokens · depth 0470llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro M3 Max (30-core GPU) 36GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)34.32733(-4%)566.4512 tokens · depth 0470llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro M3 Max (30-core GPU) 36GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)56.584046(-18%)567.59512 tokens · depth 0470llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro 16-inch M3 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)25.092125(0%)779.17512 tokens · depth 0620llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro 16-inch M3 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)42.753641(-3%)757.64512 tokens · depth 0620llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro 16-inch M3 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)66.315260(-10%)759.7512 tokens · depth 0620llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M3 Ultra (60-core GPU) 96GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)42.244340(-5%)1121.8512 tokens · depth 0930llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M3 Ultra (60-core GPU) 96GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)63.557169(+8%)1085.76512 tokens · depth 0930llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M3 Ultra (60-core GPU) 96GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)88.4100100(+14%)1073.09512 tokens · depth 0930llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)39.784341(+3%)1538.34512 tokens · depth 01,200llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)63.937168(+7%)1487.51512 tokens · depth 01,200llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)92.1410099(+8%)1471.24512 tokens · depth 01,200llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)17.181519(+9%)464.48512 tokens · depth 0360llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)30.692631(0%)449.62512 tokens · depth 0360llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)50.743843(-15%)439.78512 tokens · depth 0360llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)24.292327(+11%)736.25512 tokens · depth 0570llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)43.873844(0%)718.56512 tokens · depth 0570llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)69.955662(-11%)713.93512 tokens · depth 0570llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M4 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)31.643034(+7%)922.83512 tokens · depth 0710llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M4 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)54.055056(+3%)891.94512 tokens · depth 0710llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M4 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)83.067380(-4%)885.68512 tokens · depth 0710llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro 14-inch M5 (10-core GPU) 32GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)18.421420(+8%)715.42512 tokens · depth 0180llama.cpp discussion #4167 (Apple Silicon scoreboard)
MacBook Pro 14-inch M5 (10-core GPU) 32GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)31.882127(-15%)722.79512 tokens · depth 0180llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)21.391724(+13%)1310.78512 tokens · depth 0290llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)38.942939(+1%)1302.59512 tokens · depth 0290llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)67.594254(-20%)1340.44512 tokens · depth 0290llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)21.551724(+11%)1588.78512 tokens · depth 0370llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)38.922939(+0%)1553.86512 tokens · depth 0370llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)66.334255(-18%)1620.64512 tokens · depth 0370llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)37.113444(+19%)3158.49512 tokens · depth 0730llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00+128 generatedllama.cpp (Metal)72.425670(-3%)3143.81512 tokens · depth 0730llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00+128 generatedllama.cpp (Metal)119.928299(-18%)3219.99512 tokens · depth 0730llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M5 Ultra (80-core GPU) 256GBllama-2-7bF16 (mostly F16)0+128 generatedllama.cpp (Metal)73.66477(+5%)5207.14512 tokens · depth 01,500llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M5 Ultra (80-core GPU) 256GBllama-2-7bQ8_00+128 generatedllama.cpp (Metal)119.01110130(+7%)4910.55512 tokens · depth 01,500llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M5 Ultra (80-core GPU) 256GBllama-2-7bQ4_00+128 generatedllama.cpp (Metal)179.1150180(-1%)4944.66512 tokens · depth 01,500llama.cpp discussion #4167 (Apple Silicon scoreboard)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bF160llama.cpp (Metal)43.154245(+4%)1525.95512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00llama.cpp (Metal)73.117074(+1%)1368.18512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00llama.cpp (Metal)108.8100110(-1%)1391.78512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bF160llama.cpp (Metal)43.244245(+4%)1561.35512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00llama.cpp (Metal)73.357074(+1%)1386.97512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00llama.cpp (Metal)109.41100110(-2%)1412.42512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bF160llama.cpp (Metal)49.554244(-11%)1627.82512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ8_00llama.cpp (Metal)82.837073(-12%)1486.51512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedllama-2-7bQ4_00llama.cpp (Metal)125.21100110(-15%)1488.81512 tokens · depth 01,200llama.cpp discussion #4167 (M2 Ultra history table)
RTX 5090 32GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)290.02290310(+7%)14073.41512 tokens · depth 012,000llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX PRO 6000 Blackwell Workstation Edition 96GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)274.2290280(+3%)14854.63512 tokens · depth 014,000llama.cpp discussion #15013 (CUDA scoreboard, no FA)
H100 80GB SXM5llama-2-7bQ4_00+128 generatedllama.cpp (CUDA)267.81430370(+40%)9918.34512 tokens · depth 028,000llama.cpp discussion #15013 (CUDA scoreboard, no FA)
A100 80GB PCIellama-2-7bQ4_00+128 generatedllama.cpp (CUDA)190.88320240(+28%)4849.53512 tokens · depth 010,000llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 4090 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)186.21180190(+2%)11992.7512 tokens · depth 011,000llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 5080 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)181.99170180(-1%)8297.36512 tokens · depth 06,600llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 5070 Ti 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)176.85160170(-2%)6952.38512 tokens · depth 05,100llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 6000 Ada Generation 48GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)176.07180180(+3%)9229.23512 tokens · depth 012,000llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 3090 Ti 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)171.19180170(-1%)6567.49512 tokens · depth 05,300llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 3090 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)158.16170160(+0%)5174.69512 tokens · depth 04,700llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 4080 SUPER 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)148.33140150(-2%)8125.15512 tokens · depth 06,800llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX A6000 48GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)138.73140140(-1%)4913.93512 tokens · depth 05,100llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 4070 Ti SUPER 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)132.26130130(+1%)6924.53512 tokens · depth 05,800llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX A5000 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)130.07140130(+2%)4028.16512 tokens · depth 03,700llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 5060 Ti 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)90.948590(-1%)3737.25512 tokens · depth 02,800llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 3060 12GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)75.577170(-8%)2137.5512 tokens · depth 01,700llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 4060 Ti 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)63.865762(-3%)3394.63512 tokens · depth 02,900llama.cpp discussion #15013 (CUDA scoreboard, no FA)
NVIDIA DGX Spark Founders Edition 128GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)57.215355(-3%)3062.31512 tokens · depth 0910llama.cpp discussion #15013 (CUDA scoreboard, no FA)
Tesla P40 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)54.745454(-2%)1007.42512 tokens · depth 0440llama.cpp discussion #15013 (CUDA scoreboard, no FA)
RTX 5090 32GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)300.4290310(+2%)14970.15512 tokens · depth 012,000llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX PRO 6000 Blackwell Workstation Edition 96GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)281.11290280(0%)16618.98512 tokens · depth 014,000llama.cpp discussion #15013 (CUDA scoreboard, with FA)
H100 80GB SXM5llama-2-7bQ4_00+128 generatedllama.cpp (CUDA)280.74430370(+32%)11263.29512 tokens · depth 028,000llama.cpp discussion #15013 (CUDA scoreboard, with FA)
A100 80GB PCIellama-2-7bQ4_00+128 generatedllama.cpp (CUDA)200.9320240(+18%)5285.96512 tokens · depth 010,000llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 4090 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)188.96180190(+0%)14770.63512 tokens · depth 011,000llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 5080 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)184.68170180(-3%)9487.7512 tokens · depth 06,600llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 5070 Ti 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)182.43160170(-6%)8419.56512 tokens · depth 05,100llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 6000 Ada Generation 48GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)179.47180180(+0%)10576.85512 tokens · depth 012,000llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 3090 Ti 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)172.26180170(-2%)6924.01512 tokens · depth 05,300llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 3090 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)161.89170160(-3%)5560.06512 tokens · depth 04,700llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 4080 SUPER 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)147.48140150(-1%)9439.01512 tokens · depth 06,800llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX A6000 48GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)144.87140130(-8%)5662.39512 tokens · depth 05,100llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX A5000 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)135.83140130(-4%)4552.15512 tokens · depth 03,700llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 4070 Ti SUPER 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)132.85130130(0%)7612.32512 tokens · depth 05,800llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 5060 Ti 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)93.468589(-5%)4195.53512 tokens · depth 02,800llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 3060 12GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)76.927169(-10%)2407.67512 tokens · depth 01,700llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX 4060 Ti 16GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)64.035762(-4%)3803.45512 tokens · depth 02,900llama.cpp discussion #15013 (CUDA scoreboard, with FA)
NVIDIA DGX Spark Founders Edition 128GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)56.745356(-2%)3661.37512 tokens · depth 0910llama.cpp discussion #15013 (CUDA scoreboard, with FA)
Tesla P40 24GBllama-2-7bQ4_00+128 generatedllama.cpp (CUDA)53.735455(+2%)1079.66512 tokens · depth 0440llama.cpp discussion #15013 (CUDA scoreboard, with FA)
RTX PRO 6000 Blackwell Workstation Edition 96GBllama-2-7bQ4_00llama.cpp (CUDA)260.42290290(+11%)12742.48512 tokens · depth 014,000llama.cpp discussion #15013 (comment)
RTX PRO 6000 Blackwell Workstation Edition 96GBllama-2-7bQ4_00llama.cpp (CUDA)289.3290280(-3%)14463.16512 tokens · depth 014,000llama.cpp discussion #15013 (comment)
Radeon RX 7900 XTX 24GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)182.63140180(-2%)3726.99512 tokens · depth 02,200llama.cpp discussion #10879 (Vulkan scoreboard, no FA)
Radeon AI PRO R9700 32GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)145.67100140(-5%)5609.82512 tokens · depth 03,400llama.cpp discussion #10879 (Vulkan scoreboard, no FA)
Arc Pro B60 24GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)68.556469(+1%)522.36512 tokens · depth 0190llama.cpp discussion #10879 (Vulkan scoreboard, no FA)
NVIDIA DGX Spark Founders Edition 128GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)58.515258(-1%)2789.51512 tokens · depth 0640llama.cpp discussion #10879 (Vulkan scoreboard, no FA)
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)53.594653(0%)1288.96512 tokens · depth 01,100llama.cpp discussion #10879 (Vulkan scoreboard, no FA)
RTX 5090 32GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)273.68270270(-2%)11796.38512 tokens · depth 08,500llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
Radeon RX 7900 XTX 24GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)190.92140170(-10%)3889.54512 tokens · depth 02,200llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
RTX 4090 24GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)190.1170170(-9%)10830.41512 tokens · depth 07,500llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
RTX 3090 24GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)171.61160160(-5%)5200.68512 tokens · depth 03,300llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
Radeon AI PRO R9700 32GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)152.7100130(-13%)5907.66512 tokens · depth 03,400llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
Arc Pro B60 24GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)70.546467(-4%)274.76512 tokens · depth 0190llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
NVIDIA DGX Spark Founders Edition 128GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)60.395257(-6%)3321.91512 tokens · depth 0640llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBllama-2-7bQ4_00+128 generatedllama.cpp (Vulkan)534654(+1%)1357.07512 tokens · depth 01,100llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled)
Radeon RX 7900 XTX 24GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)167.11140160(-6%)3552.27512 tokens · depth 02,800llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA)
Radeon PRO W7900 48GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)121.18130130(+10%)3213.17512 tokens · depth 02,800llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA)
Radeon AI PRO R9700 32GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)93.849798(+4%)4443.54512 tokens · depth 04,400llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA)
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)50.014449(-1%)911.36512 tokens · depth 01,400llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA)
Radeon RX 7900 XTX 24GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)170.12140160(-9%)3874.25512 tokens · depth 02,800llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA)
Radeon PRO W7900 48GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)127.43130130(+1%)3472.86512 tokens · depth 02,800llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA)
Radeon AI PRO R9700 32GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)97.989795(-4%)4773.07512 tokens · depth 04,400llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA)
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBllama-2-7bQ4_00+128 generatedllama.cpp (ROCm)49.874450(-1%)1003.53512 tokens · depth 01,400llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bgpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF)0+32 generatedllama.cpp (CUDA)83.436479(-6%)4505.822048 tokens · depth 0 · microbatch 20481,700llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bgpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF)4,096+32 generatedllama.cpp (CUDA)79.226276(-5%)4158.342048 tokens · depth 4096 · microbatch 20481,500llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bgpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF)8,192+32 generatedllama.cpp (CUDA)75.226073(-3%)3993.812048 tokens · depth 8192 · microbatch 20481,400llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bgpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF)16,384+32 generatedllama.cpp (CUDA)70.365668(-3%)3449.982048 tokens · depth 16384 · microbatch 20481,200llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bgpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF)32,768+32 generatedllama.cpp (CUDA)61.654960(-3%)2689.422048 tokens · depth 32768 · microbatch 2048890llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bgpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF)0+32 generatedllama.cpp (CUDA)58.724352(-11%)2443.912048 tokens · depth 0 · microbatch 20481,200llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bgpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF)4,096+32 generatedllama.cpp (CUDA)55.674150(-10%)2309.842048 tokens · depth 4096 · microbatch 20481,100llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bgpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF)8,192+32 generatedllama.cpp (CUDA)52.874049(-8%)2216.682048 tokens · depth 8192 · microbatch 2048960llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bgpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF)16,384+32 generatedllama.cpp (CUDA)49.453745(-8%)1956.312048 tokens · depth 16384 · microbatch 2048810llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bgpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF)32,768+32 generatedllama.cpp (CUDA)42.763340(-7%)1567.082048 tokens · depth 32768 · microbatch 2048610llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBqwen3-coder-30b-a3bqwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF)0+32 generatedllama.cpp (CUDA)61.065365(+6%)2986.972048 tokens · depth 0 · microbatch 20481,700llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBqwen3-coder-30b-a3bqwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF)4,096+32 generatedllama.cpp (CUDA)54.774758(+5%)2633.452048 tokens · depth 4096 · microbatch 20481,200llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBqwen3-coder-30b-a3bqwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF)8,192+32 generatedllama.cpp (CUDA)48.024252(+8%)2354.142048 tokens · depth 8192 · microbatch 2048900llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBqwen3-coder-30b-a3bqwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF)16,384+32 generatedllama.cpp (CUDA)40.233644(+8%)1908.862048 tokens · depth 16384 · microbatch 2048610llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBqwen3-coder-30b-a3bqwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF)32,768+32 generatedllama.cpp (CUDA)30.212733(+9%)1348.172048 tokens · depth 32768 · microbatch 2048370llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bMXFP4 (ggml-org GGUF)4,096+32 generatedllama.cpp (CUDA)55.214150(-9%)2412.654096 tokens · depth 0 · microbatch 20481,100llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bMXFP4 (ggml-org GGUF)4,096+32 generatedllama.cpp (CUDA)139.07aggregate · batch 87894(-33%)2413.974096 tokens · depth 0 · microbatch 20481,100llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-120bMXFP4 (ggml-org GGUF)4,096+32 generatedllama.cpp (CUDA)262.2aggregate · batch 32110140(-48%)2411.234096 tokens · depth 0 · microbatch 20481,100llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bMXFP4 (ggml-org GGUF)4,096+32 generatedllama.cpp (CUDA)78.546276(-4%)4513.914096 tokens · depth 0 · microbatch 20481,600llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bMXFP4 (ggml-org GGUF)4,096+32 generatedllama.cpp (CUDA)233.11aggregate · batch 8140170(-28%)4561.444096 tokens · depth 0 · microbatch 20481,600llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench)
NVIDIA DGX Spark Founders Edition 128GBgpt-oss-20bMXFP4 (ggml-org GGUF)4,096+32 generatedllama.cpp (CUDA)473.59aggregate · batch 32340410(-13%)4557.524096 tokens · depth 0 · microbatch 20481,600llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench)
Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (Metal)115.52150150(+26%)2816.472048 tokens · depth 0 · microbatch 20482,300llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (Metal)116.0814095(-18%)2191.132048 tokens · depth 0 · microbatch 20482,200llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-120bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (Metal)79.689563(-21%)1244.572048 tokens · depth 0 · microbatch 20481,500llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (Metal)92.367979(-14%)1277.422048 tokens · depth 0 · microbatch 20481,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
Mac Studio M1 Max (32-core GPU) 64GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (Metal)75.157474(-1%)994.752048 tokens · depth 0 · microbatch 2048860llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX PRO 6000 Blackwell Max-Q Workstation Edition 96GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)249.96360280(+14%)9480.552048 tokens · depth 0 · microbatch 204824,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX PRO 6000 Blackwell Workstation Edition 96GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)286.91360300(+5%)11521.952048 tokens · depth 0 · microbatch 204827,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX PRO 6000 Blackwell Max-Q Workstation Edition 96GBgpt-oss-120bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)170.62240190(+9%)4494.22048 tokens · depth 0 · microbatch 204817,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX PRO 6000 Blackwell Workstation Edition 96GBgpt-oss-120bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)196.31240200(+1%)5518.072048 tokens · depth 0 · microbatch 204819,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX 3090 24GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)161.77210210(+30%)5170.562048 tokens · depth 0 · microbatch 20488,600llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX 4090 24GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)221.95220240(+6%)8022.332048 tokens · depth 0 · microbatch 204820,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX 4080 SUPER 16GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)186.51170170(-10%)8170.952048 tokens · depth 0 · microbatch 204813,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX 5060 Ti 16GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)111.5110087(-22%)3839.212048 tokens · depth 0 · microbatch 20485,100llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX 5070 Ti 16GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)189.45200170(-12%)6339.762048 tokens · depth 0 · microbatch 20489,400llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX 5080 16GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)204.85210180(-13%)7476.552048 tokens · depth 0 · microbatch 204812,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
RTX 5090 32GBgpt-oss-20bMXFP4 (ggml-org gpt-oss GGUF)0+128 generatedllama.cpp (CUDA)282.51360310(+8%)9848.382048 tokens · depth 0 · microbatch 204822,000llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp)
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-120bgpt-oss-120b-mxfp40+128 generatedllama.cpp (ROCm)51.813646(-10%)625.58512 tokens · depth 01,500kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-120bgpt-oss-120b-mxfp432,768+32 generatedllama.cpp (ROCm)36.322736(+0%)589.252048 tokens · depth 32768930kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-120bgpt-oss-120b-mxfp40+128 generatedllama.cpp (Vulkan)56.613753(-6%)719.91512 tokens · depth 01,400kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-120bgpt-oss-120b-mxfp432,768+32 generatedllama.cpp (Vulkan)43.012941(-6%)307.52048 tokens · depth 32768730kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-20bgpt-oss-20b-mxfp40+128 generatedllama.cpp (ROCm)73.055371(-3%)1786.17512 tokens · depth 02,600kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-20bgpt-oss-20b-mxfp432,768+32 generatedllama.cpp (ROCm)52.174155(+6%)1003.112048 tokens · depth 327681,400kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-20bgpt-oss-20b-mxfp40+128 generatedllama.cpp (Vulkan)79.785681(+2%)1692.49512 tokens · depth 02,000kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBgpt-oss-20bgpt-oss-20b-mxfp432,768+32 generatedllama.cpp (Vulkan)61.124362(+1%)555.732048 tokens · depth 327681,100kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBllama-2-7bllama-2-7b.Q4_00+128 generatedllama.cpp (ROCm)50.594449(-3%)1545.36512 tokens · depth 01,400kyuz0 amd-strix-halo-toolboxes benchmark results.json
AMD Ryzen AI Max+ 395 mini PC (generic) 128GBllama-2-7bllama-2-7b.Q4_00+128 generatedllama.cpp (Vulkan)55.734652(-6%)1337.7512 tokens · depth 01,100kyuz0 amd-strix-halo-toolboxes benchmark results.json
NVIDIA DGX Spark Founders Edition 128GBglm-4.7-flashQ8_00+32 generatedllama.cpp (CUDA)48.685873(+49%)2364.182048 tokens · depth 0 · microbatch 20481,500llama.cpp official DGX Spark benchmark report
NVIDIA DGX Spark Founders Edition 128GBglm-4.7-flashQ8_04,096+32 generatedllama.cpp (CUDA)44.625468(+52%)1684.132048 tokens · depth 4096 · microbatch 2048750llama.cpp official DGX Spark benchmark report
NVIDIA DGX Spark Founders Edition 128GBglm-4.7-flashQ8_08,192+32 generatedllama.cpp (CUDA)42.595163(+49%)1314.682048 tokens · depth 8192 · microbatch 2048500llama.cpp official DGX Spark benchmark report
NVIDIA DGX Spark Founders Edition 128GBglm-4.7-flashQ8_016,384+32 generatedllama.cpp (CUDA)38.724556(+45%)914.052048 tokens · depth 16384 · microbatch 2048300llama.cpp official DGX Spark benchmark report
NVIDIA DGX Spark Founders Edition 128GBglm-4.7-flashQ8_032,768+32 generatedllama.cpp (CUDA)32.653746(+40%)567.22048 tokens · depth 32768 · microbatch 2048170llama.cpp official DGX Spark benchmark report
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M0+128 generatedllama.cpp (CUDA)69.857181(+16%)2657.12512 tokens · depth 0 · microbatch 20481,600shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M8,192+128 generatedllama.cpp (CUDA)59.566271(+18%)2361.51512 tokens · depth 8192 · microbatch 20481,300shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M32,768+128 generatedllama.cpp (CUDA)52.065259(+14%)1876.73512 tokens · depth 32768 · microbatch 2048920shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-31bQ4_K_M0+128 generatedllama.cpp (CUDA)10.981111(-1%)743.03512 tokens · depth 0 · microbatch 2048200shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-31bQ4_K_M8,192+128 generatedllama.cpp (CUDA)10.029.910(+2%)620.04512 tokens · depth 8192 · microbatch 2048180shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-31bQ4_K_M32,768+128 generatedllama.cpp (CUDA)9.1599.2(+1%)482.44512 tokens · depth 32768 · microbatch 2048150shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M512+128 generatedllama.cpp (CUDA)67.466877(+14%)2583.05512 tokens · depth 0 · microbatch 20481,600shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M512+128 generatedllama.cpp (CUDA)107.28aggregate · batch 299110(+4%)2935.23512 tokens · depth 0 · microbatch 20481,600shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M512+128 generatedllama.cpp (CUDA)154.87aggregate · batch 4130150(-5%)2981.25512 tokens · depth 0 · microbatch 20481,600shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M512+128 generatedllama.cpp (CUDA)211.26aggregate · batch 8170190(-12%)2998.59512 tokens · depth 0 · microbatch 20481,600shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M512+128 generatedllama.cpp (CUDA)297.08aggregate · batch 16220240(-19%)2997.52512 tokens · depth 0 · microbatch 20481,600shamily Gemma 4 GB10 measurements (MIT)
ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecifiedgemma-4-26b-a4bQ4_K_M512+128 generatedllama.cpp (CUDA)409.42aggregate · batch 32310340(-17%)3022.3512 tokens · depth 0 · microbatch 20481,600shamily Gemma 4 GB10 measurements (MIT)
RTX 5090 32GBqwen3-8bQ8_00+128 generatedllama.cpp (CUDA)160.1150160(-2%)——kekzl/imp measured llama.cpp baseline (MIT)
RTX 5090 32GBqwen3-8bQ8_00llama.cpp (CUDA)———13724512 tokens · depth 010,000kekzl/imp measured llama.cpp baseline (MIT)
RTX 5090 32GBqwen3-14bQ6_K0+128 generatedllama.cpp (CUDA)114.8100110(-8%)——kekzl/imp measured llama.cpp baseline (MIT)
RTX 5090 32GBqwen3-14bQ6_K0llama.cpp (CUDA)———6522512 tokens · depth 05,600kekzl/imp measured llama.cpp baseline (MIT)
RTX 5090 32GBqwen3-30b-a3bQ4_K_M0+128 generatedllama.cpp (CUDA)295.7390330(+13%)——kekzl/imp measured llama.cpp baseline (MIT)
RTX 5090 32GBqwen3-30b-a3bQ4_K_M0llama.cpp (CUDA)———9288512 tokens · depth 024,000kekzl/imp measured llama.cpp baseline (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP40+32 generatedllama.cpp (Metal)129.9714094(-28%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP44,096+32 generatedllama.cpp (Metal)123.3814091(-26%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP48,192+32 generatedllama.cpp (Metal)117.3913088(-25%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP416,384+32 generatedllama.cpp (Metal)109.7512082(-25%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-20bMXFP432,768+32 generatedllama.cpp (Metal)98.2511073(-26%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-120bMXFP40+32 generatedllama.cpp (Metal)85.69563(-27%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-120bMXFP44,096+32 generatedllama.cpp (Metal)82.039261(-26%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-120bMXFP48,192+32 generatedllama.cpp (Metal)78.238859(-25%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-120bMXFP416,384+32 generatedllama.cpp (Metal)73.28355(-25%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedgpt-oss-120bMXFP432,768+32 generatedllama.cpp (Metal)65.527349(-26%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedqwen3-coder-30b-a3bQ8_00+32 generatedllama.cpp (Metal)78.9712080(+1%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedqwen3-coder-30b-a3bQ8_04,096+32 generatedllama.cpp (Metal)71.1811072(+1%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedqwen3-coder-30b-a3bQ8_08,192+32 generatedllama.cpp (Metal)65.119866(+1%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedqwen3-coder-30b-a3bQ8_016,384+32 generatedllama.cpp (Metal)56.878356(-2%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedqwen3-coder-30b-a3bQ8_032,768+32 generatedllama.cpp (Metal)45.266443(-6%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedglm-4.7-flashQ8_00+32 generatedllama.cpp (Metal)59.5813089(+50%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedglm-4.7-flashQ8_04,096+32 generatedllama.cpp (Metal)47.4412084(+78%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedglm-4.7-flashQ8_08,192+32 generatedllama.cpp (Metal)40.212080(+99%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedglm-4.7-flashQ8_016,384+32 generatedllama.cpp (Metal)31.510072(+129%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecifiedglm-4.7-flashQ8_032,768+32 generatedllama.cpp (Metal)21.818660(+176%)——llama.cpp official Mac M2 Ultra benchmark report (MIT)
RTX 5090 32GBgpt-oss-20bMXFP40+128 generatedllama.cpp (CUDA)335.9360290(-14%)——kekzl/imp measured llama.cpp baseline (MIT)
H100 80GB SXM5llama-3.1-8bQ8_0 GGUF921+255 generatedllama.cpp (CUDA)214.21230180(-15%)10782.65894 tokens · depth 2723,000Winder.AI H100 inference benchmark: first request after 20 warmups
H100 80GB SXM5qwen3.8-27bQ8_0 GGUF903+255 generatedllama.cpp (CUDA)67.226145(-33%)2151.76903 tokens · depth 07,000Winder.AI H100 inference benchmark: first request after 20 warmups
H100 80GB SXM5qwen3-8bBF16 (unquantized Qwen3-8B)508+511 generatedvLLM150.68345140150(+1%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5qwen3-8bBF16 (unquantized Qwen3-8B)2,008+511 generatedvLLM148.722140150(+1%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5qwen3-8bBF16 (unquantized Qwen3-8B)8,008+511 generatedvLLM142.1103130140(0%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5qwen3-8bBF16 (unquantized Qwen3-8B)12,008+511 generatedvLLM138.3221130140(-1%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5qwen3-8bBF16 (unquantized Qwen3-8B)20,008+511 generatedvLLM131.37235120130(-2%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5 ×2qwen3-8bBF16 (unquantized Qwen3-8B)508+511 generatedvLLM223.6028190220(0%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5 ×2qwen3-8bBF16 (unquantized Qwen3-8B)2,008+511 generatedvLLM220.6257190220(+0%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5 ×2qwen3-8bBF16 (unquantized Qwen3-8B)8,008+511 generatedvLLM212.7753180210(0%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5 ×2qwen3-8bBF16 (unquantized Qwen3-8B)12,008+511 generatedvLLM208.18245180210(-1%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
H100 80GB SXM5 ×2qwen3-8bBF16 (unquantized Qwen3-8B)20,008+511 generatedvLLM199.4581170200(-2%)——cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds
Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecifiedqwen3-8bQ4_K_M (Qwen official GGUF)0+256 generatedllama.cpp (Metal)98.25379284110(+9%)2778.057184512 tokens · depth 0 · microbatch 512600Nekko Labs · llama-bench on MacBook Pro M5 Max

* Leave-one-out: calibrated prediction made without this row. Skipped rows (unknown model/quant) are not used for calibration.

Recorded generation length, initial prompt depth and microbatch size are replayed when available. Missing KV precision uses an F16 assumption. New measurement source license notices: llama.cpp, shamily, kekzl/imp.