Put numbers to the test.
Measured performance helps ground the simulation. Explore the evidence or add your own.
SIMULATINGQwen3 8B
Q4 8K context 30 tok/s targetTheoretical vs measured
How well does the estimator predict real benchmarks? Calibrated errors are leave-one-out: each benchmark is predicted without using itself.
Decode · theory only
±13%
median error · p90 ±31% · n=173
Decode · calibrated
±6%
median error · p90 ±18% · n=173
Prefill · theory only
±24%
median error · p90 ±78% · n=173
Prefill · calibrated
±11%
median error · p90 ±48% · n=173
Calibration fits a correction factor per architecture family × backend, then per exact hardware, shrunk toward 1 so few data points cannot dominate. Efficiency priors per architecture, backend, quantization and model type live indata/calibration.json. Hardware without benchmarks inherits its family's factor (MEDIUM confidence) or stays purely theoretical (LOW).
Fitted correction factors
measured ÷ theoretical, after shrinkage. 1.00 = theory matched reality.
| Architecture family | backend | Decode n | Decode factor | Prefill n | Prefill factor |
|---|---|---|---|---|
| amd-rdna3 · llama.cpp-rocm | 4 | 1.07 | 4 | 1.19 |
| amd-rdna3 · llama.cpp-vulkan | 2 | 1.18 | 2 | 1.44 |
| amd-rdna4 · llama.cpp-rocm | 2 | 0.99 | 2 | 1.03 |
| amd-rdna4 · llama.cpp-vulkan | 2 | 1.29 | 2 | 1.41 |
| amd-strix-halo · llama.cpp-rocm | 7 | 1.22 | 7 | 0.72 |
| amd-strix-halo · llama.cpp-vulkan | 7 | 1.28 | 7 | 0.80 |
| apple-m1 · llama.cpp-metal | 10 | 0.92 | 10 | 1.15 |
| apple-m2 · llama.cpp-metal | 20 | 1.03 | 20 | 1.13 |
| apple-m3 · llama.cpp-metal | 13 | 1.04 | 13 | 1.19 |
| apple-m4 · llama.cpp-metal | 13 | 1.15 | 13 | 1.19 |
| apple-m5 · llama.cpp-metal | 14 | 1.28 | 14 | 3.73 |
| intel-xe2 · llama.cpp-vulkan | 2 | 1.05 | 2 | 1.59 |
| nvidia-ada · llama.cpp-cuda | 12 | 1.04 | 12 | 1.00 |
| nvidia-ada · llama.cpp-vulkan | 1 | 1.05 | 1 | 1.19 |
| nvidia-ampere · llama.cpp-cuda | 13 | 0.89 | 13 | 0.98 |
| nvidia-ampere · llama.cpp-vulkan | 1 | 1.02 | 1 | 1.26 |
| nvidia-blackwell · llama.cpp-cuda | 20 | 0.95 | 20 | 0.86 |
| nvidia-blackwell · llama.cpp-vulkan | 1 | 1.01 | 1 | 1.17 |
| nvidia-gb10 · llama.cpp-cuda | 23 | 1.30 | 23 | 2.58 |
| nvidia-gb10 · llama.cpp-vulkan | 2 | 1.09 | 2 | 2.84 |
| nvidia-hopper · llama.cpp-cuda | 2 | 0.74 | 2 | 0.51 |
| nvidia-pascal · llama.cpp-cuda | 2 | 0.99 | 2 | 1.78 |
Add a measured benchmark
Your own llama-bench / LM Studio numbers improve the calibration. Stored only in this browser.
Hardware
Model
Quantization
Backend
Decode (tg) tok/s
Prompt (pp512) tok/s
Context depth
tokens
GPU count
Source (required)
Source URL (optional)
Measured benchmarks (173)
Real observations with sources. Never mixed with estimates.
| Hardware | Model | Quant | Ctx | Backend | Measured tg | Theory | Calibrated* | Measured pp | Theory pp | Source |
|---|---|---|---|---|---|---|---|---|---|---|
| Mac Studio M1 Max (32-core GPU) 32GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 23.03 | 21 | 23(-2%) | 599.53 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Max (32-core GPU) 32GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 40.2 | 36 | 37(-7%) | 537.37 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Max (32-core GPU) 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 61.19 | 53 | 54(-11%) | 530.06 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (48-core GPU) 64GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 33.92 | 42 | 35(+2%) | 875.81 | 700 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (48-core GPU) 64GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 55.69 | 70 | 57(+3%) | 783.45 | 700 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (48-core GPU) 64GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 74.93 | 100 | 86(+15%) | 772.24 | 700 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (64-core GPU) 128GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 37.01 | 42 | 37(-1%) | 1168.89 | 940 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (64-core GPU) 128GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 59.87 | 70 | 61(+2%) | 1042.95 | 940 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (64-core GPU) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 83.73 | 100 | 90(+8%) | 1030.04 | 940 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Max (38-core GPU) 64GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 24.65 | 21 | 24(-1%) | 755.67 | 600 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Max (38-core GPU) 64GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 41.83 | 36 | 41(-3%) | 677.91 | 600 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Max (38-core GPU) 64GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 65.95 | 53 | 59(-11%) | 671.31 | 600 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (60-core GPU) 64GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 39.86 | 42 | 39(-2%) | 1128.59 | 950 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (60-core GPU) 64GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 62.14 | 70 | 66(+6%) | 1003.16 | 950 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (60-core GPU) 64GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 88.64 | 100 | 97(+9%) | 1013.81 | 950 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 41.02 | 42 | 43(+5%) | 1401.85 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 66.64 | 70 | 72(+8%) | 1248.59 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 94.27 | 100 | 100(+11%) | 1238.48 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro M3 Max (30-core GPU) 36GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 19.54 | 16 | 20(+1%) | 589.41 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro M3 Max (30-core GPU) 36GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 34.3 | 27 | 33(-5%) | 566.4 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro M3 Max (30-core GPU) 36GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 56.58 | 40 | 46(-18%) | 567.59 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 16-inch M3 Max (40-core GPU) 128GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 25.09 | 21 | 25(-1%) | 779.17 | 620 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 16-inch M3 Max (40-core GPU) 128GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 42.75 | 36 | 41(-4%) | 757.64 | 620 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 16-inch M3 Max (40-core GPU) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 66.31 | 53 | 59(-10%) | 759.7 | 620 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (60-core GPU) 96GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 42.24 | 43 | 40(-6%) | 1121.8 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (60-core GPU) 96GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 63.55 | 71 | 68(+7%) | 1085.76 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (60-core GPU) 96GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 88.4 | 100 | 100(+14%) | 1073.09 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (80-core GPU) 512GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 39.78 | 43 | 39(-2%) | 1538.34 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (80-core GPU) 512GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 63.93 | 71 | 65(+1%) | 1487.51 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (80-core GPU) 512GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 92.14 | 100 | 94(+2%) | 1471.24 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 17.19 | 15 | 19(+8%) | 381.14 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 30.54 | 26 | 31(+1%) | 367.13 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 49.64 | 38 | 45(-10%) | 364.06 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 17.18 | 15 | 19(+8%) | 464.48 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 30.69 | 26 | 31(+1%) | 449.62 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 50.74 | 38 | 45(-12%) | 439.78 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (32-core GPU) 36GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 24.29 | 23 | 27(+11%) | 736.25 | 570 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (32-core GPU) 36GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 43.87 | 38 | 44(+0%) | 718.56 | 570 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (32-core GPU) 36GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 69.95 | 56 | 63(-9%) | 713.93 | 570 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (40-core GPU) 128GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 31.64 | 30 | 34(+7%) | 922.83 | 710 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (40-core GPU) 128GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 54.05 | 50 | 56(+3%) | 891.94 | 710 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (40-core GPU) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 83.06 | 74 | 80(-3%) | 885.68 | 710 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 14-inch M5 (10-core GPU) 32GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 18.42 | 15 | 20(+9%) | 715.42 | 180 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 14-inch M5 (10-core GPU) 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 31.88 | 22 | 27(-14%) | 722.79 | 180 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 21.39 | 17 | 24(+13%) | 1310.78 | 290 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 38.94 | 29 | 39(+1%) | 1302.59 | 290 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 67.59 | 43 | 55(-19%) | 1340.44 | 290 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 21.55 | 17 | 24(+11%) | 1588.78 | 370 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 38.92 | 29 | 39(+0%) | 1553.86 | 370 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 66.33 | 43 | 55(-17%) | 1620.64 | 370 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Max (40-core GPU) 128GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 37.11 | 34 | 46(+23%) | 3158.49 | 730 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Max (40-core GPU) 128GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 72.42 | 56 | 71(-1%) | 3143.81 | 730 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Max (40-core GPU) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 119.92 | 82 | 100(-17%) | 3219.99 | 730 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Ultra (80-core GPU) 256GB | llama-2-7b | F16 (mostly F16) | 0 | llama.cpp (Metal) | 73.6 | 65 | 77(+5%) | 5207.14 | 1,500 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Ultra (80-core GPU) 256GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 119.01 | 110 | 130(+7%) | 4910.55 | 1,500 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Ultra (80-core GPU) 256GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 179.1 | 150 | 180(+0%) | 4944.66 | 1,500 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | F16 | 0 | llama.cpp (Metal) | 43.15 | 42 | 43(0%) | 1525.95 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 73.11 | 70 | 71(-3%) | 1368.18 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 108.8 | 100 | 100(-5%) | 1391.78 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | F16 | 0 | llama.cpp (Metal) | 43.24 | 42 | 43(0%) | 1561.35 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 73.35 | 70 | 71(-3%) | 1386.97 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 109.41 | 100 | 100(-6%) | 1412.42 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | F16 | 0 | llama.cpp (Metal) | 49.55 | 42 | 43(-14%) | 1627.82 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 82.83 | 70 | 71(-15%) | 1486.51 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 125.21 | 100 | 100(-18%) | 1488.81 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| RTX 5090 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 290.02 | 290 | 270(-8%) | 14073.41 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 274.2 | 290 | 250(-10%) | 14854.63 | 15,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| H100 80GB SXM5 | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 267.81 | 430 | 310(+17%) | 9918.34 | 29,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| A100 80GB PCIe | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 190.88 | 320 | 240(+27%) | 4849.53 | 10,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4090 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 186.21 | 180 | 190(+1%) | 11992.7 | 11,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5080 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 181.99 | 170 | 170(-6%) | 8297.36 | 6,600 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5070 Ti 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 176.85 | 160 | 160(-8%) | 6952.38 | 5,200 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 6000 Ada Generation 48GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 176.07 | 180 | 180(+3%) | 9229.23 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 3090 Ti 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 171.19 | 180 | 170(-2%) | 6567.49 | 5,300 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 3090 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 158.16 | 170 | 150(-6%) | 5174.69 | 4,700 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4080 SUPER 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 148.33 | 140 | 150(+0%) | 8125.15 | 6,900 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX A6000 48GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 138.73 | 140 | 140(-2%) | 4913.93 | 5,100 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4070 Ti SUPER 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 132.26 | 130 | 130(+1%) | 6924.53 | 5,800 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX A5000 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 130.07 | 140 | 130(+2%) | 4028.16 | 3,700 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5060 Ti 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 90.94 | 86 | 88(-3%) | 3737.25 | 2,800 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 3060 12GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 75.57 | 71 | 69(-8%) | 2137.5 | 1,700 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4060 Ti 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 63.86 | 57 | 62(-3%) | 3394.63 | 2,900 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 57.21 | 53 | 70(+23%) | 3062.31 | 910 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| Tesla P40 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 54.74 | 55 | 54(-1%) | 1007.42 | 440 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5090 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 300.4 | 290 | 260(-12%) | 14970.15 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 281.11 | 290 | 250(-12%) | 16618.98 | 15,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| H100 80GB SXM5 | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 280.74 | 430 | 300(+7%) | 11263.29 | 29,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| A100 80GB PCIe | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 200.9 | 320 | 240(+18%) | 5285.96 | 10,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4090 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 188.96 | 180 | 190(-1%) | 14770.63 | 11,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 5080 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 184.68 | 170 | 170(-8%) | 9487.7 | 6,600 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 5070 Ti 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 182.43 | 160 | 160(-12%) | 8419.56 | 5,200 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 6000 Ada Generation 48GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 179.47 | 180 | 180(+0%) | 10576.85 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 3090 Ti 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 172.26 | 180 | 170(-3%) | 6924.01 | 5,300 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 3090 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 161.89 | 170 | 150(-9%) | 5560.06 | 4,700 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4080 SUPER 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 147.48 | 140 | 150(+1%) | 9439.01 | 6,900 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX A6000 48GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 144.87 | 140 | 130(-8%) | 5662.39 | 5,100 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX A5000 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 135.83 | 140 | 130(-5%) | 4552.15 | 3,700 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4070 Ti SUPER 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 132.85 | 130 | 130(0%) | 7612.32 | 5,800 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 5060 Ti 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 93.46 | 86 | 88(-6%) | 4195.53 | 2,800 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 3060 12GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 76.92 | 71 | 69(-11%) | 2407.67 | 1,700 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4060 Ti 16GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 64.03 | 57 | 62(-4%) | 3803.45 | 2,900 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 56.74 | 53 | 70(+24%) | 3661.37 | 910 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| Tesla P40 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 53.73 | 55 | 55(+2%) | 1079.66 | 440 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 260.42 | 290 | 250(-4%) | 12742.48 | 15,000 | llama.cpp discussion #15013 (comment) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 289.3 | 290 | 250(-15%) | 14463.16 | 15,000 | llama.cpp discussion #15013 (comment) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 182.63 | 150 | 180(-2%) | 3726.99 | 2,200 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 145.67 | 100 | 140(-5%) | 5609.82 | 3,400 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| Arc Pro B60 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 68.55 | 65 | 69(+1%) | 522.36 | 190 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 58.51 | 52 | 58(0%) | 2789.51 | 640 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 53.59 | 46 | 63(+17%) | 1288.96 | 1,100 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| RTX 5090 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 273.68 | 270 | 270(-2%) | 11796.38 | 8,600 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 190.92 | 150 | 170(-10%) | 3889.54 | 2,200 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| RTX 4090 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 190.1 | 170 | 170(-8%) | 10830.41 | 7,600 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| RTX 3090 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 171.61 | 160 | 160(-5%) | 5200.68 | 3,300 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 152.7 | 100 | 130(-13%) | 5907.66 | 3,400 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Arc Pro B60 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 70.54 | 65 | 68(-4%) | 274.76 | 190 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 60.39 | 52 | 57(-6%) | 3321.91 | 640 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (Vulkan) | 53 | 46 | 63(+18%) | 1357.07 | 1,100 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 167.11 | 140 | 160(-6%) | 3552.27 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| Radeon PRO W7900 48GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 121.18 | 130 | 130(+10%) | 3213.17 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 93.84 | 97 | 98(+4%) | 4443.54 | 4,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 50.01 | 44 | 56(+12%) | 911.36 | 1,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 170.12 | 140 | 160(-9%) | 3874.25 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| Radeon PRO W7900 48GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 127.43 | 130 | 130(+2%) | 3472.86 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 97.98 | 97 | 95(-3%) | 4773.07 | 4,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0 | llama.cpp (ROCm) | 49.87 | 44 | 56(+13%) | 1003.53 | 1,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 0 | llama.cpp (CUDA) | 83.43 | 64 | 84(+1%) | 4505.82 | 1,700 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 4,096 | llama.cpp (CUDA) | 79.22 | 62 | 81(+3%) | 4158.34 | 1,500 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 8,192 | llama.cpp (CUDA) | 75.22 | 60 | 78(+4%) | 3993.81 | 1,400 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 16,384 | llama.cpp (CUDA) | 70.36 | 56 | 73(+4%) | 3449.98 | 1,200 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 32,768 | llama.cpp (CUDA) | 61.65 | 49 | 64(+5%) | 2689.42 | 890 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 0 | llama.cpp (CUDA) | 58.72 | 43 | 56(-4%) | 2443.91 | 1,200 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 4,096 | llama.cpp (CUDA) | 55.67 | 41 | 54(-3%) | 2309.84 | 1,100 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 8,192 | llama.cpp (CUDA) | 52.87 | 40 | 52(-1%) | 2216.68 | 960 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 16,384 | llama.cpp (CUDA) | 49.45 | 37 | 49(-2%) | 1956.31 | 810 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 32,768 | llama.cpp (CUDA) | 42.76 | 33 | 43(+0%) | 1567.08 | 610 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 0 | llama.cpp (CUDA) | 61.06 | 53 | 69(+14%) | 2986.97 | 1,700 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 4,096 | llama.cpp (CUDA) | 54.77 | 47 | 62(+13%) | 2633.45 | 1,200 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 8,192 | llama.cpp (CUDA) | 48.02 | 42 | 56(+17%) | 2354.14 | 900 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 16,384 | llama.cpp (CUDA) | 40.23 | 36 | 47(+17%) | 1908.86 | 610 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 32,768 | llama.cpp (CUDA) | 30.21 | 27 | 35(+17%) | 1348.17 | 370 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | MXFP4 (ggml-org GGUF) | 4,096 | llama.cpp (CUDA) | 55.21 | 41 | 54(-2%) | 2412.65 | 1,000 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | MXFP4 (ggml-org GGUF) | 4,096 | llama.cpp (CUDA) | 139.07 | 78 | 100(-28%) | 2413.97 | 1,000 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | MXFP4 (ggml-org GGUF) | 4,096 | llama.cpp (CUDA) | 262.2 | 110 | 150(-44%) | 2411.23 | 1,000 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | MXFP4 (ggml-org GGUF) | 4,096 | llama.cpp (CUDA) | 78.54 | 62 | 81(+4%) | 4513.91 | 1,500 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | MXFP4 (ggml-org GGUF) | 4,096 | llama.cpp (CUDA) | 233.11 | 140 | 180(-22%) | 4561.44 | 1,500 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | MXFP4 (ggml-org GGUF) | 4,096 | llama.cpp (CUDA) | 473.59 | 340 | 440(-7%) | 4557.52 | 1,500 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| Mac Studio M3 Ultra (80-core GPU) 512GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (Metal) | 115.52 | 150 | 140(+19%) | 2816.47 | 2,300 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (Metal) | 116.08 | 140 | 150(+28%) | 2191.13 | 2,200 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M2 Ultra (76-core GPU) 192GB | gpt-oss-120b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (Metal) | 79.68 | 95 | 99(+24%) | 1244.57 | 1,500 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M4 Max (32-core GPU) 36GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (Metal) | 92.36 | 79 | 91(-1%) | 1277.42 | 1,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M1 Max (32-core GPU) 64GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (Metal) | 75.15 | 74 | 68(-9%) | 994.75 | 860 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 249.96 | 360 | 320(+26%) | 9480.55 | 27,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 286.91 | 360 | 310(+8%) | 11521.95 | 27,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | gpt-oss-120b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 170.62 | 240 | 210(+23%) | 4494.2 | 11,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | gpt-oss-120b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 196.31 | 240 | 210(+5%) | 5518.07 | 11,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 3090 24GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 161.77 | 210 | 190(+19%) | 5170.56 | 8,600 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 4090 24GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 221.95 | 220 | 230(+4%) | 8022.33 | 20,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 4080 SUPER 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 186.51 | 170 | 180(-4%) | 8170.95 | 13,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5060 Ti 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 111.51 | 100 | 110(-4%) | 3839.21 | 5,100 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5070 Ti 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 189.45 | 200 | 210(+10%) | 6339.76 | 9,500 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5080 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 204.85 | 210 | 210(+5%) | 7476.55 | 12,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5090 32GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0 | llama.cpp (CUDA) | 282.51 | 360 | 360(+26%) | 9848.38 | 23,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 0 | llama.cpp (ROCm) | 51.81 | 36 | 44(-16%) | 625.58 | 1,500 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 32,768 | llama.cpp (ROCm) | 36.32 | 27 | 34(-7%) | 589.25 | 940 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 0 | llama.cpp (Vulkan) | 56.61 | 37 | 48(-15%) | 719.91 | 1,400 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 32,768 | llama.cpp (Vulkan) | 43.01 | 29 | 37(-14%) | 307.5 | 730 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 0 | llama.cpp (ROCm) | 73.05 | 53 | 66(-10%) | 1786.17 | 2,600 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 32,768 | llama.cpp (ROCm) | 52.17 | 41 | 51(-2%) | 1003.11 | 1,400 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 0 | llama.cpp (Vulkan) | 79.78 | 56 | 73(-8%) | 1692.49 | 2,000 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 32,768 | llama.cpp (Vulkan) | 61.12 | 43 | 56(-8%) | 555.73 | 1,100 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | llama-2-7b.Q4_0 | 0 | llama.cpp (ROCm) | 50.59 | 44 | 56(+11%) | 1545.36 | 1,400 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | llama-2-7b.Q4_0 | 0 | llama.cpp (Vulkan) | 55.73 | 46 | 62(+12%) | 1337.7 | 1,100 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
* Leave-one-out: calibrated prediction made without this row. Skipped rows (unknown model/quant) are not used for calibration.