Put numbers to the test.
Measured performance helps ground the simulation. Explore the evidence or add your own.
Theoretical vs measured
Calibrated errors hold out one measurement at a time. Other measurements from the same source or machine can remain, so this is not an independent validation of every model.
227 measurements cover 45 of 104 hardware profiles and 12 of 78 models. 163 start with no prior context. 5 include multiple GPUs.
Calibration fits separate corrections for dense and mixture-of-experts models, grouped by hardware family × backend, then by hardware profile. Multi-GPU groups also match GPU count, connection and split strategy. Corrections are shrunk toward 1 so sparse data has less influence. Efficiency priors live indata/calibration.json. Confidence also checks the selected model, precision, context and batch against measured coverage. Extrapolations stay LOW; ranges are estimates, not statistical guarantees.
What do PP512, TG128 and FP8 mean?
PP512 measures prompt processing for 512 input tokens. TG128 measures text generation over 128 output tokens. Both report tokens per second, but they time different work: PP512 does not describe the speed at which an answer appears. Neither alone measures the complete response time, and llama-bench excludes tokenization and sampling.
FP8 describes an 8-bit floating-point format, not a benchmark workload. Compare FP8 inference results only when the model, hardware, backend, context, batch and cache settings are comparable. A format supported by a model file does not guarantee a fast kernel on every GPU.
These measurements concern inference speed. For answer-quality evaluations such as MMLU-Pro or SWE-bench, use the model benchmark comparison. See how the performance estimates work for the calculation assumptions.
Test definitions: official llama-bench documentation. Individual rows retain their reported workloads and sources; not every row uses PP512 or TG128.
Fitted correction factors
measured ÷ theoretical, after shrinkage. 1.00 = theory matched reality.
| Hardware family · backend · model type | Decode n | Decode factor | Prefill n | Prefill factor |
|---|---|---|---|---|
| amd-rdna3 · llama.cpp-rocm · dense | 4 | 1.07 | 4 | 1.19 |
| amd-rdna3 · llama.cpp-vulkan · dense | 2 | 1.18 | 2 | 1.44 |
| amd-rdna4 · llama.cpp-rocm · dense | 2 | 0.99 | 2 | 1.03 |
| amd-rdna4 · llama.cpp-vulkan · dense | 2 | 1.29 | 2 | 1.42 |
| amd-strix-halo · llama.cpp-rocm · dense | 3 | 1.11 | 3 | 0.85 |
| amd-strix-halo · llama.cpp-rocm · moe | 4 | 1.28 | 4 | 0.67 |
| amd-strix-halo · llama.cpp-vulkan · dense | 3 | 1.13 | 3 | 1.17 |
| amd-strix-halo · llama.cpp-vulkan · moe | 4 | 1.36 | 4 | 0.62 |
| apple-m1 · llama.cpp-metal · dense | 9 | 0.92 | 9 | 1.15 |
| apple-m1 · llama.cpp-metal · moe | 1 | 1.01 | 1 | 1.08 |
| apple-m2 · llama.cpp-metal · dense | 18 | 1.05 | 18 | 1.16 |
| apple-m2 · llama.cpp-metal · moe | 22 | 0.68 | 2 | 0.93 |
| apple-m3 · llama.cpp-metal · dense | 12 | 1.06 | 12 | 1.19 |
| apple-m3 · llama.cpp-metal · moe | 1 | 0.89 | 1 | 1.11 |
| apple-m4 · llama.cpp-metal · dense | 9 | 1.14 | 9 | 1.24 |
| apple-m4 · llama.cpp-metal · moe | 1 | 1.08 | 1 | 1.11 |
| apple-m5 · llama.cpp-metal · dense | 15 | 1.28 | 15 | 3.79 |
| intel-xe2 · llama.cpp-vulkan · dense | 2 | 1.05 | 2 | 1.59 |
| nvidia-ada · llama.cpp-cuda · dense | 10 | 1.05 | 10 | 1.14 |
| nvidia-ada · llama.cpp-cuda · moe | 2 | 1.03 | 2 | 0.64 |
| nvidia-ada · llama.cpp-vulkan · dense | 1 | 1.05 | 1 | 1.20 |
| nvidia-ampere · llama.cpp-cuda · dense | 12 | 0.91 | 12 | 1.03 |
| nvidia-ampere · llama.cpp-cuda · moe | 1 | 0.88 | 1 | 0.78 |
| nvidia-ampere · llama.cpp-vulkan · dense | 1 | 1.03 | 1 | 1.26 |
| nvidia-blackwell · llama.cpp-cuda · dense | 14 | 1.03 | 14 | 1.22 |
| nvidia-blackwell · llama.cpp-cuda · moe | 10 | 0.86 | 9 | 0.49 |
| nvidia-blackwell · llama.cpp-vulkan · dense | 1 | 1.01 | 1 | 1.18 |
| nvidia-gb10 · llama.cpp-cuda · dense | 5 | 1.03 | 5 | 2.88 |
| nvidia-gb10 · llama.cpp-cuda · moe | 35 | 1.19 | 35 | 2.27 |
| nvidia-gb10 · llama.cpp-vulkan · dense | 2 | 1.09 | 2 | 2.84 |
| nvidia-hopper · llama.cpp-cuda · dense | 4 | 0.84 | 4 | 0.46 |
| nvidia-hopper · vllm · dense | 5 | 1.06 | 0 | — |
| nvidia-hopper · vllm · dense · 2gpu · nvlink · tensor | 5 | 1.14 | 0 | — |
| nvidia-pascal · llama.cpp-cuda · dense | 2 | 1.00 | 2 | 1.78 |
Add a measured benchmark
Your own llama-bench / LM Studio numbers improve the calibration. Stored only in this browser.
Measured benchmarks (227)
Reported observations with sources. Rows marked chip proxy came from the same chip in a different or unspecified computer configuration.
| Hardware | Model | Quant | Ctx | Backend | Measured tg | Theory | Calibrated* | Measured pp | Theory pp | Source |
|---|---|---|---|---|---|---|---|---|---|---|
| Mac Studio M1 Max (32-core GPU) 32GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 23.03 | 21 | 23(-2%) | 599.53512 tokens · depth 0 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Max (32-core GPU) 32GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 40.2 | 36 | 37(-7%) | 537.37512 tokens · depth 0 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Max (32-core GPU) 32GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 61.19 | 52 | 54(-12%) | 530.06512 tokens · depth 0 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (48-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 33.92 | 42 | 35(+2%) | 875.81512 tokens · depth 0 | 700 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (48-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 55.69 | 69 | 57(+3%) | 783.45512 tokens · depth 0 | 700 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (48-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 74.93 | 100 | 85(+14%) | 772.24512 tokens · depth 0 | 700 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (64-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 37.01 | 42 | 37(-1%) | 1168.89512 tokens · depth 0 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (64-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 59.87 | 69 | 61(+2%) | 1042.95512 tokens · depth 0 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M1 Ultra (64-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 83.73 | 100 | 90(+7%) | 1030.04512 tokens · depth 0 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Max (38-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 24.65 | 21 | 25(+0%) | 755.67512 tokens · depth 0 | 600 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Max (38-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 41.83 | 36 | 41(-2%) | 677.91512 tokens · depth 0 | 600 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Max (38-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 65.95 | 52 | 59(-11%) | 671.31512 tokens · depth 0 | 600 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (60-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 39.86 | 42 | 40(-1%) | 1128.59512 tokens · depth 0 | 940 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (60-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 62.14 | 69 | 67(+7%) | 1003.16512 tokens · depth 0 | 940 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (60-core GPU) 64GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 88.64 | 100 | 97(+10%) | 1013.81512 tokens · depth 0 | 940 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 41.02 | 42 | 45(+9%) | 1401.85512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 66.64 | 69 | 74(+11%) | 1248.59512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 94.27 | 100 | 110(+14%) | 1238.48512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro M3 Max (30-core GPU) 36GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 19.54 | 16 | 20(+2%) | 589.41512 tokens · depth 0 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro M3 Max (30-core GPU) 36GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 34.3 | 27 | 33(-4%) | 566.4512 tokens · depth 0 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro M3 Max (30-core GPU) 36GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 56.58 | 40 | 46(-18%) | 567.59512 tokens · depth 0 | 470 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 16-inch M3 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 25.09 | 21 | 25(0%) | 779.17512 tokens · depth 0 | 620 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 16-inch M3 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 42.75 | 36 | 41(-3%) | 757.64512 tokens · depth 0 | 620 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 16-inch M3 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 66.31 | 52 | 60(-10%) | 759.7512 tokens · depth 0 | 620 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (60-core GPU) 96GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 42.24 | 43 | 40(-5%) | 1121.8512 tokens · depth 0 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (60-core GPU) 96GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 63.55 | 71 | 69(+8%) | 1085.76512 tokens · depth 0 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (60-core GPU) 96GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 88.4 | 100 | 100(+14%) | 1073.09512 tokens · depth 0 | 930 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 39.78 | 43 | 41(+3%) | 1538.34512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 63.93 | 71 | 68(+7%) | 1487.51512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 92.14 | 100 | 99(+8%) | 1471.24512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 17.18 | 15 | 19(+9%) | 464.48512 tokens · depth 0 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 30.69 | 26 | 31(0%) | 449.62512 tokens · depth 0 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M4 Pro (14-core CPU, 20-core GPU) 24GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 50.74 | 38 | 43(-15%) | 439.78512 tokens · depth 0 | 360 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 24.29 | 23 | 27(+11%) | 736.25512 tokens · depth 0 | 570 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 43.87 | 38 | 44(0%) | 718.56512 tokens · depth 0 | 570 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 69.95 | 56 | 62(-11%) | 713.93512 tokens · depth 0 | 570 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 31.64 | 30 | 34(+7%) | 922.83512 tokens · depth 0 | 710 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 54.05 | 50 | 56(+3%) | 891.94512 tokens · depth 0 | 710 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M4 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 83.06 | 73 | 80(-4%) | 885.68512 tokens · depth 0 | 710 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 14-inch M5 (10-core GPU) 32GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 18.42 | 14 | 20(+8%) | 715.42512 tokens · depth 0 | 180 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| MacBook Pro 14-inch M5 (10-core GPU) 32GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 31.88 | 21 | 27(-15%) | 722.79512 tokens · depth 0 | 180 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 21.39 | 17 | 24(+13%) | 1310.78512 tokens · depth 0 | 290 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 38.94 | 29 | 39(+1%) | 1302.59512 tokens · depth 0 | 290 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (15-core CPU, 16-core GPU) 24GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 67.59 | 42 | 54(-20%) | 1340.44512 tokens · depth 0 | 290 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 21.55 | 17 | 24(+11%) | 1588.78512 tokens · depth 0 | 370 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 38.92 | 29 | 39(+0%) | 1553.86512 tokens · depth 0 | 370 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac mini M5 Pro (18-core CPU, 20-core GPU) 48GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 66.33 | 42 | 55(-18%) | 1620.64512 tokens · depth 0 | 370 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 37.11 | 34 | 44(+19%) | 3158.49512 tokens · depth 0 | 730 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 72.42 | 56 | 70(-3%) | 3143.81512 tokens · depth 0 | 730 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 119.92 | 82 | 99(-18%) | 3219.99512 tokens · depth 0 | 730 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Ultra (80-core GPU) 256GB | llama-2-7b | F16 (mostly F16) | 0+128 generated | llama.cpp (Metal) | 73.6 | 64 | 77(+5%) | 5207.14512 tokens · depth 0 | 1,500 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Ultra (80-core GPU) 256GB | llama-2-7b | Q8_0 | 0+128 generated | llama.cpp (Metal) | 119.01 | 110 | 130(+7%) | 4910.55512 tokens · depth 0 | 1,500 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M5 Ultra (80-core GPU) 256GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Metal) | 179.1 | 150 | 180(-1%) | 4944.66512 tokens · depth 0 | 1,500 | llama.cpp discussion #4167 (Apple Silicon scoreboard) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 | 0 | llama.cpp (Metal) | 43.15 | 42 | 45(+4%) | 1525.95512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 73.11 | 70 | 74(+1%) | 1368.18512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 108.8 | 100 | 110(-1%) | 1391.78512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 | 0 | llama.cpp (Metal) | 43.24 | 42 | 45(+4%) | 1561.35512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 73.35 | 70 | 74(+1%) | 1386.97512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 109.41 | 100 | 110(-2%) | 1412.42512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | F16 | 0 | llama.cpp (Metal) | 49.55 | 42 | 44(-11%) | 1627.82512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q8_0 | 0 | llama.cpp (Metal) | 82.83 | 70 | 73(-12%) | 1486.51512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | llama-2-7b | Q4_0 | 0 | llama.cpp (Metal) | 125.21 | 100 | 110(-15%) | 1488.81512 tokens · depth 0 | 1,200 | llama.cpp discussion #4167 (M2 Ultra history table) |
| RTX 5090 32GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 290.02 | 290 | 310(+7%) | 14073.41512 tokens · depth 0 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 274.2 | 290 | 280(+3%) | 14854.63512 tokens · depth 0 | 14,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| H100 80GB SXM5 | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 267.81 | 430 | 370(+40%) | 9918.34512 tokens · depth 0 | 28,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| A100 80GB PCIe | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 190.88 | 320 | 240(+28%) | 4849.53512 tokens · depth 0 | 10,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4090 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 186.21 | 180 | 190(+2%) | 11992.7512 tokens · depth 0 | 11,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5080 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 181.99 | 170 | 180(-1%) | 8297.36512 tokens · depth 0 | 6,600 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5070 Ti 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 176.85 | 160 | 170(-2%) | 6952.38512 tokens · depth 0 | 5,100 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 6000 Ada Generation 48GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 176.07 | 180 | 180(+3%) | 9229.23512 tokens · depth 0 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 3090 Ti 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 171.19 | 180 | 170(-1%) | 6567.49512 tokens · depth 0 | 5,300 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 3090 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 158.16 | 170 | 160(+0%) | 5174.69512 tokens · depth 0 | 4,700 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4080 SUPER 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 148.33 | 140 | 150(-2%) | 8125.15512 tokens · depth 0 | 6,800 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX A6000 48GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 138.73 | 140 | 140(-1%) | 4913.93512 tokens · depth 0 | 5,100 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4070 Ti SUPER 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 132.26 | 130 | 130(+1%) | 6924.53512 tokens · depth 0 | 5,800 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX A5000 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 130.07 | 140 | 130(+2%) | 4028.16512 tokens · depth 0 | 3,700 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5060 Ti 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 90.94 | 85 | 90(-1%) | 3737.25512 tokens · depth 0 | 2,800 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 3060 12GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 75.57 | 71 | 70(-8%) | 2137.5512 tokens · depth 0 | 1,700 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 4060 Ti 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 63.86 | 57 | 62(-3%) | 3394.63512 tokens · depth 0 | 2,900 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 57.21 | 53 | 55(-3%) | 3062.31512 tokens · depth 0 | 910 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| Tesla P40 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 54.74 | 54 | 54(-2%) | 1007.42512 tokens · depth 0 | 440 | llama.cpp discussion #15013 (CUDA scoreboard, no FA) |
| RTX 5090 32GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 300.4 | 290 | 310(+2%) | 14970.15512 tokens · depth 0 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 281.11 | 290 | 280(0%) | 16618.98512 tokens · depth 0 | 14,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| H100 80GB SXM5 | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 280.74 | 430 | 370(+32%) | 11263.29512 tokens · depth 0 | 28,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| A100 80GB PCIe | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 200.9 | 320 | 240(+18%) | 5285.96512 tokens · depth 0 | 10,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4090 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 188.96 | 180 | 190(+0%) | 14770.63512 tokens · depth 0 | 11,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 5080 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 184.68 | 170 | 180(-3%) | 9487.7512 tokens · depth 0 | 6,600 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 5070 Ti 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 182.43 | 160 | 170(-6%) | 8419.56512 tokens · depth 0 | 5,100 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 6000 Ada Generation 48GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 179.47 | 180 | 180(+0%) | 10576.85512 tokens · depth 0 | 12,000 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 3090 Ti 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 172.26 | 180 | 170(-2%) | 6924.01512 tokens · depth 0 | 5,300 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 3090 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 161.89 | 170 | 160(-3%) | 5560.06512 tokens · depth 0 | 4,700 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4080 SUPER 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 147.48 | 140 | 150(-1%) | 9439.01512 tokens · depth 0 | 6,800 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX A6000 48GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 144.87 | 140 | 130(-8%) | 5662.39512 tokens · depth 0 | 5,100 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX A5000 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 135.83 | 140 | 130(-4%) | 4552.15512 tokens · depth 0 | 3,700 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4070 Ti SUPER 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 132.85 | 130 | 130(0%) | 7612.32512 tokens · depth 0 | 5,800 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 5060 Ti 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 93.46 | 85 | 89(-5%) | 4195.53512 tokens · depth 0 | 2,800 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 3060 12GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 76.92 | 71 | 69(-10%) | 2407.67512 tokens · depth 0 | 1,700 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX 4060 Ti 16GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 64.03 | 57 | 62(-4%) | 3803.45512 tokens · depth 0 | 2,900 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 56.74 | 53 | 56(-2%) | 3661.37512 tokens · depth 0 | 910 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| Tesla P40 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (CUDA) | 53.73 | 54 | 55(+2%) | 1079.66512 tokens · depth 0 | 440 | llama.cpp discussion #15013 (CUDA scoreboard, with FA) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 260.42 | 290 | 290(+11%) | 12742.48512 tokens · depth 0 | 14,000 | llama.cpp discussion #15013 (comment) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | llama-2-7b | Q4_0 | 0 | llama.cpp (CUDA) | 289.3 | 290 | 280(-3%) | 14463.16512 tokens · depth 0 | 14,000 | llama.cpp discussion #15013 (comment) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 182.63 | 140 | 180(-2%) | 3726.99512 tokens · depth 0 | 2,200 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 145.67 | 100 | 140(-5%) | 5609.82512 tokens · depth 0 | 3,400 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| Arc Pro B60 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 68.55 | 64 | 69(+1%) | 522.36512 tokens · depth 0 | 190 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 58.51 | 52 | 58(-1%) | 2789.51512 tokens · depth 0 | 640 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 53.59 | 46 | 53(0%) | 1288.96512 tokens · depth 0 | 1,100 | llama.cpp discussion #10879 (Vulkan scoreboard, no FA) |
| RTX 5090 32GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 273.68 | 270 | 270(-2%) | 11796.38512 tokens · depth 0 | 8,500 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 190.92 | 140 | 170(-10%) | 3889.54512 tokens · depth 0 | 2,200 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| RTX 4090 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 190.1 | 170 | 170(-9%) | 10830.41512 tokens · depth 0 | 7,500 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| RTX 3090 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 171.61 | 160 | 160(-5%) | 5200.68512 tokens · depth 0 | 3,300 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 152.7 | 100 | 130(-13%) | 5907.66512 tokens · depth 0 | 3,400 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Arc Pro B60 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 70.54 | 64 | 67(-4%) | 274.76512 tokens · depth 0 | 190 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| NVIDIA DGX Spark Founders Edition 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 60.39 | 52 | 57(-6%) | 3321.91512 tokens · depth 0 | 640 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 53 | 46 | 54(+1%) | 1357.07512 tokens · depth 0 | 1,100 | llama.cpp discussion #10879 (Vulkan scoreboard, FA enabled) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 167.11 | 140 | 160(-6%) | 3552.27512 tokens · depth 0 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| Radeon PRO W7900 48GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 121.18 | 130 | 130(+10%) | 3213.17512 tokens · depth 0 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 93.84 | 97 | 98(+4%) | 4443.54512 tokens · depth 0 | 4,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 50.01 | 44 | 49(-1%) | 911.36512 tokens · depth 0 | 1,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, no FA) |
| Radeon RX 7900 XTX 24GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 170.12 | 140 | 160(-9%) | 3874.25512 tokens · depth 0 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| Radeon PRO W7900 48GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 127.43 | 130 | 130(+1%) | 3472.86512 tokens · depth 0 | 2,800 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| Radeon AI PRO R9700 32GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 97.98 | 97 | 95(-4%) | 4773.07512 tokens · depth 0 | 4,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | Q4_0 | 0+128 generated | llama.cpp (ROCm) | 49.87 | 44 | 50(-1%) | 1003.53512 tokens · depth 0 | 1,400 | llama.cpp discussion #15021 (ROCm/HIP scoreboard, with FA) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 0+32 generated | llama.cpp (CUDA) | 83.43 | 64 | 79(-6%) | 4505.822048 tokens · depth 0 · microbatch 2048 | 1,700 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 79.22 | 62 | 76(-5%) | 4158.342048 tokens · depth 4096 · microbatch 2048 | 1,500 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 8,192+32 generated | llama.cpp (CUDA) | 75.22 | 60 | 73(-3%) | 3993.812048 tokens · depth 8192 · microbatch 2048 | 1,400 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 16,384+32 generated | llama.cpp (CUDA) | 70.36 | 56 | 68(-3%) | 3449.982048 tokens · depth 16384 · microbatch 2048 | 1,200 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | gpt-oss 20B MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) | 32,768+32 generated | llama.cpp (CUDA) | 61.65 | 49 | 60(-3%) | 2689.422048 tokens · depth 32768 · microbatch 2048 | 890 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 0+32 generated | llama.cpp (CUDA) | 58.72 | 43 | 52(-11%) | 2443.912048 tokens · depth 0 · microbatch 2048 | 1,200 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 55.67 | 41 | 50(-10%) | 2309.842048 tokens · depth 4096 · microbatch 2048 | 1,100 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 8,192+32 generated | llama.cpp (CUDA) | 52.87 | 40 | 49(-8%) | 2216.682048 tokens · depth 8192 · microbatch 2048 | 960 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 16,384+32 generated | llama.cpp (CUDA) | 49.45 | 37 | 45(-8%) | 1956.312048 tokens · depth 16384 · microbatch 2048 | 810 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | gpt-oss 120B MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) | 32,768+32 generated | llama.cpp (CUDA) | 42.76 | 33 | 40(-7%) | 1567.082048 tokens · depth 32768 · microbatch 2048 | 610 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 0+32 generated | llama.cpp (CUDA) | 61.06 | 53 | 65(+6%) | 2986.972048 tokens · depth 0 · microbatch 2048 | 1,700 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 54.77 | 47 | 58(+5%) | 2633.452048 tokens · depth 4096 · microbatch 2048 | 1,200 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 8,192+32 generated | llama.cpp (CUDA) | 48.02 | 42 | 52(+8%) | 2354.142048 tokens · depth 8192 · microbatch 2048 | 900 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 16,384+32 generated | llama.cpp (CUDA) | 40.23 | 36 | 44(+8%) | 1908.862048 tokens · depth 16384 · microbatch 2048 | 610 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | qwen3-coder-30b-a3b | qwen3moe 30B.A3B Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) | 32,768+32 generated | llama.cpp (CUDA) | 30.21 | 27 | 33(+9%) | 1348.172048 tokens · depth 32768 · microbatch 2048 | 370 | llama.cpp benches/dgx-spark/dgx-spark.md (linked from discussion #16578, ggerganov) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | MXFP4 (ggml-org GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 55.21 | 41 | 50(-9%) | 2412.654096 tokens · depth 0 · microbatch 2048 | 1,100 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | MXFP4 (ggml-org GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 139.07aggregate · batch 8 | 78 | 94(-33%) | 2413.974096 tokens · depth 0 · microbatch 2048 | 1,100 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-120b | MXFP4 (ggml-org GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 262.2aggregate · batch 32 | 110 | 140(-48%) | 2411.234096 tokens · depth 0 · microbatch 2048 | 1,100 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | MXFP4 (ggml-org GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 78.54 | 62 | 76(-4%) | 4513.914096 tokens · depth 0 · microbatch 2048 | 1,600 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | MXFP4 (ggml-org GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 233.11aggregate · batch 8 | 140 | 170(-28%) | 4561.444096 tokens · depth 0 · microbatch 2048 | 1,600 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| NVIDIA DGX Spark Founders Edition 128GB | gpt-oss-20b | MXFP4 (ggml-org GGUF) | 4,096+32 generated | llama.cpp (CUDA) | 473.59aggregate · batch 32 | 340 | 410(-13%) | 4557.524096 tokens · depth 0 · microbatch 2048 | 1,600 | llama.cpp benches/dgx-spark/dgx-spark.md (llama-batched-bench) |
| Mac Studio M3 Ultra (80-core GPU) 512GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (Metal) | 115.52 | 150 | 150(+26%) | 2816.472048 tokens · depth 0 · microbatch 2048 | 2,300 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (Metal) | 116.08 | 140 | 95(-18%) | 2191.132048 tokens · depth 0 · microbatch 2048 | 2,200 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-120b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (Metal) | 79.68 | 95 | 63(-21%) | 1244.572048 tokens · depth 0 · microbatch 2048 | 1,500 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M4 Max (32-core GPU) 36GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (Metal) | 92.36 | 79 | 79(-14%) | 1277.422048 tokens · depth 0 · microbatch 2048 | 1,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| Mac Studio M1 Max (32-core GPU) 64GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (Metal) | 75.15 | 74 | 74(-1%) | 994.752048 tokens · depth 0 · microbatch 2048 | 860 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Max-Q Workstation Edition 96GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 249.96 | 360 | 280(+14%) | 9480.552048 tokens · depth 0 · microbatch 2048 | 24,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 286.91 | 360 | 300(+5%) | 11521.952048 tokens · depth 0 · microbatch 2048 | 27,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Max-Q Workstation Edition 96GB | gpt-oss-120b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 170.62 | 240 | 190(+9%) | 4494.22048 tokens · depth 0 · microbatch 2048 | 17,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX PRO 6000 Blackwell Workstation Edition 96GB | gpt-oss-120b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 196.31 | 240 | 200(+1%) | 5518.072048 tokens · depth 0 · microbatch 2048 | 19,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 3090 24GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 161.77 | 210 | 210(+30%) | 5170.562048 tokens · depth 0 · microbatch 2048 | 8,600 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 4090 24GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 221.95 | 220 | 240(+6%) | 8022.332048 tokens · depth 0 · microbatch 2048 | 20,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 4080 SUPER 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 186.51 | 170 | 170(-10%) | 8170.952048 tokens · depth 0 · microbatch 2048 | 13,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5060 Ti 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 111.51 | 100 | 87(-22%) | 3839.212048 tokens · depth 0 · microbatch 2048 | 5,100 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5070 Ti 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 189.45 | 200 | 170(-12%) | 6339.762048 tokens · depth 0 · microbatch 2048 | 9,400 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5080 16GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 204.85 | 210 | 180(-13%) | 7476.552048 tokens · depth 0 · microbatch 2048 | 12,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| RTX 5090 32GB | gpt-oss-20b | MXFP4 (ggml-org gpt-oss GGUF) | 0+128 generated | llama.cpp (CUDA) | 282.51 | 360 | 310(+8%) | 9848.382048 tokens · depth 0 · microbatch 2048 | 22,000 | llama.cpp discussion #15396 (guide: running gpt-oss with llama.cpp) |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 0+128 generated | llama.cpp (ROCm) | 51.81 | 36 | 46(-10%) | 625.58512 tokens · depth 0 | 1,500 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 32,768+32 generated | llama.cpp (ROCm) | 36.32 | 27 | 36(+0%) | 589.252048 tokens · depth 32768 | 930 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 0+128 generated | llama.cpp (Vulkan) | 56.61 | 37 | 53(-6%) | 719.91512 tokens · depth 0 | 1,400 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-120b | gpt-oss-120b-mxfp4 | 32,768+32 generated | llama.cpp (Vulkan) | 43.01 | 29 | 41(-6%) | 307.52048 tokens · depth 32768 | 730 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 0+128 generated | llama.cpp (ROCm) | 73.05 | 53 | 71(-3%) | 1786.17512 tokens · depth 0 | 2,600 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 32,768+32 generated | llama.cpp (ROCm) | 52.17 | 41 | 55(+6%) | 1003.112048 tokens · depth 32768 | 1,400 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 0+128 generated | llama.cpp (Vulkan) | 79.78 | 56 | 81(+2%) | 1692.49512 tokens · depth 0 | 2,000 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | gpt-oss-20b | gpt-oss-20b-mxfp4 | 32,768+32 generated | llama.cpp (Vulkan) | 61.12 | 43 | 62(+1%) | 555.732048 tokens · depth 32768 | 1,100 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | llama-2-7b.Q4_0 | 0+128 generated | llama.cpp (ROCm) | 50.59 | 44 | 49(-3%) | 1545.36512 tokens · depth 0 | 1,400 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| AMD Ryzen AI Max+ 395 mini PC (generic) 128GB | llama-2-7b | llama-2-7b.Q4_0 | 0+128 generated | llama.cpp (Vulkan) | 55.73 | 46 | 52(-6%) | 1337.7512 tokens · depth 0 | 1,100 | kyuz0 amd-strix-halo-toolboxes benchmark results.json |
| NVIDIA DGX Spark Founders Edition 128GB | glm-4.7-flash | Q8_0 | 0+32 generated | llama.cpp (CUDA) | 48.68 | 58 | 73(+49%) | 2364.182048 tokens · depth 0 · microbatch 2048 | 1,500 | llama.cpp official DGX Spark benchmark report |
| NVIDIA DGX Spark Founders Edition 128GB | glm-4.7-flash | Q8_0 | 4,096+32 generated | llama.cpp (CUDA) | 44.62 | 54 | 68(+52%) | 1684.132048 tokens · depth 4096 · microbatch 2048 | 750 | llama.cpp official DGX Spark benchmark report |
| NVIDIA DGX Spark Founders Edition 128GB | glm-4.7-flash | Q8_0 | 8,192+32 generated | llama.cpp (CUDA) | 42.59 | 51 | 63(+49%) | 1314.682048 tokens · depth 8192 · microbatch 2048 | 500 | llama.cpp official DGX Spark benchmark report |
| NVIDIA DGX Spark Founders Edition 128GB | glm-4.7-flash | Q8_0 | 16,384+32 generated | llama.cpp (CUDA) | 38.72 | 45 | 56(+45%) | 914.052048 tokens · depth 16384 · microbatch 2048 | 300 | llama.cpp official DGX Spark benchmark report |
| NVIDIA DGX Spark Founders Edition 128GB | glm-4.7-flash | Q8_0 | 32,768+32 generated | llama.cpp (CUDA) | 32.65 | 37 | 46(+40%) | 567.22048 tokens · depth 32768 · microbatch 2048 | 170 | llama.cpp official DGX Spark benchmark report |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 0+128 generated | llama.cpp (CUDA) | 69.85 | 71 | 81(+16%) | 2657.12512 tokens · depth 0 · microbatch 2048 | 1,600 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 8,192+128 generated | llama.cpp (CUDA) | 59.56 | 62 | 71(+18%) | 2361.51512 tokens · depth 8192 · microbatch 2048 | 1,300 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 32,768+128 generated | llama.cpp (CUDA) | 52.06 | 52 | 59(+14%) | 1876.73512 tokens · depth 32768 · microbatch 2048 | 920 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-31b | Q4_K_M | 0+128 generated | llama.cpp (CUDA) | 10.98 | 11 | 11(-1%) | 743.03512 tokens · depth 0 · microbatch 2048 | 200 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-31b | Q4_K_M | 8,192+128 generated | llama.cpp (CUDA) | 10.02 | 9.9 | 10(+2%) | 620.04512 tokens · depth 8192 · microbatch 2048 | 180 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-31b | Q4_K_M | 32,768+128 generated | llama.cpp (CUDA) | 9.15 | 9 | 9.2(+1%) | 482.44512 tokens · depth 32768 · microbatch 2048 | 150 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 512+128 generated | llama.cpp (CUDA) | 67.46 | 68 | 77(+14%) | 2583.05512 tokens · depth 0 · microbatch 2048 | 1,600 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 512+128 generated | llama.cpp (CUDA) | 107.28aggregate · batch 2 | 99 | 110(+4%) | 2935.23512 tokens · depth 0 · microbatch 2048 | 1,600 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 512+128 generated | llama.cpp (CUDA) | 154.87aggregate · batch 4 | 130 | 150(-5%) | 2981.25512 tokens · depth 0 · microbatch 2048 | 1,600 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 512+128 generated | llama.cpp (CUDA) | 211.26aggregate · batch 8 | 170 | 190(-12%) | 2998.59512 tokens · depth 0 · microbatch 2048 | 1,600 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 512+128 generated | llama.cpp (CUDA) | 297.08aggregate · batch 16 | 220 | 240(-19%) | 2997.52512 tokens · depth 0 · microbatch 2048 | 1,600 | shamily Gemma 4 GB10 measurements (MIT) |
| ASUS Ascent GX10 (NVIDIA GB10) 128GBChip proxy · configuration differs or is unspecified | gemma-4-26b-a4b | Q4_K_M | 512+128 generated | llama.cpp (CUDA) | 409.42aggregate · batch 32 | 310 | 340(-17%) | 3022.3512 tokens · depth 0 · microbatch 2048 | 1,600 | shamily Gemma 4 GB10 measurements (MIT) |
| RTX 5090 32GB | qwen3-8b | Q8_0 | 0+128 generated | llama.cpp (CUDA) | 160.1 | 150 | 160(-2%) | — | — | kekzl/imp measured llama.cpp baseline (MIT) |
| RTX 5090 32GB | qwen3-8b | Q8_0 | 0 | llama.cpp (CUDA) | — | — | — | 13724512 tokens · depth 0 | 10,000 | kekzl/imp measured llama.cpp baseline (MIT) |
| RTX 5090 32GB | qwen3-14b | Q6_K | 0+128 generated | llama.cpp (CUDA) | 114.8 | 100 | 110(-8%) | — | — | kekzl/imp measured llama.cpp baseline (MIT) |
| RTX 5090 32GB | qwen3-14b | Q6_K | 0 | llama.cpp (CUDA) | — | — | — | 6522512 tokens · depth 0 | 5,600 | kekzl/imp measured llama.cpp baseline (MIT) |
| RTX 5090 32GB | qwen3-30b-a3b | Q4_K_M | 0+128 generated | llama.cpp (CUDA) | 295.7 | 390 | 330(+13%) | — | — | kekzl/imp measured llama.cpp baseline (MIT) |
| RTX 5090 32GB | qwen3-30b-a3b | Q4_K_M | 0 | llama.cpp (CUDA) | — | — | — | 9288512 tokens · depth 0 | 24,000 | kekzl/imp measured llama.cpp baseline (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 | 0+32 generated | llama.cpp (Metal) | 129.97 | 140 | 94(-28%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 | 4,096+32 generated | llama.cpp (Metal) | 123.38 | 140 | 91(-26%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 | 8,192+32 generated | llama.cpp (Metal) | 117.39 | 130 | 88(-25%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 | 16,384+32 generated | llama.cpp (Metal) | 109.75 | 120 | 82(-25%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-20b | MXFP4 | 32,768+32 generated | llama.cpp (Metal) | 98.25 | 110 | 73(-26%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-120b | MXFP4 | 0+32 generated | llama.cpp (Metal) | 85.6 | 95 | 63(-27%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-120b | MXFP4 | 4,096+32 generated | llama.cpp (Metal) | 82.03 | 92 | 61(-26%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-120b | MXFP4 | 8,192+32 generated | llama.cpp (Metal) | 78.23 | 88 | 59(-25%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-120b | MXFP4 | 16,384+32 generated | llama.cpp (Metal) | 73.2 | 83 | 55(-25%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | gpt-oss-120b | MXFP4 | 32,768+32 generated | llama.cpp (Metal) | 65.52 | 73 | 49(-26%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | qwen3-coder-30b-a3b | Q8_0 | 0+32 generated | llama.cpp (Metal) | 78.97 | 120 | 80(+1%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | qwen3-coder-30b-a3b | Q8_0 | 4,096+32 generated | llama.cpp (Metal) | 71.18 | 110 | 72(+1%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | qwen3-coder-30b-a3b | Q8_0 | 8,192+32 generated | llama.cpp (Metal) | 65.11 | 98 | 66(+1%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | qwen3-coder-30b-a3b | Q8_0 | 16,384+32 generated | llama.cpp (Metal) | 56.87 | 83 | 56(-2%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | qwen3-coder-30b-a3b | Q8_0 | 32,768+32 generated | llama.cpp (Metal) | 45.26 | 64 | 43(-6%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | glm-4.7-flash | Q8_0 | 0+32 generated | llama.cpp (Metal) | 59.58 | 130 | 89(+50%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | glm-4.7-flash | Q8_0 | 4,096+32 generated | llama.cpp (Metal) | 47.44 | 120 | 84(+78%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | glm-4.7-flash | Q8_0 | 8,192+32 generated | llama.cpp (Metal) | 40.2 | 120 | 80(+99%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | glm-4.7-flash | Q8_0 | 16,384+32 generated | llama.cpp (Metal) | 31.5 | 100 | 72(+129%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| Mac Studio M2 Ultra (76-core GPU) 192GBChip proxy · configuration differs or is unspecified | glm-4.7-flash | Q8_0 | 32,768+32 generated | llama.cpp (Metal) | 21.81 | 86 | 60(+176%) | — | — | llama.cpp official Mac M2 Ultra benchmark report (MIT) |
| RTX 5090 32GB | gpt-oss-20b | MXFP4 | 0+128 generated | llama.cpp (CUDA) | 335.9 | 360 | 290(-14%) | — | — | kekzl/imp measured llama.cpp baseline (MIT) |
| H100 80GB SXM5 | llama-3.1-8b | Q8_0 GGUF | 921+255 generated | llama.cpp (CUDA) | 214.21 | 230 | 180(-15%) | 10782.65894 tokens · depth 27 | 23,000 | Winder.AI H100 inference benchmark: first request after 20 warmups |
| H100 80GB SXM5 | qwen3.8-27b | Q8_0 GGUF | 903+255 generated | llama.cpp (CUDA) | 67.22 | 61 | 45(-33%) | 2151.76903 tokens · depth 0 | 7,000 | Winder.AI H100 inference benchmark: first request after 20 warmups |
| H100 80GB SXM5 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 508+511 generated | vLLM | 150.68345 | 140 | 150(+1%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 2,008+511 generated | vLLM | 148.722 | 140 | 150(+1%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 8,008+511 generated | vLLM | 142.1103 | 130 | 140(0%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 12,008+511 generated | vLLM | 138.3221 | 130 | 140(-1%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 20,008+511 generated | vLLM | 131.37235 | 120 | 130(-2%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 ×2 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 508+511 generated | vLLM | 223.6028 | 190 | 220(0%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 ×2 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 2,008+511 generated | vLLM | 220.6257 | 190 | 220(+0%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 ×2 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 8,008+511 generated | vLLM | 212.7753 | 180 | 210(0%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 ×2 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 12,008+511 generated | vLLM | 208.18245 | 180 | 210(-1%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| H100 80GB SXM5 ×2 | qwen3-8b | BF16 (unquantized Qwen3-8B) | 20,008+511 generated | vLLM | 199.4581 | 170 | 200(-2%) | — | — | cadamcat H100 Qwen3-8B context ladder, median of two sequential rounds |
| Mac Studio M5 Max (40-core GPU) 128GBChip proxy · configuration differs or is unspecified | qwen3-8b | Q4_K_M (Qwen official GGUF) | 0+256 generated | llama.cpp (Metal) | 98.253792 | 84 | 110(+9%) | 2778.057184512 tokens · depth 0 · microbatch 512 | 600 | Nekko Labs · llama-bench on MacBook Pro M5 Max |
* Leave-one-out: calibrated prediction made without this row. Skipped rows (unknown model/quant) are not used for calibration.
Recorded generation length, initial prompt depth and microbatch size are replayed when available. Missing KV precision uses an F16 assumption. New measurement source license notices: llama.cpp, shamily, kekzl/imp.