ScoreScore (mean of standard and harder variant exact) Standard Harder variant, exact Harder variant, partial
AgainstCompletion tokens per task Seconds per task Cost per test (USD)
LabelsBest runs All points None
Search
Clear filters
Family
Anthropic OpenAI Google DeepSeek Other
Effort
low medium high on off fixed
0% 20% 40% 60% 80% 100% 10 20 50 100 200 500 1K 2K 5K 10K Completion tokens per task (log scale) Score (mean of standard and harder variant exact) Gemini 4 Argon · high Gemini 4 Carbon · medium GPT-6 Astra · high GPT-6 Astra · medium GPT-6.2 Astra · medium Gemini 4 Argon · low Qwen3.8-27B · off
Labelled are the runs that no other run beats on both axes: a higher score for less (Labels shows all of them). Hover any point for its numbers, click it to find its row in the table.
Model Effort Score▼ Standard Harder exact Harder partial Tokens / task Seconds / task $ / test Run date Runs Gemini 4 Argonhigh 78.9% 96.7% 61.1% 0.927 3,291 44.5 $0.0337 2026-10-30 1 Gemini 4 Carbonmedium 78.3% 93.9% 62.8% 0.903 1,874 38.4 $0.0195 2026-11-10 1 Gemini 4 Neonhigh 78.3% 96.1% 60.6% 0.936 7,394 62.0 $0.1121 2026-11-19 1 Gemini 4 Neonmedium 78.3% 96.7% 60.0% 0.931 2,455 23.6 $0.0381 2026-11-19 1 GPT-6 Astrahigh 77.5% 94.2% 60.9% 0.932 504 15.9 $0.0289 2026-09-30 2 Gemini 4 Hydrogenhigh 76.1% 92.8% 59.4% 0.921 4,407 27.6 $0.0337 2026-11-14 1 GPT-6.2 Astrahigh 76.1% 92.2% 60.0% 0.936 335 9.0 $0.0164 2026-11-18 1 Gemini 4 Neonlow 75.8% 93.9% 57.8% 0.924 1,009 11.1 $0.0164 2026-11-19 1 Gemini 4 Carbonhigh 75.5% 93.9% 57.2% 0.808 10,067 175.9 $0.1014 2026-11-10 1 Muse Wonderhigh 74.8% 91.7% 57.8% 0.929 7,284 66.1 $0.1468 2026-10-30 1 GPT-6 Astramedium 74.7% 92.0% 57.5% 0.931 258 9.9 $0.0166 2026-09-30 2 Fable 5.2high 74.4% 90.5% 58.4% 0.926 776 10.6 $0.0369 2026-11-15 2 Muse Wondermedium 73.9% 91.1% 56.7% 0.926 5,352 47.7 $0.1082 2026-10-30 1 GPT-6.2 Astramedium 73.6% 92.8% 54.4% 0.920 97 3.8 $0.0068 2026-11-18 1 Fable 5.2low 73.1% 91.4% 54.9% 0.918 449 7.3 $0.0348 2026-11-15 2 Fable 5.2medium 72.9% 91.1% 54.7% 0.917 483 7.7 $0.0261 2026-11-15 2 Gemini 4 Carbonlow 72.5% 92.8% 52.2% 0.915 254 8.4 $0.0034 2026-11-10 1 Gemini 4 Hydrogenmedium 72.5% 90.6% 54.4% 0.918 2,236 14.4 $0.0174 2026-11-14 1 GPT-6.1 Solmedium 72.2% 91.1% 53.3% 0.919 231 7.3 $0.0030 2026-09-30 1 Gemini 4 Argonlow 71.1% 90.0% 52.2% 0.914 18 3.9 $0.0010 2026-10-30 1 GPT-6.2 Astralow 70.9% 90.6% 51.1% 0.907 42 2.5 $0.0046 2026-11-18 1 Gemini 4 Argonmedium 70.5% 92.2% 48.9% 0.907 135 5.7 $0.0022 2026-10-30 1 GPT-6.1 Solhigh 70.3% 87.8% 52.8% 0.924 458 12.3 $0.0053 2026-09-30 1 GPT-6 Astralow 70.0% 88.9% 51.1% 0.919 151 6.9 $0.0112 2026-09-30 2 Opus 5.5high 69.9% 87.1% 52.6% 0.919 598 8.6 $0.0165 2026-11-15 4 Muse Wonderlow 68.3% 88.3% 48.3% 0.914 3,289 29.5 $0.0669 2026-10-30 1 GPT-6.2 Solhigh 68.0% 88.3% 47.8% 0.907 137 2.8 $0.0021 2026-11-18 1 Opus 5.5medium 67.8% 85.0% 50.7% 0.914 486 7.5 $0.0140 2026-11-15 4 Fable 5.1high 67.3% 85.2% 49.5% 0.909 515 9.7 $0.0394 2026-10-22 2 Muse Emberhigh 67.3% 86.7% 47.8% 0.906 7,614 20.7 $0.0038 2026-11-10 1 GPT-6.1 Sollow 67.3% 86.7% 47.8% 0.913 156 6.0 $0.0023 2026-09-30 1 DeepSeek V4.1 Flashhigh 66.4% 80.6% 52.2% 0.919 7,276 74.2 $0.0088 2026-09-24 1 GPT-6.1 Terrahigh 66.4% 87.8% 45.0% 0.902 303 4.9 $0.0019 2026-10-28 1 Gemini 4 Hydrogenlow 66.4% 85.0% 47.8% 0.897 875 5.7 $0.0072 2026-11-14 1 Kimi K3high 64.7% 82.2% 47.2% 0.911 4,651 88.9 $0.0685 2026-09-24 1 Opus 5.5low 64.4% 83.3% 45.4% 0.906 318 5.8 $0.0146 2026-11-15 4 Fable 5.1medium 64.3% 82.9% 45.6% 0.897 390 8.1 $0.0340 2026-10-22 2 Sonnet 5.5high 63.9% 82.3% 45.6% 0.904 911 11.1 $0.0117 2026-10-22 2 GPT-6.2 Solmedium 62.8% 79.4% 46.1% 0.893 53 1.6 $0.0013 2026-11-18 1 Fable 5.1low 62.7% 82.8% 42.5% 0.889 305 7.1 $0.0354 2026-10-22 2 DeepSeek V4.1 Flashlow 62.3% 80.6% 43.9% 0.896 5,343 47.2 $0.0065 2026-09-24 1 DeepSeek V4.1 Flashmedium 62.2% 82.2% 42.2% 0.904 5,514 49.0 $0.0067 2026-09-24 1 Muse Embermedium 62.0% 77.2% 46.7% 0.905 5,090 10.7 $0.0026 2026-11-10 1 GPT-6 Solhigh 61.7% 78.3% 45.0% 0.893 679 13.7 $0.0075 2026-09-24 1 Muse Emberlow 60.8% 78.3% 43.3% 0.896 3,352 7.5 $0.0017 2026-11-10 1 Sonnet 5.5medium 60.7% 77.5% 43.9% 0.899 662 8.8 $0.0095 2026-10-22 2 Ripple 65B Previewmedium 60.6% 74.4% 46.7% 0.901 10,813 102.1 $0.0065 2026-11-10 1 Sonnet 5medium 60.3% 76.7% 43.9% 0.884 574 8.5 $0.0097 2026-09-24 1 Kimi K3.1 Prohigh 60.3% 80.0% 40.6% 0.893 1,657 42.5 $0.0175 2026-11-10 1 Ripple 65B Previewlow 60.0% 76.7% 43.3% 0.886 11,403 108.3 $0.0069 2026-11-10 1 Sonnet 5high 59.8% 78.9% 40.6% 0.894 644 9.2 $0.0104 2026-09-24 1 GPT-6.2 Sollow 59.4% 79.4% 39.4% 0.878 19 1.1 $0.0009 2026-11-18 1 GPT-6 Solmedium 58.9% 78.3% 39.4% 0.883 406 8.4 $0.0048 2026-09-24 1 Sonnet 5.5low 58.5% 77.5% 39.5% 0.886 591 8.3 $0.0098 2026-10-22 2 Kimi K3low 58.1% 76.7% 39.4% 0.857 1,769 34.4 $0.0264 2026-09-24 1 Ripple 65B Previewhigh 57.8% 74.4% 41.1% 0.900 10,530 96.2 $0.0064 2026-11-10 1 Sonnet 5low 57.5% 79.4% 35.6% 0.864 499 7.7 $0.0089 2026-09-24 1 Muse-Glimmer-30Bhigh 57.3% 75.6% 38.9% 0.872 9,615 131.8 $0.0145 2026-09-27 1 GPT-6.1 Terramedium 56.7% 77.8% 35.6% 0.874 187 8.5 $0.0013 2026-10-28 1 DeepSeek V4 Prohigh 55.0% 76.7% 33.3% 0.866 6,001 89.7 $0.0224 2026-09-24 1 DeepSeek V4 Prolow 54.7% 73.3% 36.1% 0.853 5,724 93.7 $0.0213 2026-09-24 1 DeepSeek V4 Promedium 53.8% 70.9% 36.7% 0.867 5,928 86.1 $0.0221 2026-09-24 1 Inklinghigh 52.8% 72.2% 33.3% 0.865 6,903 69.5 $0.0283 2026-11-10 1 Kimi K3medium 52.5% 72.2% 32.8% 0.858 1,743 33.5 $0.0262 2026-09-24 1 GPT-6 Sollow 52.3% 72.8% 31.7% 0.858 209 6.5 $0.0028 2026-09-24 1 GPT-6.1 Terralow 50.8% 73.3% 28.3% 0.853 97 2.5 $0.0009 2026-10-28 1 Nemotron 3 Ultramedium 50.0% 66.7% 33.3% 0.851 8,497 77.5 $0.0206 2026-10-30 1 Muse-Glimmer-30Bmedium 49.7% 66.1% 33.3% 0.845 5,814 63.1 $0.0088 2026-09-27 1 Nemotron 3 Ultrahigh 47.8% 65.6% 30.0% 0.837 8,711 77.8 $0.0211 2026-10-30 1 Nemotron 3 Ultralow 47.8% 68.3% 27.2% 0.832 8,902 79.5 $0.0215 2026-10-30 1 GPT-6 Lunahigh 46.4% 61.1% 31.7% 0.865 2,225 30.8 $0.0011 2026-09-24 1 MiniMax M3on 46.1% 65.0% 27.2% 0.801 11,824 111.1 $0.0143 2026-09-24 1 Muse-Glimmer-30Blow 42.8% 57.8% 27.8% 0.804 2,926 30.8 $0.0045 2026-09-27 1 Kimi K3.1 Prolow 42.8% 61.1% 24.4% 0.829 494 6.3 $0.0058 2026-11-10 1 Kimi K3.1 Promedium 42.8% 64.4% 21.1% 0.815 447 5.9 $0.0053 2026-11-10 1 GPT-6 Lunamedium 40.5% 58.9% 22.2% 0.814 1,272 15.9 $0.0007 2026-09-24 1 MiniMax M3.1 Flashhigh 39.8% 56.7% 22.8% 0.826 9,912 40.0 $0.0060 2026-10-27 1 Inklingmedium 39.5% 55.6% 23.3% 0.822 2,424 32.6 $0.0101 2026-11-10 1 MiniMax M3.1 Flashmedium 39.1% 55.0% 23.3% 0.814 7,456 33.9 $0.0045 2026-10-27 1 MiMo v2.5 Proon 38.6% 58.9% 18.3% 0.765 6,356 187.9 $0.0113 2026-10-22 1 Haiku 5.5high 38.4% 52.8% 23.9% 0.830 2,188 12.7 $0.0012 2026-10-22 1 GLM-5.3high 38.3% 57.2% 19.4% 0.786 3,081 31.6 $0.0141 2026-09-24 1 Qwen3.8-27Bhigh 37.2% 53.3% 21.1% 0.797 3,821 62.2 $0.0085 2026-09-27 1 Qwen3.8-27Blow 36.4% 52.2% 20.6% 0.792 3,350 42.8 $0.0075 2026-09-27 1 Haiku 4.5fixed 36.2% 55.6% 16.7% 0.781 10,594 100.0 $0.0537 2026-09-24 1 Qwen3.8-27Bmedium 35.0% 51.7% 18.3% 0.792 3,752 47.1 $0.0083 2026-09-27 1 GLM-5.3 Flashhigh 34.8% 47.8% 21.7% 0.777 2,237 32.4 $0.0012 2026-09-24 1 Haiku 5.5medium 34.7% 49.4% 20.0% 0.802 1,368 9.2 $0.0008 2026-10-22 1 Haiku 5.5low 34.5% 52.8% 16.1% 0.772 977 7.4 $0.0007 2026-10-22 1 GPT-6 Lunalow 31.7% 50.6% 12.8% 0.764 535 7.8 $0.0003 2026-09-24 1 MiniMax M3.1 Flashlow 31.6% 46.1% 17.2% 0.779 5,652 27.1 $0.0034 2026-10-27 1 Inklinglow 26.7% 42.8% 10.6% 0.703 375 6.3 $0.0018 2026-11-10 1 Nemotron 3.5 Super VLoff 26.4% 38.3% 14.4% 0.635 5,381 24.1 $0.0027 2026-10-30 1 GLM-5.3medium 26.2% 41.7% 10.6% 0.721 927 15.8 $0.0036 2026-09-24 1 GLM-5.3 Flashlow 25.0% 43.9% 6.1% 0.661 371 6.5 $0.0002 2026-09-24 1 GLM-5.3 Flashmedium 25.0% 42.8% 7.2% 0.677 451 8.0 $0.0003 2026-09-24 1 GLM-5.3low 24.4% 41.1% 7.8% 0.714 696 11.8 $0.0030 2026-09-24 1 Qwen3.6-35B-A3Bon 23.3% 34.4% 12.2% 0.719 5,047 68.2 $0.0051 2026-09-27 1 Qwen3.8-27Boff 12.2% 21.1% 3.3% 0.567 14 0.9 $0.0001 2026-09-27 1 Qwen3.6-35B-A3Boff 7.2% 13.3% 1.1% 0.482 14 0.9 $0.0001 2026-09-27 1
Download the data (JSON)
We created a secret benchmark to validate big AI labs’ claims about their LLMs. Welcome to
Some Bench.
The tasks will never be shared with anyone, so no one can cheat. A higher score is better; this
is all you need to read it.
We give you a piece of vague evidence you can use to develop your own bias about models.
Visit us often or subscribe to know what we think those numbers mean.
Caveats
Runs : repeated evaluation runs. By default the average is displayed, but you can click to
expand and see each run.
Fable 5.1 : some examples (around 8-9) triggered Fable 5.1 safety guardrails; the score
excludes them.
Seconds per task : wall time. Take it with a grain of salt: some settings (local vs API)
can't be compared.
Cost : estimated based on OpenRouter pricing, but the source of this estimation might vary.
What we think it means
Opus 5.5 and Fable 5.2 : is Opus 5.5 nerfed around the Fable 5.2 release? We think
yes, but what is interesting is how they do it.
Very verbose LLMs : do you remember the hype about open-source models catching up
with closed APIs? The magic was gone once we saw how many tokens they actually emit for the same
task. Are they really still using this trick?
Fewest tokens : can you get a reasonably good answer in just 18 tokens?!
Best value : the top score isn't the best buy. Count tokens.
Scores standard, harder variant exact, harder variant partial
Last change today