← all projects

Some Bench

in progress

A benchmark of how well LLMs can reason if they can't cheat.

Latest: New results: Gemini 4 Neon · · RSS

Family
Effort
0%20%40%60%80%100%1020501002005001K2K5K10KCompletion tokens per task (log scale)Score (mean of standard and harder variant exact)Gemini 4 Argon · highGemini 4 Carbon · mediumGPT-6 Astra · highGPT-6 Astra · mediumGPT-6.2 Astra · mediumGemini 4 Argon · lowQwen3.8-27B · off

Labelled are the runs that no other run beats on both axes: a higher score for less (Labels shows all of them). Hover any point for its numbers, click it to find its row in the table.

Runs
Gemini 4 Argonhigh78.9%96.7%61.1%0.9273,29144.5$0.03372026-10-301
Gemini 4 Carbonmedium78.3%93.9%62.8%0.9031,87438.4$0.01952026-11-101
Gemini 4 Neonhigh78.3%96.1%60.6%0.9367,39462.0$0.11212026-11-191
Gemini 4 Neonmedium78.3%96.7%60.0%0.9312,45523.6$0.03812026-11-191
GPT-6 Astrahigh77.5%94.2%60.9%0.93250415.9$0.02892026-09-30
Gemini 4 Hydrogenhigh76.1%92.8%59.4%0.9214,40727.6$0.03372026-11-141
GPT-6.2 Astrahigh76.1%92.2%60.0%0.9363359.0$0.01642026-11-181
Gemini 4 Neonlow75.8%93.9%57.8%0.9241,00911.1$0.01642026-11-191
Gemini 4 Carbonhigh75.5%93.9%57.2%0.80810,067175.9$0.10142026-11-101
Muse Wonderhigh74.8%91.7%57.8%0.9297,28466.1$0.14682026-10-301
GPT-6 Astramedium74.7%92.0%57.5%0.9312589.9$0.01662026-09-30
Fable 5.2high74.4%90.5%58.4%0.92677610.6$0.03692026-11-15
Muse Wondermedium73.9%91.1%56.7%0.9265,35247.7$0.10822026-10-301
GPT-6.2 Astramedium73.6%92.8%54.4%0.920973.8$0.00682026-11-181
Fable 5.2low73.1%91.4%54.9%0.9184497.3$0.03482026-11-15
Fable 5.2medium72.9%91.1%54.7%0.9174837.7$0.02612026-11-15
Gemini 4 Carbonlow72.5%92.8%52.2%0.9152548.4$0.00342026-11-101
Gemini 4 Hydrogenmedium72.5%90.6%54.4%0.9182,23614.4$0.01742026-11-141
GPT-6.1 Solmedium72.2%91.1%53.3%0.9192317.3$0.00302026-09-301
Gemini 4 Argonlow71.1%90.0%52.2%0.914183.9$0.00102026-10-301
GPT-6.2 Astralow70.9%90.6%51.1%0.907422.5$0.00462026-11-181
Gemini 4 Argonmedium70.5%92.2%48.9%0.9071355.7$0.00222026-10-301
GPT-6.1 Solhigh70.3%87.8%52.8%0.92445812.3$0.00532026-09-301
GPT-6 Astralow70.0%88.9%51.1%0.9191516.9$0.01122026-09-30
Opus 5.5high69.9%87.1%52.6%0.9195988.6$0.01652026-11-15
Muse Wonderlow68.3%88.3%48.3%0.9143,28929.5$0.06692026-10-301
GPT-6.2 Solhigh68.0%88.3%47.8%0.9071372.8$0.00212026-11-181
Opus 5.5medium67.8%85.0%50.7%0.9144867.5$0.01402026-11-15
Fable 5.1high67.3%85.2%49.5%0.9095159.7$0.03942026-10-22
Muse Emberhigh67.3%86.7%47.8%0.9067,61420.7$0.00382026-11-101
GPT-6.1 Sollow67.3%86.7%47.8%0.9131566.0$0.00232026-09-301
DeepSeek V4.1 Flashhigh66.4%80.6%52.2%0.9197,27674.2$0.00882026-09-241
GPT-6.1 Terrahigh66.4%87.8%45.0%0.9023034.9$0.00192026-10-281
Gemini 4 Hydrogenlow66.4%85.0%47.8%0.8978755.7$0.00722026-11-141
Kimi K3high64.7%82.2%47.2%0.9114,65188.9$0.06852026-09-241
Opus 5.5low64.4%83.3%45.4%0.9063185.8$0.01462026-11-15
Fable 5.1medium64.3%82.9%45.6%0.8973908.1$0.03402026-10-22
Sonnet 5.5high63.9%82.3%45.6%0.90491111.1$0.01172026-10-22
GPT-6.2 Solmedium62.8%79.4%46.1%0.893531.6$0.00132026-11-181
Fable 5.1low62.7%82.8%42.5%0.8893057.1$0.03542026-10-22
DeepSeek V4.1 Flashlow62.3%80.6%43.9%0.8965,34347.2$0.00652026-09-241
DeepSeek V4.1 Flashmedium62.2%82.2%42.2%0.9045,51449.0$0.00672026-09-241
Muse Embermedium62.0%77.2%46.7%0.9055,09010.7$0.00262026-11-101
GPT-6 Solhigh61.7%78.3%45.0%0.89367913.7$0.00752026-09-241
Muse Emberlow60.8%78.3%43.3%0.8963,3527.5$0.00172026-11-101
Sonnet 5.5medium60.7%77.5%43.9%0.8996628.8$0.00952026-10-22
Ripple 65B Previewmedium60.6%74.4%46.7%0.90110,813102.1$0.00652026-11-101
Sonnet 5medium60.3%76.7%43.9%0.8845748.5$0.00972026-09-241
Kimi K3.1 Prohigh60.3%80.0%40.6%0.8931,65742.5$0.01752026-11-101
Ripple 65B Previewlow60.0%76.7%43.3%0.88611,403108.3$0.00692026-11-101
Sonnet 5high59.8%78.9%40.6%0.8946449.2$0.01042026-09-241
GPT-6.2 Sollow59.4%79.4%39.4%0.878191.1$0.00092026-11-181
GPT-6 Solmedium58.9%78.3%39.4%0.8834068.4$0.00482026-09-241
Sonnet 5.5low58.5%77.5%39.5%0.8865918.3$0.00982026-10-22
Kimi K3low58.1%76.7%39.4%0.8571,76934.4$0.02642026-09-241
Ripple 65B Previewhigh57.8%74.4%41.1%0.90010,53096.2$0.00642026-11-101
Sonnet 5low57.5%79.4%35.6%0.8644997.7$0.00892026-09-241
Muse-Glimmer-30Bhigh57.3%75.6%38.9%0.8729,615131.8$0.01452026-09-271
GPT-6.1 Terramedium56.7%77.8%35.6%0.8741878.5$0.00132026-10-281
DeepSeek V4 Prohigh55.0%76.7%33.3%0.8666,00189.7$0.02242026-09-241
DeepSeek V4 Prolow54.7%73.3%36.1%0.8535,72493.7$0.02132026-09-241
DeepSeek V4 Promedium53.8%70.9%36.7%0.8675,92886.1$0.02212026-09-241
Inklinghigh52.8%72.2%33.3%0.8656,90369.5$0.02832026-11-101
Kimi K3medium52.5%72.2%32.8%0.8581,74333.5$0.02622026-09-241
GPT-6 Sollow52.3%72.8%31.7%0.8582096.5$0.00282026-09-241
GPT-6.1 Terralow50.8%73.3%28.3%0.853972.5$0.00092026-10-281
Nemotron 3 Ultramedium50.0%66.7%33.3%0.8518,49777.5$0.02062026-10-301
Muse-Glimmer-30Bmedium49.7%66.1%33.3%0.8455,81463.1$0.00882026-09-271
Nemotron 3 Ultrahigh47.8%65.6%30.0%0.8378,71177.8$0.02112026-10-301
Nemotron 3 Ultralow47.8%68.3%27.2%0.8328,90279.5$0.02152026-10-301
GPT-6 Lunahigh46.4%61.1%31.7%0.8652,22530.8$0.00112026-09-241
MiniMax M3on46.1%65.0%27.2%0.80111,824111.1$0.01432026-09-241
Muse-Glimmer-30Blow42.8%57.8%27.8%0.8042,92630.8$0.00452026-09-271
Kimi K3.1 Prolow42.8%61.1%24.4%0.8294946.3$0.00582026-11-101
Kimi K3.1 Promedium42.8%64.4%21.1%0.8154475.9$0.00532026-11-101
GPT-6 Lunamedium40.5%58.9%22.2%0.8141,27215.9$0.00072026-09-241
MiniMax M3.1 Flashhigh39.8%56.7%22.8%0.8269,91240.0$0.00602026-10-271
Inklingmedium39.5%55.6%23.3%0.8222,42432.6$0.01012026-11-101
MiniMax M3.1 Flashmedium39.1%55.0%23.3%0.8147,45633.9$0.00452026-10-271
MiMo v2.5 Proon38.6%58.9%18.3%0.7656,356187.9$0.01132026-10-221
Haiku 5.5high38.4%52.8%23.9%0.8302,18812.7$0.00122026-10-221
GLM-5.3high38.3%57.2%19.4%0.7863,08131.6$0.01412026-09-241
Qwen3.8-27Bhigh37.2%53.3%21.1%0.7973,82162.2$0.00852026-09-271
Qwen3.8-27Blow36.4%52.2%20.6%0.7923,35042.8$0.00752026-09-271
Haiku 4.5fixed36.2%55.6%16.7%0.78110,594100.0$0.05372026-09-241
Qwen3.8-27Bmedium35.0%51.7%18.3%0.7923,75247.1$0.00832026-09-271
GLM-5.3 Flashhigh34.8%47.8%21.7%0.7772,23732.4$0.00122026-09-241
Haiku 5.5medium34.7%49.4%20.0%0.8021,3689.2$0.00082026-10-221
Haiku 5.5low34.5%52.8%16.1%0.7729777.4$0.00072026-10-221
GPT-6 Lunalow31.7%50.6%12.8%0.7645357.8$0.00032026-09-241
MiniMax M3.1 Flashlow31.6%46.1%17.2%0.7795,65227.1$0.00342026-10-271
Inklinglow26.7%42.8%10.6%0.7033756.3$0.00182026-11-101
Nemotron 3.5 Super VLoff26.4%38.3%14.4%0.6355,38124.1$0.00272026-10-301
GLM-5.3medium26.2%41.7%10.6%0.72192715.8$0.00362026-09-241
GLM-5.3 Flashlow25.0%43.9%6.1%0.6613716.5$0.00022026-09-241
GLM-5.3 Flashmedium25.0%42.8%7.2%0.6774518.0$0.00032026-09-241
GLM-5.3low24.4%41.1%7.8%0.71469611.8$0.00302026-09-241
Qwen3.6-35B-A3Bon23.3%34.4%12.2%0.7195,04768.2$0.00512026-09-271
Qwen3.8-27Boff12.2%21.1%3.3%0.567140.9$0.00012026-09-271
Qwen3.6-35B-A3Boff7.2%13.3%1.1%0.482140.9$0.00012026-09-271

Download the data (JSON)

We created a secret benchmark to validate big AI labs’ claims about their LLMs. Welcome to Some Bench.

The tasks will never be shared with anyone, so no one can cheat. A higher score is better; this is all you need to read it.

We give you a piece of vague evidence you can use to develop your own bias about models. Visit us often or subscribe to know what we think those numbers mean.

Caveats

  • Runs: repeated evaluation runs. By default the average is displayed, but you can click to expand and see each run.
  • Fable 5.1: some examples (around 8-9) triggered Fable 5.1 safety guardrails; the score excludes them.
  • Seconds per task: wall time. Take it with a grain of salt: some settings (local vs API) can't be compared.
  • Cost: estimated based on OpenRouter pricing, but the source of this estimation might vary.

What we think it means

  • Opus 5.5 and Fable 5.2: is Opus 5.5 nerfed around the Fable 5.2 release? We think yes, but what is interesting is how they do it.
  • Very verbose LLMs: do you remember the hype about open-source models catching up with closed APIs? The magic was gone once we saw how many tokens they actually emit for the same task. Are they really still using this trick?
  • Fewest tokens: can you get a reasonably good answer in just 18 tokens?!
  • Best value: the top score isn't the best buy. Count tokens.

Changelog

Full changelog (10) → Subscribe (RSS)