← Some Bench

Some Bench

in progress

A benchmark of how well LLMs can reason if they can't cheat.

Family
Effort
0%20%40%60%80%100%1020501002005001K2K5K10KCompletion tokens per task (log scale)Score (mean of standard and harder variant exact)Gemini 4 Argon · highGemini 4 Carbon · mediumGPT-6 Astra · highGPT-6 Astra · mediumGPT-6.2 Astra · mediumGemini 4 Argon · lowQwen3.8-27B · off

Labelled are the runs that no other run beats on both axes: a higher score for less (Labels shows all of them). Hover any point for its numbers, click it to find its row in the table.

Runs
Gemini 4 Argonhigh78.9%96.7%61.1%0.9273,29144.5$0.03372026-10-301
Gemini 4 Carbonmedium78.3%93.9%62.8%0.9031,87438.4$0.01952026-11-101
Gemini 4 Neonhigh78.3%96.1%60.6%0.9367,39462.0$0.11212026-11-191
Gemini 4 Neonmedium78.3%96.7%60.0%0.9312,45523.6$0.03812026-11-191
GPT-6 Astrahigh77.5%94.2%60.9%0.93250415.9$0.02892026-09-30
Gemini 4 Hydrogenhigh76.1%92.8%59.4%0.9214,40727.6$0.03372026-11-141
GPT-6.2 Astrahigh76.1%92.2%60.0%0.9363359.0$0.01642026-11-181
Gemini 4 Neonlow75.8%93.9%57.8%0.9241,00911.1$0.01642026-11-191
Gemini 4 Carbonhigh75.5%93.9%57.2%0.80810,067175.9$0.10142026-11-101
Muse Wonderhigh74.8%91.7%57.8%0.9297,28466.1$0.14682026-10-301
GPT-6 Astramedium74.7%92.0%57.5%0.9312589.9$0.01662026-09-30
Fable 5.2high74.4%90.5%58.4%0.92677610.6$0.03692026-11-15
Muse Wondermedium73.9%91.1%56.7%0.9265,35247.7$0.10822026-10-301
GPT-6.2 Astramedium73.6%92.8%54.4%0.920973.8$0.00682026-11-181
Fable 5.2low73.1%91.4%54.9%0.9184497.3$0.03482026-11-15
Fable 5.2medium72.9%91.1%54.7%0.9174837.7$0.02612026-11-15
Gemini 4 Carbonlow72.5%92.8%52.2%0.9152548.4$0.00342026-11-101
Gemini 4 Hydrogenmedium72.5%90.6%54.4%0.9182,23614.4$0.01742026-11-141
GPT-6.1 Solmedium72.2%91.1%53.3%0.9192317.3$0.00302026-09-301
Gemini 4 Argonlow71.1%90.0%52.2%0.914183.9$0.00102026-10-301
GPT-6.2 Astralow70.9%90.6%51.1%0.907422.5$0.00462026-11-181
Gemini 4 Argonmedium70.5%92.2%48.9%0.9071355.7$0.00222026-10-301
GPT-6.1 Solhigh70.3%87.8%52.8%0.92445812.3$0.00532026-09-301
GPT-6 Astralow70.0%88.9%51.1%0.9191516.9$0.01122026-09-30
Opus 5.5high69.9%87.1%52.6%0.9195988.6$0.01652026-11-15
Muse Wonderlow68.3%88.3%48.3%0.9143,28929.5$0.06692026-10-301
GPT-6.2 Solhigh68.0%88.3%47.8%0.9071372.8$0.00212026-11-181
Opus 5.5medium67.8%85.0%50.7%0.9144867.5$0.01402026-11-15
Fable 5.1high67.3%85.2%49.5%0.9095159.7$0.03942026-10-22
Muse Emberhigh67.3%86.7%47.8%0.9067,61420.7$0.00382026-11-101
GPT-6.1 Sollow67.3%86.7%47.8%0.9131566.0$0.00232026-09-301
DeepSeek V4.1 Flashhigh66.4%80.6%52.2%0.9197,27674.2$0.00882026-09-241
GPT-6.1 Terrahigh66.4%87.8%45.0%0.9023034.9$0.00192026-10-281
Gemini 4 Hydrogenlow66.4%85.0%47.8%0.8978755.7$0.00722026-11-141
Kimi K3high64.7%82.2%47.2%0.9114,65188.9$0.06852026-09-241
Opus 5.5low64.4%83.3%45.4%0.9063185.8$0.01462026-11-15
Fable 5.1medium64.3%82.9%45.6%0.8973908.1$0.03402026-10-22
Sonnet 5.5high63.9%82.3%45.6%0.90491111.1$0.01172026-10-22
GPT-6.2 Solmedium62.8%79.4%46.1%0.893531.6$0.00132026-11-181
Fable 5.1low62.7%82.8%42.5%0.8893057.1$0.03542026-10-22
DeepSeek V4.1 Flashlow62.3%80.6%43.9%0.8965,34347.2$0.00652026-09-241
DeepSeek V4.1 Flashmedium62.2%82.2%42.2%0.9045,51449.0$0.00672026-09-241
Muse Embermedium62.0%77.2%46.7%0.9055,09010.7$0.00262026-11-101
GPT-6 Solhigh61.7%78.3%45.0%0.89367913.7$0.00752026-09-241
Muse Emberlow60.8%78.3%43.3%0.8963,3527.5$0.00172026-11-101
Sonnet 5.5medium60.7%77.5%43.9%0.8996628.8$0.00952026-10-22
Ripple 65B Previewmedium60.6%74.4%46.7%0.90110,813102.1$0.00652026-11-101
Sonnet 5medium60.3%76.7%43.9%0.8845748.5$0.00972026-09-241
Kimi K3.1 Prohigh60.3%80.0%40.6%0.8931,65742.5$0.01752026-11-101
Ripple 65B Previewlow60.0%76.7%43.3%0.88611,403108.3$0.00692026-11-101
Sonnet 5high59.8%78.9%40.6%0.8946449.2$0.01042026-09-241
GPT-6.2 Sollow59.4%79.4%39.4%0.878191.1$0.00092026-11-181
GPT-6 Solmedium58.9%78.3%39.4%0.8834068.4$0.00482026-09-241
Sonnet 5.5low58.5%77.5%39.5%0.8865918.3$0.00982026-10-22
Kimi K3low58.1%76.7%39.4%0.8571,76934.4$0.02642026-09-241
Ripple 65B Previewhigh57.8%74.4%41.1%0.90010,53096.2$0.00642026-11-101
Sonnet 5low57.5%79.4%35.6%0.8644997.7$0.00892026-09-241
Muse-Glimmer-30Bhigh57.3%75.6%38.9%0.8729,615131.8$0.01452026-09-271
GPT-6.1 Terramedium56.7%77.8%35.6%0.8741878.5$0.00132026-10-281
DeepSeek V4 Prohigh55.0%76.7%33.3%0.8666,00189.7$0.02242026-09-241
DeepSeek V4 Prolow54.7%73.3%36.1%0.8535,72493.7$0.02132026-09-241
DeepSeek V4 Promedium53.8%70.9%36.7%0.8675,92886.1$0.02212026-09-241
Inklinghigh52.8%72.2%33.3%0.8656,90369.5$0.02832026-11-101
Kimi K3medium52.5%72.2%32.8%0.8581,74333.5$0.02622026-09-241
GPT-6 Sollow52.3%72.8%31.7%0.8582096.5$0.00282026-09-241
GPT-6.1 Terralow50.8%73.3%28.3%0.853972.5$0.00092026-10-281
Nemotron 3 Ultramedium50.0%66.7%33.3%0.8518,49777.5$0.02062026-10-301
Muse-Glimmer-30Bmedium49.7%66.1%33.3%0.8455,81463.1$0.00882026-09-271
Nemotron 3 Ultrahigh47.8%65.6%30.0%0.8378,71177.8$0.02112026-10-301
Nemotron 3 Ultralow47.8%68.3%27.2%0.8328,90279.5$0.02152026-10-301
GPT-6 Lunahigh46.4%61.1%31.7%0.8652,22530.8$0.00112026-09-241
MiniMax M3on46.1%65.0%27.2%0.80111,824111.1$0.01432026-09-241
Muse-Glimmer-30Blow42.8%57.8%27.8%0.8042,92630.8$0.00452026-09-271
Kimi K3.1 Prolow42.8%61.1%24.4%0.8294946.3$0.00582026-11-101
Kimi K3.1 Promedium42.8%64.4%21.1%0.8154475.9$0.00532026-11-101
GPT-6 Lunamedium40.5%58.9%22.2%0.8141,27215.9$0.00072026-09-241
MiniMax M3.1 Flashhigh39.8%56.7%22.8%0.8269,91240.0$0.00602026-10-271
Inklingmedium39.5%55.6%23.3%0.8222,42432.6$0.01012026-11-101
MiniMax M3.1 Flashmedium39.1%55.0%23.3%0.8147,45633.9$0.00452026-10-271
MiMo v2.5 Proon38.6%58.9%18.3%0.7656,356187.9$0.01132026-10-221
Haiku 5.5high38.4%52.8%23.9%0.8302,18812.7$0.00122026-10-221
GLM-5.3high38.3%57.2%19.4%0.7863,08131.6$0.01412026-09-241
Qwen3.8-27Bhigh37.2%53.3%21.1%0.7973,82162.2$0.00852026-09-271
Qwen3.8-27Blow36.4%52.2%20.6%0.7923,35042.8$0.00752026-09-271
Haiku 4.5fixed36.2%55.6%16.7%0.78110,594100.0$0.05372026-09-241
Qwen3.8-27Bmedium35.0%51.7%18.3%0.7923,75247.1$0.00832026-09-271
GLM-5.3 Flashhigh34.8%47.8%21.7%0.7772,23732.4$0.00122026-09-241
Haiku 5.5medium34.7%49.4%20.0%0.8021,3689.2$0.00082026-10-221
Haiku 5.5low34.5%52.8%16.1%0.7729777.4$0.00072026-10-221
GPT-6 Lunalow31.7%50.6%12.8%0.7645357.8$0.00032026-09-241
MiniMax M3.1 Flashlow31.6%46.1%17.2%0.7795,65227.1$0.00342026-10-271
Inklinglow26.7%42.8%10.6%0.7033756.3$0.00182026-11-101
Nemotron 3.5 Super VLoff26.4%38.3%14.4%0.6355,38124.1$0.00272026-10-301
GLM-5.3medium26.2%41.7%10.6%0.72192715.8$0.00362026-09-241
GLM-5.3 Flashlow25.0%43.9%6.1%0.6613716.5$0.00022026-09-241
GLM-5.3 Flashmedium25.0%42.8%7.2%0.6774518.0$0.00032026-09-241
GLM-5.3low24.4%41.1%7.8%0.71469611.8$0.00302026-09-241
Qwen3.6-35B-A3Bon23.3%34.4%12.2%0.7195,04768.2$0.00512026-09-271
Qwen3.8-27Boff12.2%21.1%3.3%0.567140.9$0.00012026-09-271
Qwen3.6-35B-A3Boff7.2%13.3%1.1%0.482140.9$0.00012026-09-271

Download the data (JSON)

Changelog

Releases and progress. Subscribe to Some Bench (RSS)

results

New results: Gemini 4 Neon

Added Gemini 4 Neon. Best of them: Gemini 4 Neon high, 78.3%.

Added

Model · effort Score Tokens per task Seconds per task Cost per test
Gemini 4 Neon · high 78.3% 7,394 62.0 $0.1121
Gemini 4 Neon · medium 78.3% 2,455 23.6 $0.0381
Gemini 4 Neon · low 75.8% 1,008 11.1 $0.0164

results

New results: GPT-6.2 Astra and GPT-6.2 Sol

Added GPT-6.2 Astra and GPT-6.2 Sol. Best of them: GPT-6.2 Astra high, 76.1%.

Added

Model · effort Score Tokens per task Seconds per task Cost per test
GPT-6.2 Astra · high 76.1% 335 9.0 $0.0164
GPT-6.2 Astra · medium 73.6% 97 3.8 $0.0068
GPT-6.2 Astra · low 70.9% 42 2.5 $0.0046
GPT-6.2 Sol · high 68.0% 137 2.8 $0.0021
GPT-6.2 Sol · medium 62.8% 52 1.6 $0.0013
GPT-6.2 Sol · low 59.4% 18 1.1 $0.0009

results

New results: Gemini 4 Carbon and Gemini 4 Hydrogen

Added Gemini 4 Carbon and Gemini 4 Hydrogen. Best of them: Gemini 4 Carbon medium, 78.3%. Fable 5.2 and Opus 5.5 were run again.

Added

Model · effort Score Tokens per task Seconds per task Cost per test
Gemini 4 Carbon · medium 78.3% 1,874 38.4 $0.0195
Gemini 4 Hydrogen · high 76.1% 4,406 27.6 $0.0337
Gemini 4 Carbon · high 75.5% 10,067 175.9 $0.1014
Gemini 4 Carbon · low 72.5% 254 8.4 $0.0034
Gemini 4 Hydrogen · medium 72.5% 2,236 14.4 $0.0174
Gemini 4 Hydrogen · low 66.4% 875 5.7 $0.0072

Run again

The page shows the average of all runs; these are the newest run against the average before it.

Model · effort Score Tokens per task
Fable 5.2 · high 75.8% → 73.0% 798 → 753
Fable 5.2 · low 72.5% → 73.8% 474 → 424
Fable 5.2 · medium 73.3% → 72.5% 509 → 457
Opus 5.5 · high 70.3% → 68.6% 608 → 568
Opus 5.5 · low 65.1% → 62.2% 321 → 308
Opus 5.5 · medium 68.0% → 67.5% 492 → 465

results

New results: Ripple 65B Preview and Inkling

Added Ripple 65B Preview and Inkling. Best of them: Ripple 65B Preview medium, 60.6%.

Added

Model · effort Score Tokens per task Seconds per task Cost per test
Ripple 65B Preview · medium 60.6% 10,813 102.1 $0.0065
Ripple 65B Preview · low 60.0% 11,402 108.3 $0.0069
Ripple 65B Preview · high 57.8% 10,530 96.2 $0.0064
Inkling · high 52.8% 6,903 69.5 $0.0283
Inkling · medium 39.5% 2,424 32.6 $0.0101
Inkling · low 26.7% 375 6.3 $0.0018

remark

Very verbose LLMs

Old trick played again: very very verbose LLMs

Read the remark →

results

New results: Muse Ember and Kimi K3.1 Pro

Added Muse Ember and Kimi K3.1 Pro. Best of them: Muse Ember high, 67.2%.

Added

Model · effort Score Tokens per task Seconds per task Cost per test
Muse Ember · high 67.2% 7,614 20.7 $0.0038
Muse Ember · medium 62.0% 5,090 10.7 $0.0026
Muse Ember · low 60.8% 3,352 7.5 $0.0017
Kimi K3.1 Pro · high 60.3% 1,656 42.5 $0.0175
Kimi K3.1 Pro · low 42.8% 494 6.3 $0.0058
Kimi K3.1 Pro · medium 42.8% 446 5.9 $0.0053

remark

Fewest tokens

71% on just 18 tokens

Read the remark →

results

New results: Fable 5.2, Muse Wonder, Nemotron 3 Ultra and Nemotron 3.5 Super VL

Added Fable 5.2, Muse Wonder, Nemotron 3 Ultra and Nemotron 3.5 Super VL. Best of them: Fable 5.2 high, 75.8%. Opus 5.5 was run again.

Added

Model · effort Score Tokens per task Seconds per task Cost per test
Fable 5.2 · high 75.8% 798 9.9 $0.0372
Muse Wonder · high 74.8% 7,284 66.1 $0.1468
Muse Wonder · medium 73.9% 5,352 47.7 $0.1082
Fable 5.2 · medium 73.3% 509 7.1 $0.0265
Fable 5.2 · low 72.5% 474 6.8 $0.0353
Muse Wonder · low 68.3% 3,289 29.5 $0.0669
Nemotron 3 Ultra · medium 50.0% 8,496 77.5 $0.0206
Nemotron 3 Ultra · high 47.8% 8,710 77.8 $0.0211
Nemotron 3 Ultra · low 47.8% 8,902 79.5 $0.0215
Nemotron 3.5 Super VL · off 26.4% 5,380 24.1 $0.0027

Run again

The page shows the average of all runs; these are the newest run against the average before it.

Model · effort Score Tokens per task
Opus 5.5 · high 71.4% → 68.0% 636 → 552
Opus 5.5 · low 65.1% → 65.0% 322 → 319
Opus 5.5 · medium 67.8% → 68.3% 503 → 472

remark

Opus 5.5 and Fable 5.2

Is Opus 5.5 nerfed after Fable 5.2's release?

Read the remark →

remark

Best value

The top score isn't the best buy

Read the remark →