← Some Bench

Some Bench

in progress

A benchmark of how well LLMs can reason if they can't cheat.

Family
Effort
0%20%40%60%80%100%1020501002005001K2K5K10KCompletion tokens per task (log scale)Score (mean of standard and harder variant exact)Gemini 4 Argon · highGemini 4 Carbon · mediumGPT-6 Astra · highGPT-6 Astra · mediumGPT-6.2 Astra · mediumGemini 4 Argon · lowQwen3.8-27B · off

Labelled are the runs that no other run beats on both axes: a higher score for less (Labels shows all of them). Hover any point for its numbers, click it to find its row in the table.

Runs
Gemini 4 Argonhigh78.9%96.7%61.1%0.9273,29144.5$0.03372026-10-301
Gemini 4 Carbonmedium78.3%93.9%62.8%0.9031,87438.4$0.01952026-11-101
Gemini 4 Neonhigh78.3%96.1%60.6%0.9367,39462.0$0.11212026-11-191
Gemini 4 Neonmedium78.3%96.7%60.0%0.9312,45523.6$0.03812026-11-191
GPT-6 Astrahigh77.5%94.2%60.9%0.93250415.9$0.02892026-09-30
Gemini 4 Hydrogenhigh76.1%92.8%59.4%0.9214,40727.6$0.03372026-11-141
GPT-6.2 Astrahigh76.1%92.2%60.0%0.9363359.0$0.01642026-11-181
Gemini 4 Neonlow75.8%93.9%57.8%0.9241,00911.1$0.01642026-11-191
Gemini 4 Carbonhigh75.5%93.9%57.2%0.80810,067175.9$0.10142026-11-101
Muse Wonderhigh74.8%91.7%57.8%0.9297,28466.1$0.14682026-10-301
GPT-6 Astramedium74.7%92.0%57.5%0.9312589.9$0.01662026-09-30
Fable 5.2high74.4%90.5%58.4%0.92677610.6$0.03692026-11-15
Muse Wondermedium73.9%91.1%56.7%0.9265,35247.7$0.10822026-10-301
GPT-6.2 Astramedium73.6%92.8%54.4%0.920973.8$0.00682026-11-181
Fable 5.2low73.1%91.4%54.9%0.9184497.3$0.03482026-11-15
Fable 5.2medium72.9%91.1%54.7%0.9174837.7$0.02612026-11-15
Gemini 4 Carbonlow72.5%92.8%52.2%0.9152548.4$0.00342026-11-101
Gemini 4 Hydrogenmedium72.5%90.6%54.4%0.9182,23614.4$0.01742026-11-141
GPT-6.1 Solmedium72.2%91.1%53.3%0.9192317.3$0.00302026-09-301
Gemini 4 Argonlow71.1%90.0%52.2%0.914183.9$0.00102026-10-301
GPT-6.2 Astralow70.9%90.6%51.1%0.907422.5$0.00462026-11-181
Gemini 4 Argonmedium70.5%92.2%48.9%0.9071355.7$0.00222026-10-301
GPT-6.1 Solhigh70.3%87.8%52.8%0.92445812.3$0.00532026-09-301
GPT-6 Astralow70.0%88.9%51.1%0.9191516.9$0.01122026-09-30
Opus 5.5high69.9%87.1%52.6%0.9195988.6$0.01652026-11-15
Muse Wonderlow68.3%88.3%48.3%0.9143,28929.5$0.06692026-10-301
GPT-6.2 Solhigh68.0%88.3%47.8%0.9071372.8$0.00212026-11-181
Opus 5.5medium67.8%85.0%50.7%0.9144867.5$0.01402026-11-15
Fable 5.1high67.3%85.2%49.5%0.9095159.7$0.03942026-10-22
Muse Emberhigh67.3%86.7%47.8%0.9067,61420.7$0.00382026-11-101
GPT-6.1 Sollow67.3%86.7%47.8%0.9131566.0$0.00232026-09-301
DeepSeek V4.1 Flashhigh66.4%80.6%52.2%0.9197,27674.2$0.00882026-09-241
GPT-6.1 Terrahigh66.4%87.8%45.0%0.9023034.9$0.00192026-10-281
Gemini 4 Hydrogenlow66.4%85.0%47.8%0.8978755.7$0.00722026-11-141
Kimi K3high64.7%82.2%47.2%0.9114,65188.9$0.06852026-09-241
Opus 5.5low64.4%83.3%45.4%0.9063185.8$0.01462026-11-15
Fable 5.1medium64.3%82.9%45.6%0.8973908.1$0.03402026-10-22
Sonnet 5.5high63.9%82.3%45.6%0.90491111.1$0.01172026-10-22
GPT-6.2 Solmedium62.8%79.4%46.1%0.893531.6$0.00132026-11-181
Fable 5.1low62.7%82.8%42.5%0.8893057.1$0.03542026-10-22
DeepSeek V4.1 Flashlow62.3%80.6%43.9%0.8965,34347.2$0.00652026-09-241
DeepSeek V4.1 Flashmedium62.2%82.2%42.2%0.9045,51449.0$0.00672026-09-241
Muse Embermedium62.0%77.2%46.7%0.9055,09010.7$0.00262026-11-101
GPT-6 Solhigh61.7%78.3%45.0%0.89367913.7$0.00752026-09-241
Muse Emberlow60.8%78.3%43.3%0.8963,3527.5$0.00172026-11-101
Sonnet 5.5medium60.7%77.5%43.9%0.8996628.8$0.00952026-10-22
Ripple 65B Previewmedium60.6%74.4%46.7%0.90110,813102.1$0.00652026-11-101
Sonnet 5medium60.3%76.7%43.9%0.8845748.5$0.00972026-09-241
Kimi K3.1 Prohigh60.3%80.0%40.6%0.8931,65742.5$0.01752026-11-101
Ripple 65B Previewlow60.0%76.7%43.3%0.88611,403108.3$0.00692026-11-101
Sonnet 5high59.8%78.9%40.6%0.8946449.2$0.01042026-09-241
GPT-6.2 Sollow59.4%79.4%39.4%0.878191.1$0.00092026-11-181
GPT-6 Solmedium58.9%78.3%39.4%0.8834068.4$0.00482026-09-241
Sonnet 5.5low58.5%77.5%39.5%0.8865918.3$0.00982026-10-22
Kimi K3low58.1%76.7%39.4%0.8571,76934.4$0.02642026-09-241
Ripple 65B Previewhigh57.8%74.4%41.1%0.90010,53096.2$0.00642026-11-101
Sonnet 5low57.5%79.4%35.6%0.8644997.7$0.00892026-09-241
Muse-Glimmer-30Bhigh57.3%75.6%38.9%0.8729,615131.8$0.01452026-09-271
GPT-6.1 Terramedium56.7%77.8%35.6%0.8741878.5$0.00132026-10-281
DeepSeek V4 Prohigh55.0%76.7%33.3%0.8666,00189.7$0.02242026-09-241
DeepSeek V4 Prolow54.7%73.3%36.1%0.8535,72493.7$0.02132026-09-241
DeepSeek V4 Promedium53.8%70.9%36.7%0.8675,92886.1$0.02212026-09-241
Inklinghigh52.8%72.2%33.3%0.8656,90369.5$0.02832026-11-101
Kimi K3medium52.5%72.2%32.8%0.8581,74333.5$0.02622026-09-241
GPT-6 Sollow52.3%72.8%31.7%0.8582096.5$0.00282026-09-241
GPT-6.1 Terralow50.8%73.3%28.3%0.853972.5$0.00092026-10-281
Nemotron 3 Ultramedium50.0%66.7%33.3%0.8518,49777.5$0.02062026-10-301
Muse-Glimmer-30Bmedium49.7%66.1%33.3%0.8455,81463.1$0.00882026-09-271
Nemotron 3 Ultrahigh47.8%65.6%30.0%0.8378,71177.8$0.02112026-10-301
Nemotron 3 Ultralow47.8%68.3%27.2%0.8328,90279.5$0.02152026-10-301
GPT-6 Lunahigh46.4%61.1%31.7%0.8652,22530.8$0.00112026-09-241
MiniMax M3on46.1%65.0%27.2%0.80111,824111.1$0.01432026-09-241
Muse-Glimmer-30Blow42.8%57.8%27.8%0.8042,92630.8$0.00452026-09-271
Kimi K3.1 Prolow42.8%61.1%24.4%0.8294946.3$0.00582026-11-101
Kimi K3.1 Promedium42.8%64.4%21.1%0.8154475.9$0.00532026-11-101
GPT-6 Lunamedium40.5%58.9%22.2%0.8141,27215.9$0.00072026-09-241
MiniMax M3.1 Flashhigh39.8%56.7%22.8%0.8269,91240.0$0.00602026-10-271
Inklingmedium39.5%55.6%23.3%0.8222,42432.6$0.01012026-11-101
MiniMax M3.1 Flashmedium39.1%55.0%23.3%0.8147,45633.9$0.00452026-10-271
MiMo v2.5 Proon38.6%58.9%18.3%0.7656,356187.9$0.01132026-10-221
Haiku 5.5high38.4%52.8%23.9%0.8302,18812.7$0.00122026-10-221
GLM-5.3high38.3%57.2%19.4%0.7863,08131.6$0.01412026-09-241
Qwen3.8-27Bhigh37.2%53.3%21.1%0.7973,82162.2$0.00852026-09-271
Qwen3.8-27Blow36.4%52.2%20.6%0.7923,35042.8$0.00752026-09-271
Haiku 4.5fixed36.2%55.6%16.7%0.78110,594100.0$0.05372026-09-241
Qwen3.8-27Bmedium35.0%51.7%18.3%0.7923,75247.1$0.00832026-09-271
GLM-5.3 Flashhigh34.8%47.8%21.7%0.7772,23732.4$0.00122026-09-241
Haiku 5.5medium34.7%49.4%20.0%0.8021,3689.2$0.00082026-10-221
Haiku 5.5low34.5%52.8%16.1%0.7729777.4$0.00072026-10-221
GPT-6 Lunalow31.7%50.6%12.8%0.7645357.8$0.00032026-09-241
MiniMax M3.1 Flashlow31.6%46.1%17.2%0.7795,65227.1$0.00342026-10-271
Inklinglow26.7%42.8%10.6%0.7033756.3$0.00182026-11-101
Nemotron 3.5 Super VLoff26.4%38.3%14.4%0.6355,38124.1$0.00272026-10-301
GLM-5.3medium26.2%41.7%10.6%0.72192715.8$0.00362026-09-241
GLM-5.3 Flashlow25.0%43.9%6.1%0.6613716.5$0.00022026-09-241
GLM-5.3 Flashmedium25.0%42.8%7.2%0.6774518.0$0.00032026-09-241
GLM-5.3low24.4%41.1%7.8%0.71469611.8$0.00302026-09-241
Qwen3.6-35B-A3Bon23.3%34.4%12.2%0.7195,04768.2$0.00512026-09-271
Qwen3.8-27Boff12.2%21.1%3.3%0.567140.9$0.00012026-09-271
Qwen3.6-35B-A3Boff7.2%13.3%1.1%0.482140.9$0.00012026-09-271

Download the data (JSON)

Very verbose LLMs

Old trick played again: very very verbose LLMs

Muse, Kimi and DeepSeek models do relatively well in the ranking, but they talk a lot to get there. Muse Wonder at high effort is fourth with 74.8%, on 7,284 completion tokens per task; the three runs around it, GPT-6 Astra at high effort (77.5%), Fable 5.2 at high effort (75.8%) and GPT-6 Astra at medium effort (74.7%), need 504, 798 and 258.

Model · effort Score Tokens per task Seconds per task Cost per test
Muse Wonder · high 74.8% 7,284 66.1 $0.1468
Muse Ember · high 67.2% 7,614 20.7 $0.0038
DeepSeek V4.1 Flash · high 66.4% 7,276 74.2 $0.0088
Kimi K3 · high 64.7% 4,651 88.9 $0.0685
DeepSeek V4 Pro · high 55.0% 6,001 89.7 $0.0224
GPT-6 Astra · high 77.5% 504 15.9 $0.0289
Fable 5.2 · high 75.8% 798 9.9 $0.0372

That's fine for a public benchmark, which only counts the score. Nobody seems to care about the practical side: a low price per token is misleading if the model needs ten times the tokens to answer, and every one of those tokens is also time you wait.

Compare working with Gemini 4 Argon at low effort and Muse Wonder at high effort:

Gemini 4 Argon · low Muse Wonder · high Muse Wonder is
Score 71.1% 74.8% 3.7 points better
Tokens per task 18 7,284 416 times as many
Seconds per task 3.9 66.1 17 times as slow
Cost per test $0.0010 $0.1468 147 times as expensive

Over a thousand tasks, Argon answers in about an hour for about a dollar; Muse Wonder takes over 18 hours and about $147, for 3.7 points more.