← Some Bench

Some Bench

in progress

A benchmark of how well LLMs can reason if they can't cheat.

Family
Effort
0%20%40%60%80%100%1020501002005001K2K5K10KCompletion tokens per task (log scale)Score (mean of standard and harder variant exact)Gemini 4 Argon · highGemini 4 Carbon · mediumGPT-6 Astra · highGPT-6 Astra · mediumGPT-6.2 Astra · mediumGemini 4 Argon · lowQwen3.8-27B · off

Labelled are the runs that no other run beats on both axes: a higher score for less (Labels shows all of them). Hover any point for its numbers, click it to find its row in the table.

Runs
Gemini 4 Argonhigh78.9%96.7%61.1%0.9273,29144.5$0.03372026-10-301
Gemini 4 Carbonmedium78.3%93.9%62.8%0.9031,87438.4$0.01952026-11-101
Gemini 4 Neonhigh78.3%96.1%60.6%0.9367,39462.0$0.11212026-11-191
Gemini 4 Neonmedium78.3%96.7%60.0%0.9312,45523.6$0.03812026-11-191
GPT-6 Astrahigh77.5%94.2%60.9%0.93250415.9$0.02892026-09-30
Gemini 4 Hydrogenhigh76.1%92.8%59.4%0.9214,40727.6$0.03372026-11-141
GPT-6.2 Astrahigh76.1%92.2%60.0%0.9363359.0$0.01642026-11-181
Gemini 4 Neonlow75.8%93.9%57.8%0.9241,00911.1$0.01642026-11-191
Gemini 4 Carbonhigh75.5%93.9%57.2%0.80810,067175.9$0.10142026-11-101
Muse Wonderhigh74.8%91.7%57.8%0.9297,28466.1$0.14682026-10-301
GPT-6 Astramedium74.7%92.0%57.5%0.9312589.9$0.01662026-09-30
Fable 5.2high74.4%90.5%58.4%0.92677610.6$0.03692026-11-15
Muse Wondermedium73.9%91.1%56.7%0.9265,35247.7$0.10822026-10-301
GPT-6.2 Astramedium73.6%92.8%54.4%0.920973.8$0.00682026-11-181
Fable 5.2low73.1%91.4%54.9%0.9184497.3$0.03482026-11-15
Fable 5.2medium72.9%91.1%54.7%0.9174837.7$0.02612026-11-15
Gemini 4 Carbonlow72.5%92.8%52.2%0.9152548.4$0.00342026-11-101
Gemini 4 Hydrogenmedium72.5%90.6%54.4%0.9182,23614.4$0.01742026-11-141
GPT-6.1 Solmedium72.2%91.1%53.3%0.9192317.3$0.00302026-09-301
Gemini 4 Argonlow71.1%90.0%52.2%0.914183.9$0.00102026-10-301
GPT-6.2 Astralow70.9%90.6%51.1%0.907422.5$0.00462026-11-181
Gemini 4 Argonmedium70.5%92.2%48.9%0.9071355.7$0.00222026-10-301
GPT-6.1 Solhigh70.3%87.8%52.8%0.92445812.3$0.00532026-09-301
GPT-6 Astralow70.0%88.9%51.1%0.9191516.9$0.01122026-09-30
Opus 5.5high69.9%87.1%52.6%0.9195988.6$0.01652026-11-15
Muse Wonderlow68.3%88.3%48.3%0.9143,28929.5$0.06692026-10-301
GPT-6.2 Solhigh68.0%88.3%47.8%0.9071372.8$0.00212026-11-181
Opus 5.5medium67.8%85.0%50.7%0.9144867.5$0.01402026-11-15
Fable 5.1high67.3%85.2%49.5%0.9095159.7$0.03942026-10-22
Muse Emberhigh67.3%86.7%47.8%0.9067,61420.7$0.00382026-11-101
GPT-6.1 Sollow67.3%86.7%47.8%0.9131566.0$0.00232026-09-301
DeepSeek V4.1 Flashhigh66.4%80.6%52.2%0.9197,27674.2$0.00882026-09-241
GPT-6.1 Terrahigh66.4%87.8%45.0%0.9023034.9$0.00192026-10-281
Gemini 4 Hydrogenlow66.4%85.0%47.8%0.8978755.7$0.00722026-11-141
Kimi K3high64.7%82.2%47.2%0.9114,65188.9$0.06852026-09-241
Opus 5.5low64.4%83.3%45.4%0.9063185.8$0.01462026-11-15
Fable 5.1medium64.3%82.9%45.6%0.8973908.1$0.03402026-10-22
Sonnet 5.5high63.9%82.3%45.6%0.90491111.1$0.01172026-10-22
GPT-6.2 Solmedium62.8%79.4%46.1%0.893531.6$0.00132026-11-181
Fable 5.1low62.7%82.8%42.5%0.8893057.1$0.03542026-10-22
DeepSeek V4.1 Flashlow62.3%80.6%43.9%0.8965,34347.2$0.00652026-09-241
DeepSeek V4.1 Flashmedium62.2%82.2%42.2%0.9045,51449.0$0.00672026-09-241
Muse Embermedium62.0%77.2%46.7%0.9055,09010.7$0.00262026-11-101
GPT-6 Solhigh61.7%78.3%45.0%0.89367913.7$0.00752026-09-241
Muse Emberlow60.8%78.3%43.3%0.8963,3527.5$0.00172026-11-101
Sonnet 5.5medium60.7%77.5%43.9%0.8996628.8$0.00952026-10-22
Ripple 65B Previewmedium60.6%74.4%46.7%0.90110,813102.1$0.00652026-11-101
Sonnet 5medium60.3%76.7%43.9%0.8845748.5$0.00972026-09-241
Kimi K3.1 Prohigh60.3%80.0%40.6%0.8931,65742.5$0.01752026-11-101
Ripple 65B Previewlow60.0%76.7%43.3%0.88611,403108.3$0.00692026-11-101
Sonnet 5high59.8%78.9%40.6%0.8946449.2$0.01042026-09-241
GPT-6.2 Sollow59.4%79.4%39.4%0.878191.1$0.00092026-11-181
GPT-6 Solmedium58.9%78.3%39.4%0.8834068.4$0.00482026-09-241
Sonnet 5.5low58.5%77.5%39.5%0.8865918.3$0.00982026-10-22
Kimi K3low58.1%76.7%39.4%0.8571,76934.4$0.02642026-09-241
Ripple 65B Previewhigh57.8%74.4%41.1%0.90010,53096.2$0.00642026-11-101
Sonnet 5low57.5%79.4%35.6%0.8644997.7$0.00892026-09-241
Muse-Glimmer-30Bhigh57.3%75.6%38.9%0.8729,615131.8$0.01452026-09-271
GPT-6.1 Terramedium56.7%77.8%35.6%0.8741878.5$0.00132026-10-281
DeepSeek V4 Prohigh55.0%76.7%33.3%0.8666,00189.7$0.02242026-09-241
DeepSeek V4 Prolow54.7%73.3%36.1%0.8535,72493.7$0.02132026-09-241
DeepSeek V4 Promedium53.8%70.9%36.7%0.8675,92886.1$0.02212026-09-241
Inklinghigh52.8%72.2%33.3%0.8656,90369.5$0.02832026-11-101
Kimi K3medium52.5%72.2%32.8%0.8581,74333.5$0.02622026-09-241
GPT-6 Sollow52.3%72.8%31.7%0.8582096.5$0.00282026-09-241
GPT-6.1 Terralow50.8%73.3%28.3%0.853972.5$0.00092026-10-281
Nemotron 3 Ultramedium50.0%66.7%33.3%0.8518,49777.5$0.02062026-10-301
Muse-Glimmer-30Bmedium49.7%66.1%33.3%0.8455,81463.1$0.00882026-09-271
Nemotron 3 Ultrahigh47.8%65.6%30.0%0.8378,71177.8$0.02112026-10-301
Nemotron 3 Ultralow47.8%68.3%27.2%0.8328,90279.5$0.02152026-10-301
GPT-6 Lunahigh46.4%61.1%31.7%0.8652,22530.8$0.00112026-09-241
MiniMax M3on46.1%65.0%27.2%0.80111,824111.1$0.01432026-09-241
Muse-Glimmer-30Blow42.8%57.8%27.8%0.8042,92630.8$0.00452026-09-271
Kimi K3.1 Prolow42.8%61.1%24.4%0.8294946.3$0.00582026-11-101
Kimi K3.1 Promedium42.8%64.4%21.1%0.8154475.9$0.00532026-11-101
GPT-6 Lunamedium40.5%58.9%22.2%0.8141,27215.9$0.00072026-09-241
MiniMax M3.1 Flashhigh39.8%56.7%22.8%0.8269,91240.0$0.00602026-10-271
Inklingmedium39.5%55.6%23.3%0.8222,42432.6$0.01012026-11-101
MiniMax M3.1 Flashmedium39.1%55.0%23.3%0.8147,45633.9$0.00452026-10-271
MiMo v2.5 Proon38.6%58.9%18.3%0.7656,356187.9$0.01132026-10-221
Haiku 5.5high38.4%52.8%23.9%0.8302,18812.7$0.00122026-10-221
GLM-5.3high38.3%57.2%19.4%0.7863,08131.6$0.01412026-09-241
Qwen3.8-27Bhigh37.2%53.3%21.1%0.7973,82162.2$0.00852026-09-271
Qwen3.8-27Blow36.4%52.2%20.6%0.7923,35042.8$0.00752026-09-271
Haiku 4.5fixed36.2%55.6%16.7%0.78110,594100.0$0.05372026-09-241
Qwen3.8-27Bmedium35.0%51.7%18.3%0.7923,75247.1$0.00832026-09-271
GLM-5.3 Flashhigh34.8%47.8%21.7%0.7772,23732.4$0.00122026-09-241
Haiku 5.5medium34.7%49.4%20.0%0.8021,3689.2$0.00082026-10-221
Haiku 5.5low34.5%52.8%16.1%0.7729777.4$0.00072026-10-221
GPT-6 Lunalow31.7%50.6%12.8%0.7645357.8$0.00032026-09-241
MiniMax M3.1 Flashlow31.6%46.1%17.2%0.7795,65227.1$0.00342026-10-271
Inklinglow26.7%42.8%10.6%0.7033756.3$0.00182026-11-101
Nemotron 3.5 Super VLoff26.4%38.3%14.4%0.6355,38124.1$0.00272026-10-301
GLM-5.3medium26.2%41.7%10.6%0.72192715.8$0.00362026-09-241
GLM-5.3 Flashlow25.0%43.9%6.1%0.6613716.5$0.00022026-09-241
GLM-5.3 Flashmedium25.0%42.8%7.2%0.6774518.0$0.00032026-09-241
GLM-5.3low24.4%41.1%7.8%0.71469611.8$0.00302026-09-241
Qwen3.6-35B-A3Bon23.3%34.4%12.2%0.7195,04768.2$0.00512026-09-271
Qwen3.8-27Boff12.2%21.1%3.3%0.567140.9$0.00012026-09-271
Qwen3.6-35B-A3Boff7.2%13.3%1.1%0.482140.9$0.00012026-09-271

Download the data (JSON)