← Some Bench

Some Bench

in progress

A benchmark of how well LLMs can reason if they can't cheat.

Family
Effort
0%20%40%60%80%100%1020501002005001K2K5K10KCompletion tokens per task (log scale)Score (mean of standard and harder variant exact)Gemini 4 Argon · highGemini 4 Carbon · mediumGPT-6 Astra · highGPT-6 Astra · mediumGPT-6.2 Astra · mediumGemini 4 Argon · lowQwen3.8-27B · off

Labelled are the runs that no other run beats on both axes: a higher score for less (Labels shows all of them). Hover any point for its numbers, click it to find its row in the table.

Runs
Gemini 4 Argonhigh78.9%96.7%61.1%0.9273,29144.5$0.03372026-10-301
Gemini 4 Carbonmedium78.3%93.9%62.8%0.9031,87438.4$0.01952026-11-101
Gemini 4 Neonhigh78.3%96.1%60.6%0.9367,39462.0$0.11212026-11-191
Gemini 4 Neonmedium78.3%96.7%60.0%0.9312,45523.6$0.03812026-11-191
GPT-6 Astrahigh77.5%94.2%60.9%0.93250415.9$0.02892026-09-30
Gemini 4 Hydrogenhigh76.1%92.8%59.4%0.9214,40727.6$0.03372026-11-141
GPT-6.2 Astrahigh76.1%92.2%60.0%0.9363359.0$0.01642026-11-181
Gemini 4 Neonlow75.8%93.9%57.8%0.9241,00911.1$0.01642026-11-191
Gemini 4 Carbonhigh75.5%93.9%57.2%0.80810,067175.9$0.10142026-11-101
Muse Wonderhigh74.8%91.7%57.8%0.9297,28466.1$0.14682026-10-301
GPT-6 Astramedium74.7%92.0%57.5%0.9312589.9$0.01662026-09-30
Fable 5.2high74.4%90.5%58.4%0.92677610.6$0.03692026-11-15
Muse Wondermedium73.9%91.1%56.7%0.9265,35247.7$0.10822026-10-301
GPT-6.2 Astramedium73.6%92.8%54.4%0.920973.8$0.00682026-11-181
Fable 5.2low73.1%91.4%54.9%0.9184497.3$0.03482026-11-15
Fable 5.2medium72.9%91.1%54.7%0.9174837.7$0.02612026-11-15
Gemini 4 Carbonlow72.5%92.8%52.2%0.9152548.4$0.00342026-11-101
Gemini 4 Hydrogenmedium72.5%90.6%54.4%0.9182,23614.4$0.01742026-11-141
GPT-6.1 Solmedium72.2%91.1%53.3%0.9192317.3$0.00302026-09-301
Gemini 4 Argonlow71.1%90.0%52.2%0.914183.9$0.00102026-10-301
GPT-6.2 Astralow70.9%90.6%51.1%0.907422.5$0.00462026-11-181
Gemini 4 Argonmedium70.5%92.2%48.9%0.9071355.7$0.00222026-10-301
GPT-6.1 Solhigh70.3%87.8%52.8%0.92445812.3$0.00532026-09-301
GPT-6 Astralow70.0%88.9%51.1%0.9191516.9$0.01122026-09-30
Opus 5.5high69.9%87.1%52.6%0.9195988.6$0.01652026-11-15
Muse Wonderlow68.3%88.3%48.3%0.9143,28929.5$0.06692026-10-301
GPT-6.2 Solhigh68.0%88.3%47.8%0.9071372.8$0.00212026-11-181
Opus 5.5medium67.8%85.0%50.7%0.9144867.5$0.01402026-11-15
Fable 5.1high67.3%85.2%49.5%0.9095159.7$0.03942026-10-22
Muse Emberhigh67.3%86.7%47.8%0.9067,61420.7$0.00382026-11-101
GPT-6.1 Sollow67.3%86.7%47.8%0.9131566.0$0.00232026-09-301
DeepSeek V4.1 Flashhigh66.4%80.6%52.2%0.9197,27674.2$0.00882026-09-241
GPT-6.1 Terrahigh66.4%87.8%45.0%0.9023034.9$0.00192026-10-281
Gemini 4 Hydrogenlow66.4%85.0%47.8%0.8978755.7$0.00722026-11-141
Kimi K3high64.7%82.2%47.2%0.9114,65188.9$0.06852026-09-241
Opus 5.5low64.4%83.3%45.4%0.9063185.8$0.01462026-11-15
Fable 5.1medium64.3%82.9%45.6%0.8973908.1$0.03402026-10-22
Sonnet 5.5high63.9%82.3%45.6%0.90491111.1$0.01172026-10-22
GPT-6.2 Solmedium62.8%79.4%46.1%0.893531.6$0.00132026-11-181
Fable 5.1low62.7%82.8%42.5%0.8893057.1$0.03542026-10-22
DeepSeek V4.1 Flashlow62.3%80.6%43.9%0.8965,34347.2$0.00652026-09-241
DeepSeek V4.1 Flashmedium62.2%82.2%42.2%0.9045,51449.0$0.00672026-09-241
Muse Embermedium62.0%77.2%46.7%0.9055,09010.7$0.00262026-11-101
GPT-6 Solhigh61.7%78.3%45.0%0.89367913.7$0.00752026-09-241
Muse Emberlow60.8%78.3%43.3%0.8963,3527.5$0.00172026-11-101
Sonnet 5.5medium60.7%77.5%43.9%0.8996628.8$0.00952026-10-22
Ripple 65B Previewmedium60.6%74.4%46.7%0.90110,813102.1$0.00652026-11-101
Sonnet 5medium60.3%76.7%43.9%0.8845748.5$0.00972026-09-241
Kimi K3.1 Prohigh60.3%80.0%40.6%0.8931,65742.5$0.01752026-11-101
Ripple 65B Previewlow60.0%76.7%43.3%0.88611,403108.3$0.00692026-11-101
Sonnet 5high59.8%78.9%40.6%0.8946449.2$0.01042026-09-241
GPT-6.2 Sollow59.4%79.4%39.4%0.878191.1$0.00092026-11-181
GPT-6 Solmedium58.9%78.3%39.4%0.8834068.4$0.00482026-09-241
Sonnet 5.5low58.5%77.5%39.5%0.8865918.3$0.00982026-10-22
Kimi K3low58.1%76.7%39.4%0.8571,76934.4$0.02642026-09-241
Ripple 65B Previewhigh57.8%74.4%41.1%0.90010,53096.2$0.00642026-11-101
Sonnet 5low57.5%79.4%35.6%0.8644997.7$0.00892026-09-241
Muse-Glimmer-30Bhigh57.3%75.6%38.9%0.8729,615131.8$0.01452026-09-271
GPT-6.1 Terramedium56.7%77.8%35.6%0.8741878.5$0.00132026-10-281
DeepSeek V4 Prohigh55.0%76.7%33.3%0.8666,00189.7$0.02242026-09-241
DeepSeek V4 Prolow54.7%73.3%36.1%0.8535,72493.7$0.02132026-09-241
DeepSeek V4 Promedium53.8%70.9%36.7%0.8675,92886.1$0.02212026-09-241
Inklinghigh52.8%72.2%33.3%0.8656,90369.5$0.02832026-11-101
Kimi K3medium52.5%72.2%32.8%0.8581,74333.5$0.02622026-09-241
GPT-6 Sollow52.3%72.8%31.7%0.8582096.5$0.00282026-09-241
GPT-6.1 Terralow50.8%73.3%28.3%0.853972.5$0.00092026-10-281
Nemotron 3 Ultramedium50.0%66.7%33.3%0.8518,49777.5$0.02062026-10-301
Muse-Glimmer-30Bmedium49.7%66.1%33.3%0.8455,81463.1$0.00882026-09-271
Nemotron 3 Ultrahigh47.8%65.6%30.0%0.8378,71177.8$0.02112026-10-301
Nemotron 3 Ultralow47.8%68.3%27.2%0.8328,90279.5$0.02152026-10-301
GPT-6 Lunahigh46.4%61.1%31.7%0.8652,22530.8$0.00112026-09-241
MiniMax M3on46.1%65.0%27.2%0.80111,824111.1$0.01432026-09-241
Muse-Glimmer-30Blow42.8%57.8%27.8%0.8042,92630.8$0.00452026-09-271
Kimi K3.1 Prolow42.8%61.1%24.4%0.8294946.3$0.00582026-11-101
Kimi K3.1 Promedium42.8%64.4%21.1%0.8154475.9$0.00532026-11-101
GPT-6 Lunamedium40.5%58.9%22.2%0.8141,27215.9$0.00072026-09-241
MiniMax M3.1 Flashhigh39.8%56.7%22.8%0.8269,91240.0$0.00602026-10-271
Inklingmedium39.5%55.6%23.3%0.8222,42432.6$0.01012026-11-101
MiniMax M3.1 Flashmedium39.1%55.0%23.3%0.8147,45633.9$0.00452026-10-271
MiMo v2.5 Proon38.6%58.9%18.3%0.7656,356187.9$0.01132026-10-221
Haiku 5.5high38.4%52.8%23.9%0.8302,18812.7$0.00122026-10-221
GLM-5.3high38.3%57.2%19.4%0.7863,08131.6$0.01412026-09-241
Qwen3.8-27Bhigh37.2%53.3%21.1%0.7973,82162.2$0.00852026-09-271
Qwen3.8-27Blow36.4%52.2%20.6%0.7923,35042.8$0.00752026-09-271
Haiku 4.5fixed36.2%55.6%16.7%0.78110,594100.0$0.05372026-09-241
Qwen3.8-27Bmedium35.0%51.7%18.3%0.7923,75247.1$0.00832026-09-271
GLM-5.3 Flashhigh34.8%47.8%21.7%0.7772,23732.4$0.00122026-09-241
Haiku 5.5medium34.7%49.4%20.0%0.8021,3689.2$0.00082026-10-221
Haiku 5.5low34.5%52.8%16.1%0.7729777.4$0.00072026-10-221
GPT-6 Lunalow31.7%50.6%12.8%0.7645357.8$0.00032026-09-241
MiniMax M3.1 Flashlow31.6%46.1%17.2%0.7795,65227.1$0.00342026-10-271
Inklinglow26.7%42.8%10.6%0.7033756.3$0.00182026-11-101
Nemotron 3.5 Super VLoff26.4%38.3%14.4%0.6355,38124.1$0.00272026-10-301
GLM-5.3medium26.2%41.7%10.6%0.72192715.8$0.00362026-09-241
GLM-5.3 Flashlow25.0%43.9%6.1%0.6613716.5$0.00022026-09-241
GLM-5.3 Flashmedium25.0%42.8%7.2%0.6774518.0$0.00032026-09-241
GLM-5.3low24.4%41.1%7.8%0.71469611.8$0.00302026-09-241
Qwen3.6-35B-A3Bon23.3%34.4%12.2%0.7195,04768.2$0.00512026-09-271
Qwen3.8-27Boff12.2%21.1%3.3%0.567140.9$0.00012026-09-271
Qwen3.6-35B-A3Boff7.2%13.3%1.1%0.482140.9$0.00012026-09-271

Download the data (JSON)

Opus 5.5 and Fable 5.2

Is Opus 5.5 nerfed after Fable 5.2's release?

A theory goes around online that Anthropic's older models get worse whenever a new one is released. Some Bench happened to run Opus 5.5 before and after one release, Fable 5.2, with the same tests and the same settings, so here is what that showed.

At high effort, Opus 5.5 got shorter and slightly worse. At low and medium effort, nothing moved beyond the usual run-to-run variation. One release and one run after it can't confirm the theory, but the high-effort change is the largest token drop in our data.

The runs

Fable 5.2 came out on 10 November, and the third run of Opus 5.5 is from that day.

Opus 5.5 24 Sep 22 Oct 10 Nov, Fable 5.2 release day
High: score 71.4% 71.4% 68.0%
High: harder variant, exact 56.1% 55.0% 50.0%
High: completion tokens per task 614 656 552 (−16%)
High: seconds per task 9.2 9.8 7.5
High: cost per test $0.0200 $0.0179 $0.0139
Medium: score 68.9% 66.6% 68.3%
Medium: completion tokens per task 506 499 472
Low: score 65.0% 65.3% 65.0%
Low: completion tokens per task 318 324 319

At high effort the model wrote 16% fewer completion tokens (which include its reasoning), answered 23% faster, cost 22% less per test and lost 3.4 points, almost all of them on the harder variant. That looks like less thinking per answer, not a different model: low effort, which thinks least, didn't change at all.

Is that more than noise?

Change in score and in tokens between two runs of the same model and effort; the Opus 5.5
high-effort pair across the Fable 5.2 release sits furthest down, at −3.4 points and −16%
tokens

Running the same model twice never gives exactly the same result. Every model we ran more than once is on the chart, so the orange points can be compared with the normal spread in grey:

  • Score: a 3.4-point drop is the largest in our data, but only just. Sonnet 5.5 at medium effort dropped 3.0 points between 29 September and 22 October, with no release in between.
  • Tokens: a 16% drop stands out more clearly. Every other pair of runs stayed within 7%.

What this doesn't show

  • One event. To test "older models get worse at each release", the same check has to hold across several releases, not one.
  • No comparison run after the release. Only Opus 5.5 was run again on 10 November. Running Sonnet 5.5, Fable 5.1 and a model from another company on the same day would show whether something changed for everyone, such as load on the provider's side.
  • Why it changed. From outside we can only see that the model writes less and scores lower at high effort. Whether that's a deliberate change, a different thinking budget or a side effect of serving a new model, the numbers can't tell.

We'll run Opus 5.5 at high effort again to see whether the drop holds, and repeat this check at the next Anthropic release.