<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:base="https://codecat.io/">
  <title>codecat updates</title>
  <id>https://codecat.io/feed.xml</id>
  <link rel="self" href="https://codecat.io/feed.xml"/>
  <link rel="alternate" type="text/html" href="https://codecat.io/news/"/>
  <updated>2026-11-19T00:00:00Z</updated>
  <author><name>codecat</name></author>
  <entry>
    <title>Some Bench: New results: Gemini 4 Neon</title>
    <id>https://codecat.io/projects/some-bench/updates/new-results-gemini-4-neon/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/updates/new-results-gemini-4-neon/"/>
    <updated>2026-11-19T00:00:00Z</updated>
    <summary>Added Gemini 4 Neon. Best of them: Gemini 4 Neon high, 78.3%.</summary>
    <content type="html">&lt;p&gt;Added Gemini 4 Neon. Best of them: Gemini 4 Neon high, 78.3%.&lt;/p&gt;
&lt;h2 id=&quot;added&quot;&gt;Added&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Neon&quot;&gt;Gemini 4 Neon&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;78.3%&lt;/td&gt;
&lt;td&gt;7,394&lt;/td&gt;
&lt;td&gt;62.0&lt;/td&gt;
&lt;td&gt;$0.1121&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Neon&quot;&gt;Gemini 4 Neon&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;78.3%&lt;/td&gt;
&lt;td&gt;2,455&lt;/td&gt;
&lt;td&gt;23.6&lt;/td&gt;
&lt;td&gt;$0.0381&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Neon&quot;&gt;Gemini 4 Neon&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;75.8%&lt;/td&gt;
&lt;td&gt;1,008&lt;/td&gt;
&lt;td&gt;11.1&lt;/td&gt;
&lt;td&gt;$0.0164&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: New results: GPT-6.2 Astra and GPT-6.2 Sol</title>
    <id>https://codecat.io/projects/some-bench/updates/new-results-gpt-6-2-astra-gpt-6-2-sol/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/updates/new-results-gpt-6-2-astra-gpt-6-2-sol/"/>
    <updated>2026-11-18T00:00:00Z</updated>
    <summary>Added GPT-6.2 Astra and GPT-6.2 Sol. Best of them: GPT-6.2 Astra high, 76.1%.</summary>
    <content type="html">&lt;p&gt;Added GPT-6.2 Astra and GPT-6.2 Sol. Best of them: GPT-6.2 Astra high, 76.1%.&lt;/p&gt;
&lt;h2 id=&quot;added&quot;&gt;Added&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=GPT-6.2%20Astra&quot;&gt;GPT-6.2 Astra&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;76.1%&lt;/td&gt;
&lt;td&gt;335&lt;/td&gt;
&lt;td&gt;9.0&lt;/td&gt;
&lt;td&gt;$0.0164&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=GPT-6.2%20Astra&quot;&gt;GPT-6.2 Astra&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;73.6%&lt;/td&gt;
&lt;td&gt;97&lt;/td&gt;
&lt;td&gt;3.8&lt;/td&gt;
&lt;td&gt;$0.0068&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=GPT-6.2%20Astra&quot;&gt;GPT-6.2 Astra&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;70.9%&lt;/td&gt;
&lt;td&gt;42&lt;/td&gt;
&lt;td&gt;2.5&lt;/td&gt;
&lt;td&gt;$0.0046&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=GPT-6.2%20Sol&quot;&gt;GPT-6.2 Sol&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;68.0%&lt;/td&gt;
&lt;td&gt;137&lt;/td&gt;
&lt;td&gt;2.8&lt;/td&gt;
&lt;td&gt;$0.0021&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=GPT-6.2%20Sol&quot;&gt;GPT-6.2 Sol&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;62.8%&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;td&gt;1.6&lt;/td&gt;
&lt;td&gt;$0.0013&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=GPT-6.2%20Sol&quot;&gt;GPT-6.2 Sol&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;59.4%&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;1.1&lt;/td&gt;
&lt;td&gt;$0.0009&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: New results: Gemini 4 Carbon and Gemini 4 Hydrogen</title>
    <id>https://codecat.io/projects/some-bench/updates/new-results-gemini-4-carbon-gemini-4-hydrogen/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/updates/new-results-gemini-4-carbon-gemini-4-hydrogen/"/>
    <updated>2026-11-17T00:00:00Z</updated>
    <summary>Added Gemini 4 Carbon and Gemini 4 Hydrogen. Best of them: Gemini 4 Carbon medium, 78.3%. Fable 5.2 and Opus 5.5 were run again.</summary>
    <content type="html">&lt;p&gt;Added Gemini 4 Carbon and Gemini 4 Hydrogen. Best of them: Gemini 4 Carbon medium, 78.3%. Fable 5.2 and Opus 5.5 were run again.&lt;/p&gt;
&lt;h2 id=&quot;added&quot;&gt;Added&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Carbon&quot;&gt;Gemini 4 Carbon&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;78.3%&lt;/td&gt;
&lt;td&gt;1,874&lt;/td&gt;
&lt;td&gt;38.4&lt;/td&gt;
&lt;td&gt;$0.0195&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Hydrogen&quot;&gt;Gemini 4 Hydrogen&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;76.1%&lt;/td&gt;
&lt;td&gt;4,406&lt;/td&gt;
&lt;td&gt;27.6&lt;/td&gt;
&lt;td&gt;$0.0337&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Carbon&quot;&gt;Gemini 4 Carbon&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;75.5%&lt;/td&gt;
&lt;td&gt;10,067&lt;/td&gt;
&lt;td&gt;175.9&lt;/td&gt;
&lt;td&gt;$0.1014&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Carbon&quot;&gt;Gemini 4 Carbon&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;72.5%&lt;/td&gt;
&lt;td&gt;254&lt;/td&gt;
&lt;td&gt;8.4&lt;/td&gt;
&lt;td&gt;$0.0034&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Hydrogen&quot;&gt;Gemini 4 Hydrogen&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;72.5%&lt;/td&gt;
&lt;td&gt;2,236&lt;/td&gt;
&lt;td&gt;14.4&lt;/td&gt;
&lt;td&gt;$0.0174&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Gemini%204%20Hydrogen&quot;&gt;Gemini 4 Hydrogen&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;66.4%&lt;/td&gt;
&lt;td&gt;875&lt;/td&gt;
&lt;td&gt;5.7&lt;/td&gt;
&lt;td&gt;$0.0072&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;h2 id=&quot;run-again&quot;&gt;Run again&lt;/h2&gt;
&lt;p&gt;The page shows the average of all runs; these are the newest run against the average before it.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Fable%205.2&quot;&gt;Fable 5.2&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;75.8% → 73.0%&lt;/td&gt;
&lt;td&gt;798 → 753&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Fable%205.2&quot;&gt;Fable 5.2&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;72.5% → 73.8%&lt;/td&gt;
&lt;td&gt;474 → 424&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Fable%205.2&quot;&gt;Fable 5.2&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;73.3% → 72.5%&lt;/td&gt;
&lt;td&gt;509 → 457&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Opus%205.5&quot;&gt;Opus 5.5&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;70.3% → 68.6%&lt;/td&gt;
&lt;td&gt;608 → 568&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Opus%205.5&quot;&gt;Opus 5.5&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;65.1% → 62.2%&lt;/td&gt;
&lt;td&gt;321 → 308&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Opus%205.5&quot;&gt;Opus 5.5&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;68.0% → 67.5%&lt;/td&gt;
&lt;td&gt;492 → 465&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: New results: Ripple 65B Preview and Inkling</title>
    <id>https://codecat.io/projects/some-bench/updates/new-results-ripple-65b-preview-inkling/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/updates/new-results-ripple-65b-preview-inkling/"/>
    <updated>2026-11-10T00:00:00Z</updated>
    <summary>Added Ripple 65B Preview and Inkling. Best of them: Ripple 65B Preview medium, 60.6%.</summary>
    <content type="html">&lt;p&gt;Added Ripple 65B Preview and Inkling. Best of them: Ripple 65B Preview medium, 60.6%.&lt;/p&gt;
&lt;h2 id=&quot;added&quot;&gt;Added&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Ripple%2065B%20Preview&quot;&gt;Ripple 65B Preview&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;60.6%&lt;/td&gt;
&lt;td&gt;10,813&lt;/td&gt;
&lt;td&gt;102.1&lt;/td&gt;
&lt;td&gt;$0.0065&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Ripple%2065B%20Preview&quot;&gt;Ripple 65B Preview&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;60.0%&lt;/td&gt;
&lt;td&gt;11,402&lt;/td&gt;
&lt;td&gt;108.3&lt;/td&gt;
&lt;td&gt;$0.0069&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Ripple%2065B%20Preview&quot;&gt;Ripple 65B Preview&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;57.8%&lt;/td&gt;
&lt;td&gt;10,530&lt;/td&gt;
&lt;td&gt;96.2&lt;/td&gt;
&lt;td&gt;$0.0064&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Inkling&quot;&gt;Inkling&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;52.8%&lt;/td&gt;
&lt;td&gt;6,903&lt;/td&gt;
&lt;td&gt;69.5&lt;/td&gt;
&lt;td&gt;$0.0283&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Inkling&quot;&gt;Inkling&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;39.5%&lt;/td&gt;
&lt;td&gt;2,424&lt;/td&gt;
&lt;td&gt;32.6&lt;/td&gt;
&lt;td&gt;$0.0101&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Inkling&quot;&gt;Inkling&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;26.7%&lt;/td&gt;
&lt;td&gt;375&lt;/td&gt;
&lt;td&gt;6.3&lt;/td&gt;
&lt;td&gt;$0.0018&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: Very verbose LLMs</title>
    <id>https://codecat.io/projects/some-bench/verbose/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/verbose/"/>
    <updated>2026-11-10T00:00:00Z</updated>
    <summary>Old trick played again: very very verbose LLMs</summary>
    <content type="html">&lt;p&gt;Muse, Kimi and DeepSeek models do relatively well in the ranking, but they talk a lot to get
there. Muse Wonder at high effort is fourth with 74.8%, on 7,284 completion tokens per task;
the three runs around it, GPT-6 Astra at high effort (77.5%), Fable 5.2 at high effort (75.8%)
and GPT-6 Astra at medium effort (74.7%), need 504, 798 and 258.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;Muse Wonder · high&lt;/td&gt;
&lt;td&gt;74.8%&lt;/td&gt;
&lt;td&gt;7,284&lt;/td&gt;
&lt;td&gt;66.1&lt;/td&gt;
&lt;td&gt;$0.1468&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Muse Ember · high&lt;/td&gt;
&lt;td&gt;67.2%&lt;/td&gt;
&lt;td&gt;7,614&lt;/td&gt;
&lt;td&gt;20.7&lt;/td&gt;
&lt;td&gt;$0.0038&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek V4.1 Flash · high&lt;/td&gt;
&lt;td&gt;66.4%&lt;/td&gt;
&lt;td&gt;7,276&lt;/td&gt;
&lt;td&gt;74.2&lt;/td&gt;
&lt;td&gt;$0.0088&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kimi K3 · high&lt;/td&gt;
&lt;td&gt;64.7%&lt;/td&gt;
&lt;td&gt;4,651&lt;/td&gt;
&lt;td&gt;88.9&lt;/td&gt;
&lt;td&gt;$0.0685&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek V4 Pro · high&lt;/td&gt;
&lt;td&gt;55.0%&lt;/td&gt;
&lt;td&gt;6,001&lt;/td&gt;
&lt;td&gt;89.7&lt;/td&gt;
&lt;td&gt;$0.0224&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6 Astra · high&lt;/td&gt;
&lt;td&gt;77.5%&lt;/td&gt;
&lt;td&gt;504&lt;/td&gt;
&lt;td&gt;15.9&lt;/td&gt;
&lt;td&gt;$0.0289&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fable 5.2 · high&lt;/td&gt;
&lt;td&gt;75.8%&lt;/td&gt;
&lt;td&gt;798&lt;/td&gt;
&lt;td&gt;9.9&lt;/td&gt;
&lt;td&gt;$0.0372&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;That&amp;#39;s fine for a public benchmark, which only counts the score. Nobody seems to care about the
practical side: a low price per token is misleading if the model needs ten times the tokens to
answer, and every one of those tokens is also time you wait.&lt;/p&gt;
&lt;p&gt;Compare working with Gemini 4 Argon at low effort and Muse Wonder at high effort:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Gemini 4 Argon · low&lt;/th&gt;
&lt;th&gt;Muse Wonder · high&lt;/th&gt;
&lt;th&gt;Muse Wonder is&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;Score&lt;/td&gt;
&lt;td&gt;71.1%&lt;/td&gt;
&lt;td&gt;74.8%&lt;/td&gt;
&lt;td&gt;3.7 points better&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens per task&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;7,284&lt;/td&gt;
&lt;td&gt;416 times as many&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Seconds per task&lt;/td&gt;
&lt;td&gt;3.9&lt;/td&gt;
&lt;td&gt;66.1&lt;/td&gt;
&lt;td&gt;17 times as slow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost per test&lt;/td&gt;
&lt;td&gt;$0.0010&lt;/td&gt;
&lt;td&gt;$0.1468&lt;/td&gt;
&lt;td&gt;147 times as expensive&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Over a thousand tasks, Argon answers in about an hour for about a dollar; Muse Wonder takes over
18 hours and about $147, for 3.7 points more.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: New results: Muse Ember and Kimi K3.1 Pro</title>
    <id>https://codecat.io/projects/some-bench/updates/new-results-muse-ember-kimi-k3-1-pro/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/updates/new-results-muse-ember-kimi-k3-1-pro/"/>
    <updated>2026-11-10T00:00:00Z</updated>
    <summary>Added Muse Ember and Kimi K3.1 Pro. Best of them: Muse Ember high, 67.2%.</summary>
    <content type="html">&lt;p&gt;Added Muse Ember and Kimi K3.1 Pro. Best of them: Muse Ember high, 67.2%.&lt;/p&gt;
&lt;h2 id=&quot;added&quot;&gt;Added&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Muse%20Ember&quot;&gt;Muse Ember&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;67.2%&lt;/td&gt;
&lt;td&gt;7,614&lt;/td&gt;
&lt;td&gt;20.7&lt;/td&gt;
&lt;td&gt;$0.0038&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Muse%20Ember&quot;&gt;Muse Ember&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;62.0%&lt;/td&gt;
&lt;td&gt;5,090&lt;/td&gt;
&lt;td&gt;10.7&lt;/td&gt;
&lt;td&gt;$0.0026&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Muse%20Ember&quot;&gt;Muse Ember&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;60.8%&lt;/td&gt;
&lt;td&gt;3,352&lt;/td&gt;
&lt;td&gt;7.5&lt;/td&gt;
&lt;td&gt;$0.0017&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Kimi%20K3.1%20Pro&quot;&gt;Kimi K3.1 Pro&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;60.3%&lt;/td&gt;
&lt;td&gt;1,656&lt;/td&gt;
&lt;td&gt;42.5&lt;/td&gt;
&lt;td&gt;$0.0175&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Kimi%20K3.1%20Pro&quot;&gt;Kimi K3.1 Pro&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;42.8%&lt;/td&gt;
&lt;td&gt;494&lt;/td&gt;
&lt;td&gt;6.3&lt;/td&gt;
&lt;td&gt;$0.0058&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Kimi%20K3.1%20Pro&quot;&gt;Kimi K3.1 Pro&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;42.8%&lt;/td&gt;
&lt;td&gt;446&lt;/td&gt;
&lt;td&gt;5.9&lt;/td&gt;
&lt;td&gt;$0.0053&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: Fewest tokens</title>
    <id>https://codecat.io/projects/some-bench/fewest-tokens/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/fewest-tokens/"/>
    <updated>2026-11-10T00:00:00Z</updated>
    <summary>71% on just 18 tokens</summary>
    <content type="html">&lt;p&gt;Gemini 4 Argon at low effort scores 71.1% on just 18 completion tokens per
task, as much as GPT-6.1 Sol gets at medium (72.2% on 231 tokens) or high effort (70.3% on 458),
in 3.9 seconds and for $0.0010 per test. It even beats Argon&amp;#39;s own medium effort (70.5% on 135).
For a cheap, fast answer that is still reasonably good, nothing else comes close.&lt;/p&gt;
&lt;p&gt;Every run that scores 70% or more, fewest tokens first:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;× Argon low&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Gemini 4 Argon · low&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;71.1%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;18&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3.9&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.0010&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 4 Argon · medium&lt;/td&gt;
&lt;td&gt;70.5%&lt;/td&gt;
&lt;td&gt;135&lt;/td&gt;
&lt;td&gt;7.7&lt;/td&gt;
&lt;td&gt;5.7&lt;/td&gt;
&lt;td&gt;$0.0022&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6.1 Sol · medium&lt;/td&gt;
&lt;td&gt;72.2%&lt;/td&gt;
&lt;td&gt;231&lt;/td&gt;
&lt;td&gt;13&lt;/td&gt;
&lt;td&gt;7.3&lt;/td&gt;
&lt;td&gt;$0.0030&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6 Astra · medium&lt;/td&gt;
&lt;td&gt;74.7%&lt;/td&gt;
&lt;td&gt;258&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;9.9&lt;/td&gt;
&lt;td&gt;$0.0166&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6.1 Sol · high&lt;/td&gt;
&lt;td&gt;70.3%&lt;/td&gt;
&lt;td&gt;458&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;td&gt;12.3&lt;/td&gt;
&lt;td&gt;$0.0053&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fable 5.2 · low&lt;/td&gt;
&lt;td&gt;72.5%&lt;/td&gt;
&lt;td&gt;474&lt;/td&gt;
&lt;td&gt;27&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;td&gt;$0.0353&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6 Astra · high&lt;/td&gt;
&lt;td&gt;77.5%&lt;/td&gt;
&lt;td&gt;504&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;15.9&lt;/td&gt;
&lt;td&gt;$0.0289&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fable 5.2 · medium&lt;/td&gt;
&lt;td&gt;73.3%&lt;/td&gt;
&lt;td&gt;509&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;7.1&lt;/td&gt;
&lt;td&gt;$0.0265&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Opus 5.5 · high&lt;/td&gt;
&lt;td&gt;70.3%&lt;/td&gt;
&lt;td&gt;608&lt;/td&gt;
&lt;td&gt;35&lt;/td&gt;
&lt;td&gt;8.8&lt;/td&gt;
&lt;td&gt;$0.0173&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fable 5.2 · high&lt;/td&gt;
&lt;td&gt;75.8%&lt;/td&gt;
&lt;td&gt;798&lt;/td&gt;
&lt;td&gt;46&lt;/td&gt;
&lt;td&gt;9.9&lt;/td&gt;
&lt;td&gt;$0.0372&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 4 Argon · high&lt;/td&gt;
&lt;td&gt;78.9%&lt;/td&gt;
&lt;td&gt;3,290&lt;/td&gt;
&lt;td&gt;188&lt;/td&gt;
&lt;td&gt;44.5&lt;/td&gt;
&lt;td&gt;$0.0337&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Muse Wonder · medium&lt;/td&gt;
&lt;td&gt;73.9%&lt;/td&gt;
&lt;td&gt;5,352&lt;/td&gt;
&lt;td&gt;306&lt;/td&gt;
&lt;td&gt;47.7&lt;/td&gt;
&lt;td&gt;$0.1082&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Muse Wonder · high&lt;/td&gt;
&lt;td&gt;74.8%&lt;/td&gt;
&lt;td&gt;7,284&lt;/td&gt;
&lt;td&gt;416&lt;/td&gt;
&lt;td&gt;66.1&lt;/td&gt;
&lt;td&gt;$0.1468&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Above about 500 tokens, more tokens buy little: the best runs gain a few points for several
times the tokens, and Muse Wonder spends over 400 times as many as Argon at low effort for
3.7 points more.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: New results: Fable 5.2, Muse Wonder, Nemotron 3 Ultra and Nemotron 3.5 Super VL</title>
    <id>https://codecat.io/projects/some-bench/updates/new-results-fable-5-2-muse-wonder-nemotron-3-ultra-nemotron/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/updates/new-results-fable-5-2-muse-wonder-nemotron-3-ultra-nemotron/"/>
    <updated>2026-11-10T00:00:00Z</updated>
    <summary>Added Fable 5.2, Muse Wonder, Nemotron 3 Ultra and Nemotron 3.5 Super VL. Best of them: Fable 5.2 high, 75.8%. Opus 5.5 was run again.</summary>
    <content type="html">&lt;p&gt;Added Fable 5.2, Muse Wonder, Nemotron 3 Ultra and Nemotron 3.5 Super VL. Best of them: Fable 5.2 high, 75.8%. Opus 5.5 was run again.&lt;/p&gt;
&lt;h2 id=&quot;added&quot;&gt;Added&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Fable%205.2&quot;&gt;Fable 5.2&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;75.8%&lt;/td&gt;
&lt;td&gt;798&lt;/td&gt;
&lt;td&gt;9.9&lt;/td&gt;
&lt;td&gt;$0.0372&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Muse%20Wonder&quot;&gt;Muse Wonder&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;74.8%&lt;/td&gt;
&lt;td&gt;7,284&lt;/td&gt;
&lt;td&gt;66.1&lt;/td&gt;
&lt;td&gt;$0.1468&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Muse%20Wonder&quot;&gt;Muse Wonder&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;73.9%&lt;/td&gt;
&lt;td&gt;5,352&lt;/td&gt;
&lt;td&gt;47.7&lt;/td&gt;
&lt;td&gt;$0.1082&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Fable%205.2&quot;&gt;Fable 5.2&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;73.3%&lt;/td&gt;
&lt;td&gt;509&lt;/td&gt;
&lt;td&gt;7.1&lt;/td&gt;
&lt;td&gt;$0.0265&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Fable%205.2&quot;&gt;Fable 5.2&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;72.5%&lt;/td&gt;
&lt;td&gt;474&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;td&gt;$0.0353&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Muse%20Wonder&quot;&gt;Muse Wonder&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;68.3%&lt;/td&gt;
&lt;td&gt;3,289&lt;/td&gt;
&lt;td&gt;29.5&lt;/td&gt;
&lt;td&gt;$0.0669&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Nemotron%203%20Ultra&quot;&gt;Nemotron 3 Ultra&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;50.0%&lt;/td&gt;
&lt;td&gt;8,496&lt;/td&gt;
&lt;td&gt;77.5&lt;/td&gt;
&lt;td&gt;$0.0206&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Nemotron%203%20Ultra&quot;&gt;Nemotron 3 Ultra&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;47.8%&lt;/td&gt;
&lt;td&gt;8,710&lt;/td&gt;
&lt;td&gt;77.8&lt;/td&gt;
&lt;td&gt;$0.0211&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Nemotron%203%20Ultra&quot;&gt;Nemotron 3 Ultra&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;47.8%&lt;/td&gt;
&lt;td&gt;8,902&lt;/td&gt;
&lt;td&gt;79.5&lt;/td&gt;
&lt;td&gt;$0.0215&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Nemotron%203.5%20Super%20VL&quot;&gt;Nemotron 3.5 Super VL&lt;/a&gt; · off&lt;/td&gt;
&lt;td&gt;26.4%&lt;/td&gt;
&lt;td&gt;5,380&lt;/td&gt;
&lt;td&gt;24.1&lt;/td&gt;
&lt;td&gt;$0.0027&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;h2 id=&quot;run-again&quot;&gt;Run again&lt;/h2&gt;
&lt;p&gt;The page shows the average of all runs; these are the newest run against the average before it.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Opus%205.5&quot;&gt;Opus 5.5&lt;/a&gt; · high&lt;/td&gt;
&lt;td&gt;71.4% → 68.0%&lt;/td&gt;
&lt;td&gt;636 → 552&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Opus%205.5&quot;&gt;Opus 5.5&lt;/a&gt; · low&lt;/td&gt;
&lt;td&gt;65.1% → 65.0%&lt;/td&gt;
&lt;td&gt;322 → 319&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href=&quot;/projects/some-bench/./?q=Opus%205.5&quot;&gt;Opus 5.5&lt;/a&gt; · medium&lt;/td&gt;
&lt;td&gt;67.8% → 68.3%&lt;/td&gt;
&lt;td&gt;503 → 472&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: Opus 5.5 and Fable 5.2</title>
    <id>https://codecat.io/projects/some-bench/release/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/release/"/>
    <updated>2026-11-10T00:00:00Z</updated>
    <summary>Is Opus 5.5 nerfed after Fable 5.2&#39;s release?</summary>
    <content type="html">&lt;p&gt;A theory goes around online that Anthropic&amp;#39;s older models get worse whenever a new one is
released. Some Bench happened to run Opus 5.5 before and after one release, Fable 5.2, with the
same tests and the same settings, so here is what that showed.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;At high effort, Opus 5.5 got shorter and slightly worse. At low and medium effort, nothing
moved beyond the usual run-to-run variation.&lt;/strong&gt; One release and one run after it can&amp;#39;t confirm
the theory, but the high-effort change is the largest token drop in our data.&lt;/p&gt;
&lt;h2 id=&quot;the-runs&quot;&gt;The runs&lt;/h2&gt;
&lt;p&gt;Fable 5.2 came out on 10 November, and the third run of Opus 5.5 is from that day.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Opus 5.5&lt;/th&gt;
&lt;th&gt;24 Sep&lt;/th&gt;
&lt;th&gt;22 Oct&lt;/th&gt;
&lt;th&gt;10 Nov, Fable 5.2 release day&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;High&lt;/strong&gt;: score&lt;/td&gt;
&lt;td&gt;71.4%&lt;/td&gt;
&lt;td&gt;71.4%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;68.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;High: harder variant, exact&lt;/td&gt;
&lt;td&gt;56.1%&lt;/td&gt;
&lt;td&gt;55.0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;50.0%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;High: completion tokens per task&lt;/td&gt;
&lt;td&gt;614&lt;/td&gt;
&lt;td&gt;656&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;552&lt;/strong&gt; (−16%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;High: seconds per task&lt;/td&gt;
&lt;td&gt;9.2&lt;/td&gt;
&lt;td&gt;9.8&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7.5&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;High: cost per test&lt;/td&gt;
&lt;td&gt;$0.0200&lt;/td&gt;
&lt;td&gt;$0.0179&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.0139&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Medium: score&lt;/td&gt;
&lt;td&gt;68.9%&lt;/td&gt;
&lt;td&gt;66.6%&lt;/td&gt;
&lt;td&gt;68.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Medium: completion tokens per task&lt;/td&gt;
&lt;td&gt;506&lt;/td&gt;
&lt;td&gt;499&lt;/td&gt;
&lt;td&gt;472&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Low: score&lt;/td&gt;
&lt;td&gt;65.0%&lt;/td&gt;
&lt;td&gt;65.3%&lt;/td&gt;
&lt;td&gt;65.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Low: completion tokens per task&lt;/td&gt;
&lt;td&gt;318&lt;/td&gt;
&lt;td&gt;324&lt;/td&gt;
&lt;td&gt;319&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;At high effort the model wrote 16% fewer completion tokens (which include its reasoning),
answered 23% faster, cost 22% less per test and lost 3.4 points, almost all of them on the harder variant. That
looks like less thinking per answer, not a different model: low effort, which thinks least,
didn&amp;#39;t change at all.&lt;/p&gt;
&lt;h2 id=&quot;is-that-more-than-noise&quot;&gt;Is that more than noise?&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;/projects/some-bench/release-runs.png&quot; alt=&quot;Change in score and in tokens between two runs of the same model and effort; the Opus 5.5
high-effort pair across the Fable 5.2 release sits furthest down, at −3.4 points and −16%
tokens&quot; loading=&quot;lazy&quot;&gt;&lt;/p&gt;
&lt;p&gt;Running the same model twice never gives exactly the same result. Every model we ran more than
once is on the chart, so the orange points can be compared with the normal spread in grey:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Score:&lt;/strong&gt; a 3.4-point drop is the largest in our data, but only just. Sonnet 5.5 at medium
effort dropped 3.0 points between 29 September and 22 October, with no release in between.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tokens:&lt;/strong&gt; a 16% drop stands out more clearly. Every other pair of runs stayed within 7%.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;what-this-doesn-t-show&quot;&gt;What this doesn&amp;#39;t show&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;One event.&lt;/strong&gt; To test &amp;quot;older models get worse at each release&amp;quot;, the same check has to hold
across several releases, not one.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No comparison run after the release.&lt;/strong&gt; Only Opus 5.5 was run again on 10 November. Running
Sonnet 5.5, Fable 5.1 and a model from another company on the same day would show whether
something changed for everyone, such as load on the provider&amp;#39;s side.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Why it changed.&lt;/strong&gt; From outside we can only see that the model writes less and scores lower
at high effort. Whether that&amp;#39;s a deliberate change, a different thinking budget or a
side effect of serving a new model, the numbers can&amp;#39;t tell.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;We&amp;#39;ll run Opus 5.5 at high effort again to see whether the drop holds, and repeat this check
at the next Anthropic release.&lt;/p&gt;
</content>
  </entry>
  <entry>
    <title>Some Bench: Best value</title>
    <id>https://codecat.io/projects/some-bench/best-value/</id>
    <link rel="alternate" type="text/html" href="https://codecat.io/projects/some-bench/best-value/"/>
    <updated>2026-11-10T00:00:00Z</updated>
    <summary>The top score isn&#39;t the best buy</summary>
    <content type="html">&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rank&lt;/th&gt;
&lt;th&gt;Model · effort&lt;/th&gt;
&lt;th&gt;Score&lt;/th&gt;
&lt;th&gt;Tokens per task&lt;/th&gt;
&lt;th&gt;Cost per test&lt;/th&gt;
&lt;th&gt;Seconds per task&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Gemini 4 Argon · high&lt;/td&gt;
&lt;td&gt;78.9%&lt;/td&gt;
&lt;td&gt;3,290&lt;/td&gt;
&lt;td&gt;$0.0337&lt;/td&gt;
&lt;td&gt;44.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;GPT-6 Astra · high&lt;/td&gt;
&lt;td&gt;77.5%&lt;/td&gt;
&lt;td&gt;504&lt;/td&gt;
&lt;td&gt;$0.0289&lt;/td&gt;
&lt;td&gt;15.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Fable 5.2 · high&lt;/td&gt;
&lt;td&gt;75.8%&lt;/td&gt;
&lt;td&gt;798&lt;/td&gt;
&lt;td&gt;$0.0372&lt;/td&gt;
&lt;td&gt;9.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Muse Wonder · high&lt;/td&gt;
&lt;td&gt;74.8%&lt;/td&gt;
&lt;td&gt;7,284&lt;/td&gt;
&lt;td&gt;$0.1468&lt;/td&gt;
&lt;td&gt;66.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;GPT-6 Astra · medium&lt;/td&gt;
&lt;td&gt;74.7%&lt;/td&gt;
&lt;td&gt;258&lt;/td&gt;
&lt;td&gt;$0.0166&lt;/td&gt;
&lt;td&gt;9.9&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Gemini 4 Argon at high effort has the best score, but it pays for it: 6.5 times the tokens of
GPT-6 Astra for 1.4 points more. On any reasonable token budget GPT-6 Astra is the better
model; at medium effort it still scores 74.7% on 258 tokens. Muse Wonder is fourth, but at
7,284 tokens and five times Astra&amp;#39;s cost per test, the score isn&amp;#39;t worth what it spends.&lt;/p&gt;
</content>
  </entry>
</feed>
