TLDR
On 5 September I put in writing that I was hating the output of the model I was using most.
Two days later I blind-tested eight models on six of my own messages. The model I had complained about tied for first, and Fable 5.1, the most expensive in the set, scored zero.
Six questions and one judge cannot rank eight models. Evan Miller’s error-bars paper puts the sample needed at roughly a thousand questions.
Across the last seven days, cache reads were 92.9% of every token I spent and output was 0.69%.
On a week shaped like that, the model with twice the list price prices at 1.33 times the week.
I blamed a model in writing. A blind test two days later and a week of token numbers showed I was blaming the wrong thing.
The complaint is on record before the test is. On 5 September I put it in writing:
“I’m kind of hating Opus as the output”
That was the same day I ran a small blind comparison. Three models, five of my own prompts, answers shuffled so I could not see which came from where. I picked Opus 5 three times out of five, Sonnet once, Fable once. So I was complaining about a model and choosing it in the same afternoon, which is the sort of result you either bury or go and test properly.
What the blind test actually judged
Two days later I ran the bigger version. Eight models from three vendors, six of my own messages, every answer shuffled and unlabelled, and every model stripped down to the same plain brief: no rules file, no repo, no tools. One judge, me.
The size of that comes first, because the size is the whole caveat. Evan Miller’s error-bars paper puts the sample needed to detect a three-point difference at 80% power at roughly a thousand questions. I had six, and one judge who had put his opinion in writing two days earlier. Six questions cannot rank eight models. The full score table sits at the end of this piece, where it can be read for what it is, which is one person’s afternoon.
What the six questions did do was contradict me. The model I had complained about in writing two days earlier tied for first, level with gemini-3.8-flash-high. The most expensive model in the set, Fable 5.1, scored zero.
The obvious way that result is wrong is that I might have been scoring length. Of the three medians I have, gpt-6-astra answered in 94 words, Opus in 302, Fable in 366. Zheng and colleagues call this verbosity bias, a judge favouring “longer, verbose responses, even if they are not as clear, high-quality, or accurate as shorter alternatives.” LMSYS found it in their own arena and said so plainly: “We controlled for the effect of length and markdown, and indeed, the ranking changed.” My pick was shorter than the model I scored zero. That does not clear me. It only means length by itself does not explain what happened.
My own first reading of the result was habit:
“I think that is due to how I have been using Opus as my daily driver since the switch a couple of weeks ago.”
And on how I actually scored the six:
“I didn’t even take previous context into account in this scenario. I just picked the answer which seemed the most well-rounded.”
So what was I hating?
The test handed every model a plain brief with no rules file and no repo attached. That is a bare model answering a question cold. What I had been complaining about was the same model inside my working setup, where a normal session was replaying about 287K tokens of context to it on every turn, and a fresh desktop session started at 124K to 157K tokens before I typed a word. One name, two different objects. One of them is a model answering a question. The other is a model plus everything I have ever told it, replayed at full length, every time I press return. I had been blaming the first one for what the second one did.
The bill says the same thing
Two words first. A cache read is the charge for handing the model context it has already seen, and a cache write is the charge for storing that context the first time. Last seven days to 19 September, across 5,385 assistant messages: cache reads 772,347,049 tokens, or 92.9% of everything. Cache writes 53,408,155, or 6.4%. Output, the part where a model writes something, 5,709,243 tokens, or 0.69%. Fresh input I typed myself, 61,104 tokens, or 0.007%.
Picture a taxi meter in stopped traffic. The car is the model. The distance driven is the output, the part that actually takes you somewhere. The meter is the context replayed on every turn, and it ticks whether the car moves or not. On my week, 92.9% of the tokens were the meter and 0.69% was distance. Changing cars does not stop a meter, and my only complaint had been about the car.
One eight-minute session in August came to $7.52 at list rates. Cache reads were $7.45 of it. 7.45 / 7.52 = 99% of an eight-minute conversation, spent re-reading what had already been said.
I pay a flat subscription, so what follows is the shape of the bill rather than a bill I was charged. Priced at live list rates, that same week of tokens comes to $172.60 on Haiku 4.5, $345.20 on Sonnet 5, $863.01 on Opus 5 and $1,146.76 on Fable 5.1. Cache reads alone account for $77.23, $154.47, $386.17 and $193.09 of those four totals.
That last pair is the one worth sitting with. Fable’s list price is twice Opus’s, $10 per million input tokens against $5. Its cache reads are half the price, $0.25 per million against $0.50. On a week that is 92.9% cache reads, the model that costs twice as much prices at 1,146.76 / 863.01 = 1.33 times the week. Model choice moved a third of it. The meter set the rest.
I write the division out instead of the answer for a reason I have given before:
“That helps me understand things better”
The vendor’s own cost guide reports the same shape from the inside. Anthropic finds agent loops read a median 84% of their input from cache, and calls cache reads “routinely the largest single component of task cost, making caching worth more than most model-choice decisions”. Their page on choosing a model is shorter about it: “Tuning effort is often a better lever than switching models.” The same guide reports a case where “a multi-model configuration that looked cheaper than the default single model cost more than that same model at lower effort”.
What a designed loop looks like
“You do not need, for example, Fables or Astra, whereas if you are letting your AI do grunt work, then a haiku or a sonnet is good enough. It is all about designing the workflows or the loops.”
The part of my own setup that already follows that sentence is the part I never complain about, which is the tell. The scheduled scripts call Haiku 4.5 first. On a non-zero exit, a timeout, empty output or a context overflow they escalate once to Sonnet 5. On an authentication failure they do not escalate at all. If the escalation fails too, they fail closed and return nothing rather than return something plausible.
Across 3,471 logged calls from 29 May to 18 September, 12 escalated and 2 hit a contract failure. That ledger covers several model families, so it is 12 of 3,471 calls, not 12 of 3,471 Haiku calls. 12 / 3,471 = 0.35% of runs needed the larger model.
The same lesson shows up when I get it wrong. Three subagent calls of very different size all landed close together: 122,036, 125,898 and 133,477 tokens, so 133,477 - 122,036 = 11,441 between the largest and the smallest. What I asked each of them to do barely moved the number. The context each one was handed set it, and I had handed all three the same pile.
The scheduled scripts get a fixed brief, the smallest model that clears the job, and only the context the job needs. Same vendor, same week, same me. Different loop, and no complaint attached to it.
Which leaves the line I would rather not write. I am drafting this in a Fable 5.1 session, the model that scored zero on my own test. This week that model carried 2,205 of my 5,385 assistant messages, more than Opus at 1,600, against a rule I wrote for myself saying not to default to it. My reason for that: my own choice. I feel like Opus has been too verbose all the time. The test’s own medians ran the other way: Opus 302 words, Fable 366.
Of the 831,525,551 tokens that week, 5,709,243 were a model writing something new. The rest was the meter.
Appendix: the full eight-model score table.
☕ Thanks for reading Product Field Notes!
If you have ever swapped models to fix a bill that was really your own context replaying on every turn, this one was written for you, so hit subscribe and I will keep sending them. And if you know a builder about to pay for a bigger model to fix a loop they have not looked at yet, send it their way first.