I Tested the Complaints About Opus 5. One Was True. One Wasn't. One I Couldn't Test.

Let me start with two important caveats. First, this is a private test suite. That is useful because the models are unlikely to have trained on it, but it also means you cannot independently check the tasks. You have to judge the method and take the numbers for what they are. Second, I found two small scoring problems while reviewing the results. I have left the published scores unchanged because fixing them would mean re-scoring frozen baseline runs. Correcting them would move the overall quality difference by about 0.01. It would not change the conclusion.

With that out of the way, here is what I wanted to find out.

For the past few weeks, I have heard the same complaints about Opus 5: it talks too much, expands the scope of simple tasks, and loses track of things in long conversations. Anthropic's own documentation says its default answers are longer than those of earlier models and that it may verify its work without being asked.

The problem is that most of these comparisons are based on memory. Someone compares a task they ran yesterday with a different task they ran last month. The prompt changed. The context changed. Maybe their patience changed too. It quickly turns into one set of impressions arguing with another.

I maintain a private evaluation suite for testing new models. It has 163 tasks covering coding, reasoning, factuality, long context, document work, instruction following, and structured output. I also have frozen results from the previous Opus model.

So I ran Opus 5 through the same suite and compared the results task by task.

I found three things. The first two were fairly straightforward. The third was the real surprise.

How I ran the test

I compared the two models on each task and used 2,000 bootstrap runs to calculate 95% confidence intervals.

I kept reasoning effort at the API default for both models. Anthropic recommends higher effort for some coding and agentic tasks on the new model, but changing that would make the comparison with my frozen baseline less useful. A separate high-effort run may score better, but that is a different test.

I did explicitly match the thinking setting. Opus 5 has thinking on by default, while the older model does not. Comparing those defaults directly would create an artificial gap, so both models used the same adaptive setting.

In total, the run produced 207 rows. The models attempted 163 tasks, and 154 could be scored cleanly. They used roughly 625,000 input tokens and 297,000 output tokens. The total cost was $11.59.

Finding 1: I found no meaningful drop in quality

Across 154 matched tasks, Opus 5 scored 0.018 lower than the previous Opus. The 95% confidence interval was −0.04 to +0.01. Since that interval includes zero, the result does not show a statistically clear quality regression.

There are a few tempting numbers in this table, but they need context.

The everyday category is the only one whose confidence interval does not cross zero. Even there, the difference is 0.068. My measured judge-disagreement margin for that category is 0.07, so the result is still within judge noise.

The larger drops in coding and structured_hard come from only eight and four tasks. Their intervals touch zero, and both categories include the scoring issues I mentioned earlier. Fixing those issues would move the overall result from −0.018 to roughly −0.01.

Opus 5 also did better on structured output and the harder everyday tasks.

The simple reading is this: whatever users dislike about the new model, this test does not show that its answers are generally less correct.

Finding 2: yes, it is much more verbose

On the same prompts, Opus 5 produced 1.81 times more output tokens. The 95% confidence interval was 1.62x to 2.02x.

I calculated this using the geometric mean of the token ratio for each task. That detail matters. A simple arithmetic mean can be badly distorted by short answers. If one model answers in 5 tokens and another uses 50, that single task creates a 10x ratio even though the absolute difference is tiny.

I made that mistake on another model comparison in this suite. It turned a real 1.05x difference into an apparent 1.26x difference. The geometric mean gives a much more honest picture here.

The extra words were not limited to one kind of task. But they were most noticeable where the answer should have been short: factual questions, small coding tasks, and document Q&A.

The two reasoning categories were different. Opus 5 used slightly fewer visible output tokens there, at 0.94x.

That changes how I would describe the problem. It is not that Opus 5 rambles while solving difficult problems. It is that it often turns a one-line answer into a paragraph.

The quality stayed about the same, but the output bill nearly doubled. Even when cost is not a concern, the extra explanation can get tiring when all you wanted was the answer.

Finding 3: I could not properly test long context

This was the complaint I most wanted to check. Forgetting earlier context is a serious problem for a coding model, especially one advertised with a larger context window.

But I could not get a clean result because Opus 5 refused too many of the tasks.

Twelve tasks triggered at least one refusal. Every refused request returned HTTP 200 with a refusal stop reason and a cyber-safety category. None of them was billed.

Three things stood out.

First, the tasks were harmless. The long-context material is a synthetic collection of internal company documents, including IT incident reports with severity labels. I scanned the whole corpus for security vocabulary and found zero occurrences of breach, malware, exploit, credential, phish, ransom, CVE, or attack. Some refused coding tasks were even more ordinary: count unique IP addresses in a log file, or debug a Python traceback.

The material looks a little like security work without actually being security work. That seemed to be enough to trigger the filter.

Second, the refusals were inconsistent. The exact same prompt and context could pass once and fail the next time. One task failed all seven attempts. Another failed once and then passed twice.

Some requests stopped before producing any text. Others ran for several seconds and started streaming tokens before the refusal appeared. This did not behave like a simple, fixed filter applied before generation.

Third, length did not explain it. The smallest long-context file could fail while a file twice its size passed. Two tasks using the same context also behaved differently.

Only five of the nine long-context tasks could be scored. That is below my minimum coverage requirement of 70%, so the category gets no verdict.

The table above shows a −0.125 long-context difference, but that number only covers the tasks the model chose to answer. It is a biased subset and should not be treated as evidence of a long-context regression.

This can quietly break a leaderboard

If an evaluation treats a refusal like an ordinary wrong answer, it records a zero. That zero then looks like a loss in capability.

This is especially dangerous when refusals are random. A benchmark that runs each task once may get a different score every time. Unless the evaluation records why the response stopped, a safety false positive becomes an apparent model failure.

That is exactly what could have happened here. I started this test because people said the model had become worse at long context. A naive evaluation would have produced numbers that seemed to confirm the complaint.

I caught it because my harness records stop reasons. Had I run the same test a few weeks earlier, before adding that field, I might have published a long-context regression that the test did not actually prove.

There are benchmarks designed to measure refusals. That is not what happened here. These refusals leaked into a normal capability test where nobody was trying to test safety behaviour.

I also found a bug in my scorer

While checking the unusual results, I found another problem, this time in my own code.

My scorer took the last fenced code block from an answer and ran it. That had worked well for years. Opus 5, however, sometimes put the solution in one code block and added a usage example in a second block.

The scorer grabbed the example instead of the solution. It then failed with a NameError because the class used by the example was defined in the earlier block. The model had solved the task correctly, but my scorer gave it a zero.

This is one of the two scoring issues I flagged at the top. It touched only a few stored rows, and correcting it and the other one together moves the overall figure from −0.018 to about −0.01, so the conclusion holds either way. The lesson is bigger than those rows.

A change in writing style looked like a change in coding ability.

Any evaluation that blindly takes the first or last code block has the same weakness. A new model release can change how answers are formatted even when the underlying capability stays the same. Prompts need checking, but scorers do too.

What I took away from the test

If you are deciding whether to use Opus 5, the verbosity complaint is fair. In this suite, it used about 1.8 times more output tokens for roughly the same quality. Most of the extra wording appeared on tasks that should have had short answers.

I did not find evidence of a broad quality regression.

I also cannot give a fair verdict on long-context performance. Too many tasks were refused, and the tasks that survived were not a representative sample.

If you run model evaluations, I would make three changes:

• Record the stop reason for every request. An HTTP status code is not enough.
• Treat coverage as a result. If a model answers only five of nine tasks, report that instead of averaging the five and pretending the other four never existed.
• Recheck your scoring and extraction rules whenever a new model arrives. A harmless formatting change can turn a correct answer into a zero.

This run added a new failure mode to my list: a random refusal that looks like a capability regression.

That one worries me more than an obvious broken response. An obvious failure makes you investigate. This kind gives you a believable number, in the exact area you were already worried about, and invites you to publish it.

Full caveats: this is a private suite and cannot be independently verified. Both models used default effort, not the higher setting recommended for some coding work. The judge panel includes a model from the same family as the model being tested; I audited for family bias across five model families and found none, but it is still worth disclosing. Several categories have small samples, especially long context (5), structured_hard (4), and media_ui (2). Two known scoring issues remain in the reported figures; correcting them moves the overall quality difference to about −0.01. The total run cost was $11.59. Refused requests were not billed.