Same tier, different personalities: Qwen3.8-Max vs Kimi K3 on my private benchmark

Alibaba shipped Qwen3.8-Max on August 3rd. By the next evening it had been through my entire benchmark, and I had an answer to the only question I really care about when a new model drops: everyone says it's frontier-level, but is it actually?

Short version: yes. It's the first model I've tested that lands a hair above my frontier reference line, and it's statistically tied with Kimi K3, my best open contestant so far. But the two models get to the same tier in completely different ways, and the details are more interesting than the headline. This post is the long version of my LinkedIn post: the numbers, the failure stories, and the serving traps I hit on the way.

How the benchmark works, and why you can't see the tasks

I keep a private benchmark of around 163 tasks across 18 dimensions, all based on realistic everyday work: coding, repo-level bug fixes and test writing, document extraction, OCR, long-context retrieval, structured output, tricky reasoning, and awkward judgment calls with no clean answer. Every task is something I, or a small business I work with, would actually need done.

Four frontier models (Opus, Sonnet, GPT, Gemini) are frozen in as reference anchors. They were evaluated once in July 2026 and never re-run. Every new model gets scored against the anchors on identical content, task by task, and the whole thing boils down to a single lag number: how far behind or ahead of the frontier line the model sits, with a confidence interval around it. A panel of judge models does the first scoring pass, I audit a sample by hand every cycle, and any suspiciously low score gets re-inspected before it's allowed to stand.

Why keep the tasks private? Contamination. The moment a benchmark's tasks are public, they end up in the next training run, and the benchmark stops measuring the model and starts measuring memorization. So what you'll see here is scores, deltas, and failure stories. Never the prompts. The methodology summary and full standings live on the tracker page.

The headline: first positive point estimate

Qwen3.8-Max lands at +0.005 [−0.01, +0.02] against the frozen frontier line. Statistically that reads as "at frontier, within noise". But it's the first model in the series whose point estimate is positive at all. The series so far:

Eleven of the eighteen dimensions came back at a flat 1.00, meaning indistinguishable from the frontier anchors on the same content: both reasoning bands, both structured-output bands, both instruction-following bands, factuality core and hard, long context, document extraction, and OCR. Coding sat at 0.98, and reading UI screenshots at 0.95.

One asterisk before anything else: at evaluation time this is an API-only model. The weights are promised for around August 10, and the license hasn't been announced. Until they actually land, this is a proprietary frontier datapoint sitting in an open-model tracker on a promise.

The two classic failure modes, both cleared

Every strong open model I've tested so far has broken on one of two things. Qwen3.8-Max is the first to clear both in the same run.

Hallucination baits. The suite contains tasks that are traps by design: questions about things that don't exist, premises that are quietly false. Fake movie titles are my favourite example. Inkling, otherwise excellent, invented confident plot summaries for both fake films. Qwen3.8-Max is the first open contestant to simply say "that doesn't exist" every single time. Factuality came back 1.00 on both the core and hard bands.

Full-repo coding. Small, isolated coding problems stopped separating models a while ago, nearly everything aces them now. Where open models fall off is repo-level work: a real codebase, a real bug or feature, tests that must pass. Kimi K3 scores 0.83 there. Qwen3.8-Max held 0.93, with 17 of 18 repo tasks at 0.83 or above.

The dents, both verified by hand

Two dimensions came back below frontier, and both survived re-inspection as genuine model behaviour rather than scorer error. That's actually a first too. Every previous intake caught at least one scoring bug that was unfairly taxing the whole field. The harness is settling down.

Everyday answers: 0.91, and it's a trait, not bad luck. Ask it to explain UPI to a non-technical relative in plain words, with a word cap, and you get back nearly double the cap, stuffed with unexplained acronyms like VPA and PSP. It did this on both the clean and the messy phrasing of the task, and the judge panel was unanimous on the same two problems both times. That's not a random miss, that's the model's default voice fighting the assignment. Short, plain answers for regular people is the one place where its personality works against it. Kimi is clearly better at this.

Repo work: one hard zero, and it's an instructive one. The 0.93 repo score hides a single 0.00 on a test-writing task. The model wrote 92 tests against a small repo in the suite, and 88 passed. Three failed because it called a helper function with a keyword argument that function doesn't have. The function it had itself written, moments earlier. The last one failed because it asserted behaviour the documentation never promised, and its own inline comment ("let's check regex") shows it knew it was guessing. My rule for test-writing tasks is strict: the suite must pass on correct code before any credit is given, because a test suite that fails on correct code is worse than no suite at all. The zero stands.

Both dents come from the same habit: too much confidence, too much output. Too many words in the everyday answers, too many assertions in the test suite.

Head-to-head with Kimi K3

Compared task by task across 160 tasks, Qwen3.8-Max comes out at +0.022 [−0.01, +0.05]. Statistically tied, and equivalent on 8 of the 18 dimensions. Underneath, the split is remarkably clean:

Kimi's edge on hard judgment calls deserves its own sentence. On my hardest category, messy ethical dilemmas with no clean answer, Kimi K3 scored perfect on every single one. No other model I've tested has done that, including the frontier anchors. Qwen wasn't bad there, it engaged every dilemma head-on and scored right around where the frontier models do. But Kimi's sweep stands alone.

So, same tier, different shapes. Kimi is the better conversationalist, the one you'd want talking to people. Qwen is the stronger workhorse for documents, repos, and facts. If you're picking one for a pipeline, that table above is the decision.

The quiet headline: cost

Running my entire suite through Qwen3.8-Max cost $3.73, generation plus judging. Per task, it came out roughly 3x cheaper than Kimi K3, generated about 40% fewer tokens, and was more than twice as fast. In absolute terms it's mid-field: 44 to 73 tokens per second, first token in 0.6 to 2.2 seconds. Much faster than Kimi in thinking mode, well behind Inkling's 150 to 370 tokens per second. List pricing is $2 in, $6 out per million tokens.

Frontier-level answers at a third of the cost is quietly the biggest story in this release. Benchmark threads argue about half-points of quality. A 3x cost gap at the same quality tier is what actually changes deployment decisions.

Serving traps, read this before you integrate it

Every model I take in seems to ship with one configuration surprise. Qwen3.8-Max ships with three.

1. Thinking is OFF by default. That's a reversal from Qwen3.7-Max, where it was on by default. If you upgrade in place and don't notice, you're suddenly running a non-reasoning model and wondering why quality dropped. This is the fourth time I've logged a vendor flipping the thinking default, so at this point: always check.
2. Once you turn thinking on, the default effort is the maximum tier (xhigh), which is a cost trap in the opposite direction. I pinned medium for the whole run, same as I did for its peers. The officially supported levels are low, medium, and xhigh.
3. A new trap I hadn't seen before: the multimodal token floor. Send any request with an image and the API silently pins a thinking budget of 32,768 tokens, then rejects the request with a 400 error unless your max_completion_tokens is higher than that. Text-only requests are unaffected. So if your pipeline sets one modest max-tokens cap everywhere and occasionally attaches images, only the image requests will fail, and the error message won't obviously tell you why.

Caveats, honestly

• Single run, hosted. The API rejects temperature control in reasoning mode, so I have no variance estimate. Differences inside the confidence intervals are noise. Read the tiers, not the decimals.
• Effort pinned at medium, not the xhigh default. My numbers may understate the model's ceiling on hard tasks. Though everything hard was already at 1.00, and the verbosity problem wouldn't plausibly improve with more thinking.
• I'm measuring at a different altitude than Alibaba is marketing. Their release blog is all about long-horizon agentic work: multi-day autonomous coding runs, 500-turn chip design loops. My repo tasks are single-shot. Their own headline numbers (Terminal-Bench, SWE-bench Pro) come from agentic harnesses, exactly the kind of thing I don't measure yet. The community's day-one signals don't contradict my result, but whether that agentic premium is real, I simply can't say from here.

What happens next

The promised open weights are the part I'm actually waiting for. My local test rig is a single 16GB GPU with 128GB of system RAM, so the model I really want is the 27B when it ships. The day the weights drop, I'll run it locally and put it through the full suite. That's the moment "frontier-quality answers" meets "hardware a small business can actually own", which is the question this whole tracker exists to answer.