https://ringarc.ai/labs/ Vikas's Open Model Benchmark updates and short engineering notes on AI in practice — benchmarking open models, building agents, embedding AI into real businesses. en Sun, 17 Aug 2026 15:00:00 +0530 https://ringarc.ai/labs/tech/claude-reader tech-claude-reader Sun, 17 Aug 2026 15:00:00 +0530 claude-reader is a small Textual TUI that tails the JSONL transcript Claude Code already writes to ~/.claude/projects/, keeps only the assistant text blocks and your prompts, and renders them as markdown in a sidebar plus reading pane that updates twice a second. No hooks into Claude Code, no server, works in any terminal split and over ssh. Three design details dictated by the neighbouring pane (mouse off by default, no clock in the header, clipboard through tmux load-buffer), and the outside review that found four real crashers (null message shapes, a multibyte split at EOF, unbounded memory, no argparse) before the 0.2.0 release. pipx install claude-reader. https://ringarc.ai/labs/tech/what-agent-apps-actually-use tech-what-agent-apps-actually-use Sat, 15 Aug 2026 22:30:00 +0530 I read the dependency files of 703 starred GitHub repos tagged as AI agents. Of 257 apps with a detectable LLM stack, 145 use only raw SDKs, more than all frameworks combined. More apps use LangChain's parts (35) than the full framework (26), LangGraph has overtaken LangChain among shipping apps, and production-grade apps keep frameworks slightly MORE often than POC-like ones (25% vs 15%). Also: the AutoGen "exit wave" that collapsed from 52 issues to 2 once the search term was disambiguated from GNU autogen.sh. https://ringarc.ai/labs/tech/context-rot-lost-middle tech-context-rot-lost-middle Fri, 14 Aug 2026 17:30:00 +0530 A million-token context window does not mean the model can use a million tokens. What the research actually shows: the U-shaped "lost in the middle" curve, attention sinks (remove the first 4 tokens and Llama-2's perplexity jumps from 5.4 to 5,158), positional decay, distractors beating raw length as the failure driver, and why needle-in-a-haystack is easy mode: on NoLiMa, 11 of 13 models claiming 128K+ fall below half their short-context score at just 32K. Plus the 5-minute experiment anyone can run, and practical prompt habits that follow from the mechanisms. https://ringarc.ai/labs/tech/qwen36-35b-local-frontier tech-qwen36-35b-local-frontier Thu, 13 Aug 2026 17:00:00 +0530 Qwen3.6-35B-A3B, running 4-bit on a single 16GB RTX 5070 Ti, lands at +0.005 [−0.01, +0.02] on the private 163-task benchmark: statistically at the frontier, tied for the best point estimate in the series, first local model at the line, perfect scores on reasoning, factuality, long context and instruction following, 66 tokens per second, $1.24 for the whole run. Also: why the 35B MoE fits a 16GB card while the 27B dense model doesn't, the reasoning-budget and language-drift harness fixes, and a three-week-old scoring correction for Gemma 4 12B. https://ringarc.ai/labs/tech/opus5-complaints-measured tech-opus5-complaints-measured Wed, 12 Aug 2026 10:00:00 +0530 Opus 5 against a frozen previous-Opus baseline on 163 private tasks: 1.81x more output tokens for statistically identical quality, no measurable capability regression, and six of nine long-context tasks refused as cyber content on a corpus of synthetic office memos. Why a stochastic refusal recorded as a zero can manufacture a capability regression on any leaderboard, plus the scorer bug that turned a formatting change into apparent capability loss. https://ringarc.ai/labs/tech/rag-five-debates tech-rag-five-debates Mon, 10 Aug 2026 12:00:00 +0530 Part 1 of the RAG series. There is no RAG debate, there are five: grep vs vectors, long context, GraphRAG, CAG, and memory. Pulling them apart shows why the boring baseline keeps winning, what chunking and indexing actually need, when reranking earns its keep, and the signals that you genuinely need the advanced tier. https://ringarc.ai/labs/tech/local-inference-comparison tech-local-inference-comparison Fri, 07 Aug 2026 21:30:00 +0530 Three engines, three model sizes, one 16GB GPU. Why vLLM collapsed to 33 tok/s on identical weights and then won the batch round five times over, how llama.cpp serves an 18GB MoE model at 71 tok/s via expert offload, and the one env var every Ollama box should set. https://ringarc.ai/labs/tech/qwen38-max-vs-kimi-k3 tech-qwen38-max-vs-kimi-k3 Fri, 07 Aug 2026 18:00:00 +0530 The long version of the day-one Qwen3.8-Max intake: first positive point estimate against frozen frontier anchors, the head-to-head dimension split with Kimi K3, two hand-verified failure stories, the 3x cost gap, and three serving traps (thinking off by default, xhigh cost trap, the multimodal token floor). https://ringarc.ai/labs/tech/project-brain tech-project-brain Fri, 07 Aug 2026 08:00:00 +0530 The story of project-brain, now open source (MIT): a long-standing pain (re-explaining my own projects to Claude Code), a 30-day transcript-deletion discovery, a research pass through the memory-tool landscape, and the layered plain-Markdown architecture we landed on — which turned out to match Karpathy's LLM-wiki pattern and Google's OKF spec almost exactly. https://ringarc.ai/labs/notes/#qwen38-max-day-one note-qwen38-max-day-one Tue, 04 Aug 2026 08:00:00 +0530 Day-one run of Qwen3.8-Max on my private benchmark: first model to land a hair above the frontier reference line, clean on hallucination baits and full-repo coding — but verbose in everyday answers. Roughly 3x cheaper per task than Kimi K3. https://ringarc.ai/labs/notes/#cloudflare-meerkat note-cloudflare-meerkat Mon, 03 Aug 2026 09:00:00 +0530 Meerkat uses QuePaxa to get strong, globally consistent control-plane coordination without a single leader — no timeouts for progress, every replica can accept writes. https://ringarc.ai/labs/notes/#local-engines-benchmark note-local-engines-benchmark Sat, 01 Aug 2026 09:00:00 +0530 Three engines, three model sizes, one 16GB GPU. The answer isn't "which engine is best" — it's: does your model fit, and how many people are hitting it? https://ringarc.ai/labs/notes/#how-local-inference-works note-how-local-inference-works Fri, 31 Jul 2026 09:00:00 +0530 Running an LLM locally is not a compute problem. VRAM, system RAM, PCIe: wide pipe, medium pipe, straw — and why MoE expert offload lets a 16GB card run an 18GB model at 70 tok/s. https://ringarc.ai/labs/notes/#ai-worming-through-word note-ai-worming-through-word Thu, 30 Jul 2026 10:00:00 +0530 A disclosed vulnerability class turns Copilot for Word into a document-borne AI worm via hidden instructions. No complete fix yet — practical habits for teams using Copilot. https://ringarc.ai/labs/notes/#tiny-ocr-long-documents note-tiny-ocr-long-documents Thu, 30 Jul 2026 09:00:00 +0530 Baidu's Unlimited OCR reads 40+ page PDFs in one go, runs locally, and keeps whole-document context — OCR catching up with how documents actually look. https://ringarc.ai/labs/notes/#mcp-goes-stateless note-mcp-goes-stateless Wed, 29 Jul 2026 09:00:00 +0530 The 2026-07-28 MCP spec makes every request self-contained, gateway-routable, and load-balancer friendly. MCP servers start to look like standard microservices. https://ringarc.ai/labs/notes/#photon-llm-efficiency note-photon-llm-efficiency Tue, 28 Jul 2026 09:00:00 +0530 Fujitsu's PHOTON processes semantic units hierarchically instead of token-by-token: up to 475× multi-query throughput per GPU in their tests, with built-in best-of-N aggregation. https://ringarc.ai/labs/notes/#inkling-on-mfold note-inkling-on-mfold Mon, 27 Jul 2026 10:00:00 +0530 Leaderboards put Inkling mid-pack; on realistic everyday work it's statistically tied with the frontier — the best open-model result on my suite. Both verdicts are true. https://ringarc.ai/labs/notes/#why-build-my-own-benchmark note-why-build-my-own-benchmark Mon, 27 Jul 2026 09:00:00 +0530 Private tasks, frozen frontier anchors, a bias-audited 3-model judge panel, and deployment-aware local runs — the design choices behind Mfold and why public leaderboards weren't answering my two questions. https://ringarc.ai/labs/notes/#hotlang-honest-version note-hotlang-honest-version Fri, 24 Jul 2026 09:00:00 +0530 A purpose-built decision-kernel language: no heap, no unbounded loops, compile-time latency guarantees — and the same source emits verified-identical Verilog. https://ringarc.ai/labs/notes/#gpt56-cache-write-pricing note-gpt56-cache-write-pricing Sat, 18 Jul 2026 09:00:00 +0530 GPT-5.6 charges 1.25x input rate for cache writes, and caching is on by default with no off toggle — the explicit-mode workaround and when it matters. https://ringarc.ai/labs/open-model-tracker tracker-2026-08-04-qwen38max Tue, 04 Aug 2026 09:00:00 +0530 Qwen3.8-Max posts the series' first positive frontier-lag point estimate: +0.005 [−0.01, +0.02] — statistically at the frontier, and the first model clean on both historic failure modes (hallucination baits + full-stack repo). Statistically tied with Kimi K3 head-to-head. API-only at evaluation; open weights announced for ~Aug 10. https://ringarc.ai/labs/open-model-tracker tracker-2026-07-27-inkling Mon, 27 Jul 2026 09:00:00 +0530 Inkling (Thinking Machines, 975B/41B MoE, Apache-2.0) lands at −0.014 [−0.04, +0.01] — statistically at the frontier, and the first open model to hold the full-stack repo band. Weakness: invented confident plots for both hallucination-bait films. https://ringarc.ai/labs/open-model-tracker tracker-2026-07-22-launch Wed, 22 Jul 2026 09:00:00 +0530 The suite goes live: Opus 4.8, Sonnet 5, Haiku 4.5 and GPT-5.4-mini frozen as calibration anchors; Kimi K3 (−0.019, statistically at frontier), gpt-oss-20b (−0.078, ties GPT-5.4-mini at $0 on a 16GB GPU), Qwen3-Coder-30B and Gemma 4 12B (~−0.20) measured.