Everyone Tells You Basic RAG Is Dumb. It Is Not!

Part 1 of a short series for people who are tired of RAG content. The pitch is simple: most of what you scroll past is written about extreme cases or written to sell something. If you are one of the 90% of engineers building a normal retrieval system over normal documents, this series is the map and the playbook, and then you can stop reading RAG posts with a clear conscience.

Two admissions before we start, because this article is about to accuse a lot of people of selling things with made-up numbers. That 90% is a made-up number. I believe the shape of it, and I will argue for the shape as we go, but nobody has surveyed retrieval systems and I am not going to pretend otherwise. And yes, this series is also content, written by someone making a claim in this field. Judge it the way I am about to judge the rest of the feed: by whether the claims survive checking.

Quick level-set before we start, because I want this readable whether you shipped your first RAG system last month or you manage the team that did. RAG stands for retrieval-augmented generation. Instead of hoping the model already knows your company's documents, you search those documents first, pull out the most relevant pieces, and paste them into the prompt along with the user's question. The model answers from what you gave it. Every debate in this article is really about one step in that process: how you find the right pieces to paste in.

If you follow AI engineering content on LinkedIn or Medium, you have seen this post a hundred times. It opens with your RAG demo working beautifully in a notebook and dying in production. Then it tells you the real problem is retrieval, not the model. Then comes a numbered list of mistakes, and chunking is always mistake number one. Then the fix list: semantic chunking, hybrid search, reranking, metadata filters, evals. Then a paywall, or a consulting pitch, or the author's own tool.

I collected ten of these articles over the last few weeks and actually read them side by side. Two things jumped out.

First, every single one presents "the problem is retrieval, not the LLM" as a contrarian insight. When every article makes the same contrarian claim, that claim is the consensus. Nobody is out there defending the other side. (The consensus is also a little too neat: swap the model under an identical pipeline and answer quality moves, so the model was never irrelevant. But retrieval-first is directionally right, and more importantly, it is not the brave minority position the framing implies.)

Second, they contradict each other, directly. One article lists "Graph RAG overengineering" as a top-ten production mistake. Three others sell knowledge graphs as the fix. One says start with dumb fixed-size chunks, log real queries for two weeks, and measure before you optimize anything. Two others say fixed-size chunking is the number one mistake and you should fix it immediately. One article insists nothing counts until you measure it, then offers a "Hit Rate above 70%" threshold with no data behind it. And my favorite: a whole article about how Google's Open Knowledge Format is replacing the vector database. OKF is real, I checked. It is a 450-line spec from Google's data analytics team that describes a folder of Markdown files with YAML frontmatter. It has one required field. Google Search does not read it. It replaces nothing. And the comparison never made sense in the first place: OKF is a convention for how to write files down, and a vector database is a way to search them. A tidy folder of Markdown is nice, but something still has to find the right file when a user asks a question. A format cannot replace a search mechanism any more than a filing cabinet replaces the person who knows which drawer to open.

So the average AI engineer reads all this and comes out worse than they went in: loaded with new terms, buried in techniques with no way to rank them, and still clueless about what to actually build. Plus a vague feeling that whatever they shipped last quarter is now embarrassing.

I think the confusion has a specific cause, and it is not that the authors are wrong about the techniques. "The RAG debate" is not one debate. It is five separate debates that all got tagged #RAG. Each one has different evidence, a different domain where it holds, and a different answer. Once you pull them apart, most of the haze clears on its own.

Debate 1: Grep vs. vectors

First, the two contenders in plain terms. Grep is the old Unix tool for exact text search: you give it a word, it finds every line containing that word. Vector search (embeddings) works differently. It converts text into lists of numbers that capture meaning, so "refund" and "give me my money back" end up close together even though they share no words. Vector search is the engine inside most RAG tutorials, so "just use grep" sounds like heresy.

Except it worked. This debate started when the Claude Code team dropped vector search entirely and let the agent grep the codebase instead, and results improved. Windsurf, Amp and others went the same way. An Amazon paper measured a grep-based agent (literally ripgrep and pdfgrep behind a ReAct loop) reaching 94.5% of a RAG pipeline's faithfulness score with no vector store at all.

Sounds like embeddings are dead. Except Cursor published the counter-evidence from their own product: adding a custom-trained embedding model to their grep-using agent improved eval accuracy by 12.5% on average, and users on large codebases kept 2.6% more of the generated code. Their conclusion was that the combination of grep and semantic search wins, and their agent uses both.

And notice where all the grep-won evidence comes from: code, plus one small conversational-memory benchmark. Code is the friendliest possible territory for grep. Function names are exact and stable, the repo is a bounded search space, and an agent can search again if the first attempt misses. Nobody has shown the same result on a messy pile of enterprise PDFs where the same concept appears under five different names. The academic paper behind a lot of this discourse is even titled as a question, "Is Grep All You Need?", and its actual finding is that the agent harness matters more than the retrieval method. That nuance did not survive contact with LinkedIn.

Where this lands: if an agent is working over code or anything with exact, stable identifiers, let it search with tools first and add embeddings only if you can measure the gain. For fuzzy human language across large document sets, similarity search is still doing real work.

Debate 2: RAG vs. long context

Context window is the amount of text a model can take in at once, and it has grown from a few thousand tokens to a million-plus. The argument here is that this makes retrieval pointless. Why build a search pipeline when you can paste every document you have into the prompt?

Two pieces of evidence killed the strong version of this. Chroma's context rot report tested 18 frontier models and found that performance degrades as input grows, on every model, even on trivial tasks, and the degradation starts long before you hit the advertised limit. The models do not read your million tokens the way you imagine. The second piece is cost. Elastic ran the comparison on their own docs corpus: about $0.00008 per RAG query versus about $0.10 per full-context query, with 44-second latencies for the stuffed version. That is where the viral "RAG is 1,250x cheaper" number comes from.

Now the honest caveats, because that number gets abused too. It was one corpus of about a million tokens, one cheap model, and no prompt caching. The ratio is basically "your whole corpus vs. a few relevant chunks", so it grows with corpus size and shrinks a lot if you cache the prompt. It is a real measurement, not a law of physics.

Where this lands: long context is a genuine gift for how big your retrieved chunks can be and how much conversation history you can keep. You no longer have to be stingy about what you paste in, and that quietly fixes a lot of "the answer got cut off" problems. And there is one case where the long-context crowd simply wins, and I want to say it plainly because it cuts against my own thesis: if your whole corpus is a few hundred thousand tokens and rarely changes, pasting all of it with prompt caching is a legitimate architecture, not a hack. That is a lot of internal-docs use cases, and it is the simplest possible system, which is exactly the kind of boring I am arguing for. (Debate 4 gives that pattern its formal name.) For everything bigger or faster-moving, selection is still the whole game, which is why Anthropic and others now talk about context engineering, treating context as a budget you spend, with retrieval as one way to spend it well.

Debate 3: RAG vs. GraphRAG

This is the debate where junior engineers get lost fastest, so let me explain what GraphRAG actually is before we argue about it.

Standard RAG treats your documents as a pile of independent text chunks. GraphRAG adds a step: at indexing time, an LLM reads every document and extracts the entities in it (people, products, vendors, projects) and the relationships between them ("Acme supplies part X", "Priya approved contract Y"). The result is a knowledge graph, a web of facts you can walk across. The problem it solves is the multi-hop question: "who approved the vendor that supplied the part that failed" needs three facts from three different documents joined together. Plain similarity search struggles here, because no single chunk contains the whole chain. A graph can walk it hop by hop.

So why not always use it? For a while the honest answer was price. GraphRAG's indexing used to be absurdly expensive: one practitioner writeup puts LLM-based graph extraction on a 5 GB legal corpus at around $33,000 in early 2024, falling to roughly $33 by mid-2025 with LazyGraphRAG-style approaches. That is a single writeup, so treat the exact figures as one data point rather than a benchmark, but the direction matches what falling token prices and lazier extraction strategies would predict: the build cost is no longer the thing that should stop you.

What should stop you is everything that comes after the build. You are now maintaining entity resolution (deciding that "IBM", "I.B.M." and "International Business Machines" are the same node), an ontology (the schema of what kinds of entities and relationships you track), and a graph that drifts out of date as documents change, which means re-running extraction and reconciling the results, forever. A plain vector index of the same corpus carries none of that ongoing burden. A knowledge graph is a second database with its own maintenance bill, and the bill is paid in engineering time, not API credits. Demand evidence that you need it before you sign up. If your query logs are mostly "what is our refund policy", a graph is an expensive way to feel sophisticated.

The advice I found most sensible in this whole genre, buried in an article whose own headline contradicts it, was this: log your real queries for two or three weeks and classify them before deciding. If multi-hop questions are rare, you do not have a GraphRAG problem.

Debate 4: RAG vs. CAG

Cache-augmented generation is the one genuinely new architecture in the pile, and also the most oversold. Here is the idea in plain terms. When a model processes your prompt, it builds up an internal working state called the KV cache. Normally that state is thrown away after each request. CAG says: load your entire document collection into the model's context once, save that internal state, and answer every future question against the saved state. No search step at all, ever. With provider prompt caching, the economics can actually work, because you pay the big processing cost once instead of on every query.

The fine print is right there in the original paper, which is a short paper evaluated on small QA corpora: the entire knowledge source has to fit in the context window, and context rot (see Debate 2) applies well before the window is full. So CAG is a real option for a bounded, static corpus. A product manual. A policy handbook. A single large contract. Calling it a RAG replacement is a category error, because the moment your data outgrows the window or changes daily, you are back to selecting what goes in, which means you are back to retrieval.

Debate 5: RAG vs. memory

The newest rebranding. Agent memory products store what your agent learned in past sessions and pull the relevant bits back into context on later runs, so the agent "remembers" your preferences or past decisions. Strip away the framing and this is retrieval over your own past interactions, with writes.

The write side is where the genuinely new engineering lives, and it is worth spelling out because it is a different problem from anything in classic RAG. A memory system has to decide, at write time, what is worth keeping at all ("user prefers TypeScript" yes, the full transcript of a debugging session no), what to compress (a 40-message conversation becomes a two-line summary, and the summarizer's judgment about what mattered is now baked in forever), and what to expire (the user's preference from January that they reversed in March needs to lose to the March one, not tie with it). Get those three wrong and no retrieval quality saves you, because the store itself is full of noise, stale facts, or summaries that dropped the detail you needed.

But notice what happens on the read side. When the agent needs a memory back, it searches a store of text by relevance to the current situation. That is retrieval, the same problem, the same techniques, and you can verify this from the vendors' own marketing: the benchmarks memory products publish are recall and precision numbers, which are retrieval benchmarks. Retrieval did not die here either. It moved into a new layer, picked up a hard write-side problem, and got a friendlier name.

The part the narrative oversells: chunking

Chunking is the opposite problem from the five debates above. Those get argued endlessly; chunking gets no debate at all, just a unanimous verdict that fixed-size chunking is mistake number one and you must fix it immediately. When something is that oversold, it deserves its own section, without the panic.

Chunking is deciding how to split your documents into pieces before indexing them. You cannot embed a 200-page PDF as one unit, so you cut it up, and the cut points matter: split a table in half and neither half answers the question about the table. The debate is between fixed-size chunking (cut every N tokens, with some overlap so sentences at the boundary appear in both pieces) and fancier schemes: semantic chunking (cut where the topic shifts), structure-aware chunking (cut at headings and sections), and more exotic recursive setups.

But before any of that, an unglamorous truth the chunking discourse steps right over: chunking operates on whatever your parser produced, and parsing is where enterprise RAG actually bleeds. Real corpora are PDFs with tables that extract as word salad, scanned contracts, slide decks, SharePoint exports. If the parser mangled the table, no chunking strategy on earth recovers it, and I suspect a good share of "fixed-size chunking ruined my retrieval" stories are really parsing failures wearing a chunking costume. Budget real time for looking at what your extraction actually produces, on your ugliest documents, before you spend a minute debating cut points.

Here is what the shouting obscures. For most corpora, fixed-size chunks of a few hundred tokens with overlap are a fine starting point, and one structure-aware upgrade, splitting at headings when your documents have headings, gets you most of the remaining benefit for almost no work. The reason to start dumb is not laziness. You cannot know whether chunking is your actual problem until you measure retrieval on your own queries, and teams routinely spend a week on semantic chunking when their real failure was vocabulary mismatch, which no chunking strategy fixes. Modern long context also softens the whole issue: when you can afford to retrieve a small chunk and then feed the model the entire surrounding section, the cost of a slightly wrong cut point drops a lot.

Start fixed-size, respect document structure where it obviously exists, and let your eval numbers tell you if you need more. Usually you do not.

And its quieter sibling: indexing

Indexing is how the system stores your chunks so they can be found later, and it gets a fraction of chunking's airtime while causing just as many real failures.

You have three basic options. A keyword index (the classic is called BM25, the same family of tech behind old-school search engines) finds exact words: product codes, error messages, names. A vector index finds meaning: paraphrases, synonyms, "the customer said it differently than the doc". And hybrid search runs both and merges the two result lists. The tutorials push you straight to vector-only, and that is where systems quietly fail, because embeddings are bad at exactly the things keywords are good at: a user searching for error "E4023" or product "ZX-300" wants an exact match, and similarity search will happily return something merely similar.

For most systems, hybrid search is the right default, not an upgrade. Both index types are commodity infrastructure now, most vector databases and even Postgres can do both, and merging the results is a few lines of code. The one addition that earns its keep early is metadata: tag chunks with product, version and date at ingestion, so you can filter at query time. If users ask about v3 and retrieval keeps surfacing v2 docs, no embedding model fixes that. A filter does.

What you can skip until proven otherwise: fine-tuned embedding models, multiple specialized indexes, and anything that requires re-architecting ingestion. Boring hybrid plus metadata covers the overwhelming majority of corpora.

Reranking, the upgrade everyone calls non-negotiable

One more technique shows up in every fix list, usually with the phrase "non-negotiable", so it deserves the same treatment: what it is, what it fixes, and whether you need it.

Your search index is built for speed. It compares the query against millions of chunks in milliseconds, and that speed comes from a shortcut: the chunks were converted to vectors ahead of time, without knowing what anyone would ask. A reranker removes the shortcut for a small set of finalists. You retrieve wide, say the top 20 or 50 candidates, then a second model reads the actual query and each actual candidate together and scores how well they really match. Slower and more expensive per document, which is why you only run it on the shortlist, but noticeably better at judging relevance.

The problem it solves is very specific: the right passage is being found, just not ranked high enough. If the answer usually sits somewhere in your top 20 but not in the top 5 you actually feed the model, a reranker fixes exactly that, and rerankers are available today as a one-line API call, so testing one is an afternoon.

But look at what that failure signature requires: you have to know where the right passage lands in your rankings, which means you need measurements before "non-negotiable" means anything. If your retrieval is missing the right passage entirely, reranking cannot help, because you cannot re-order your way to a document that was never retrieved. And if the right passage already lands in your top 5, a reranker adds latency and cost for nothing. Long context weakens the case further: when you can afford to feed the model 20 chunks instead of 5, precise ordering inside that set matters less. Try it when your numbers show the specific gap it fills. Skip the word "non-negotiable".

The one question underneath all five

Pull the five debates apart and you find they are all asking the same thing: given what my corpus looks like, what my queries look like, how fresh the data needs to be, and what a query is allowed to cost, what is the cheapest selection mechanism that survives my failure cases?

That is it. Grep, embeddings, graphs, full context, KV caches and memory layers are all just selection mechanisms with different price tags and different failure modes. Code with exact names favors grep. Fuzzy prose favors embeddings. Multi-hop questions favor structure. Small static corpora favor caching. Agents that can retry favor cheap tools in a loop, because the loop itself compensates for a weak first attempt. A human asking one question gets one shot, so that first retrieval has to be good.

One more thing the posts never mention, and if you are the person who approves budgets, this paragraph is for you. Agentic retrieval, where the model searches, reads the results, and searches again, often costs more per query than the one-shot pipeline it replaced. It buys robustness, and you pay for it in tokens on every single query. Whether that trade is good depends on your traffic and your margins, which is a business question, not an architecture one, and it belongs in the meeting where you decide this, not discovered on the first invoice.

I should be fair to the trend here, because it cuts against my one-shot framing. If you are building an agent product, and a growing share of new AI products are agent products, search-as-a-tool is not an upgrade you buy later. It is the natural starting architecture, and the coding assistants proved it works. The skepticism in this article is aimed at teams answering user questions over a document pile, where the one-shot pipeline is still the honest default. If your product is an agent, your default is different, and that is fine. The discipline is the same either way: measure what the loop actually buys you, in accuracy and in tokens.

What actually works for most of you

Time to wire it together. If your situation is normal, normal documents, normal questions, a small team, here is the shape that covers the large majority of retrieval systems:

• Log real queries first. Two or three weeks of actual questions, classified by hand into lookup, fuzzy, and multi-hop. Every architecture decision is secretly a bet on this distribution, so find out what it is.
• Build the boring baseline. Fixed-size chunks with overlap, hybrid search (keyword plus vector), metadata filters, top results into a generously sized prompt. Every framework does this in an hour on clean text. The honest budget for a real corpus is longer, and most of the extra time goes into parsing your actual documents properly, which is the least discussed and most consequential part of the pipeline.
• Measure before you upgrade. A set of fifty real questions with known answers, scored by hand, beats any framework and any influencer. Change one thing, rerun, keep it only if the number moves.
• Spend long context on generosity, not replacement. Bigger retrieved sections and more conversation history, yes. Pasting the whole corpus, no.

And here are the specific signals that you are in the minority that genuinely needs more, which is exactly what Part 2 digs into:

• Your query logs show real multi-hop demand, questions that join facts across documents. That is when GraphRAG stops being overengineering and starts earning its complexity.
• Your corpus is small, static, and fits in a context window. That is CAG territory, and it can delete your whole retrieval pipeline.
• You are running an agent loop anyway, over code or tool-friendly data. Then search-as-a-tool with retries may beat any pipeline, if the per-query token cost works for your traffic.
• Your users' vocabulary and your documents' vocabulary barely overlap, or your eval numbers stay bad after the cheap fixes. That is when rerankers, query rewriting and the rest of the upgrade shelf are worth actual evaluation.

One caveat on my own evidence before I close, because I have spent this whole article demanding it from others. Every number here is someone else's measurement: Cursor's, Elastic's, Chroma's, one practitioner's GraphRAG bill. I checked the sources and flagged the weak ones, but I have not handed you a before-and-after from a system of mine, which is the same gap I am criticizing. The real answer to that gap is not a better article. It is the fifty-question eval in Part 2, run on your system, which outranks every number in this piece, including mine.

And to be clear, the point was never that all RAG writing is junk. There is genuinely good work out there, usually recognizable by the same tells: it publishes its evals, names its corpus, and admits where its conclusions stop applying. The genre this series pushes back on is the other kind. So next time a post tells you RAG is dead, check which of the five debates it is actually about, then check which domain its evidence comes from, and then check whether your corpus and your queries look anything like that domain. Usually the answer is no, and you can close the tab and get back to work.

In Part 2, I lay out the boring playbook in full: the baseline to build, the fifty eval questions that separate you from the influencers, the order in which upgrades are worth buying, when the advanced tier (graphs, CAG, agentic search) is actually worth it, and, most importantly, when to stop. It covers what 90% of retrieval systems actually need, which is a lot less than your feed suggests.