Retrieval-augmented generation gives a language model access to documents it was never trained on. You index a corpus, search it at query time, and place the results into the prompt so the model answers from your data instead of its parametric memory. That is the whole idea. The difficulty is not the idea. It is that a RAG system is a pipeline of several stages, any one of which can fail while the others look fine, and the failure usually shows up only as a bad answer with no error attached.
This is a reference for working on such systems. It is organised around a single question: given a bad answer, which stage broke, how would you confirm it, and what is the smallest intervention that measurement supports? It draws on Jason Liu’s RAG writing at jxnl.co (the RAG Master Series, the Context Engineering series, and the Coding Agents Speaker Series) and on the practitioners he interviewed: Skylar Payne, Colin Flaherty, Nik Pash, Walden Yan, and Beyang Liu. Where a claim is one person’s experience rather than an established principle, it is labelled as such. Where the sources disagree, the disagreement is kept rather than smoothed over.
A note on attribution. The experiments, client numbers and strong opinions below belong to the people named. They are not the publisher’s. Treat every specific threshold as a data point from one context, not a standard to copy.
The one law
The single principle that organises everything else: search quality is the ceiling on RAG quality. If the right passage never reaches the prompt, no prompt engineering, reranker, or model upgrade will recover it. Jason Liu states this plainly across his series, and Skylar Payne, who has debugged RAG systems at Google and LinkedIn and across many client engagements, frames the same point as the most common failure he sees: teams spend weeks tuning generation when the relevant information was simply never retrieved.
This law reorganises how you should spend effort. Before touching the model, confirm that retrieval is surfacing the right content. Before adding a reranker, confirm retrieval is missing things a reranker could fix. The most expensive mistake in RAG is optimising a stage downstream of the one that is actually broken.
The second thing worth internalising is that RAG failures are silent. A traditional service throws an exception when it breaks. A RAG system returns a fluent, confident, wrong answer. There is no stack trace. This is why evaluation and observability are not optional polish; they are the only way the system can tell you it is failing.
The diagnostic frame
A RAG pipeline has five stages. Most descriptions list them as architecture. It is more useful to read them as the five places a bad answer can originate, each with its own signature.
Ingestion and indexing. Documents are parsed, chunked, enriched with metadata, embedded, and stored. Failures here are quiet and cumulative: documents silently dropped by a parser, chunks too small to carry meaning, stale content never refreshed, permissions never recorded. The symptom appears later as “the system does not know about X,” and by then the cause is several stages back.
Retrieval. The query is turned into candidate passages. Failures here look like the right document existing in the corpus but never being returned, or irrelevant documents crowding out relevant ones. This is where the form mismatch between questions and documents bites, and where hybrid search and query transformation earn their place.
Ranking and context construction. Candidates are reranked, filtered, and assembled into the context block. Failures here look like the right passage being retrieved but ranked eighth, or the context being filled with near-duplicate fragments that push useful content out.
Generation. The model reads the context and writes an answer. Failures here look like the right passages sitting in the prompt but the model ignoring them, contradicting them, or inventing details they do not contain. This is the stage everyone suspects first and the one that is actually broken least often.
Evaluation and feedback. The loop that tells you which of the above failed. Absent this, you are guessing, and the guessing is expensive because the other four stages are all plausible suspects for any given bad answer.
The practical discipline is to localise the failure before intervening. Skylar Payne’s recurring advice, quoted across his interview, is to start from the user, work backwards, and look at the data at every step. Concretely: pull the actual query, pull the actual retrieved chunks, and look at them. If the right chunk is not in the retrieved set, the problem is retrieval or indexing, and no amount of prompt work will help. If the right chunk is there but the answer is wrong, the problem is generation or context construction. That single inspection resolves most “the AI is broken” reports into a stage, and the stage tells you which lever to pull.
Ingestion: where most failures are buried
Chunking
Chunking splits documents into the units you embed and retrieve. The default in many tutorials is a small fixed size, often around 200 characters, a habit carried over from early models with tiny context windows. Skylar Payne singles this out as a recurring anti-pattern. In one e-commerce project he describes, product spec sheets were split into fragments so small that no single chunk contained a complete answer, and the model hallucinated to fill the gaps. His broader point is that modern models tolerate much larger chunks, and that the right size depends on the corpus and the model, not on a number copied from a guide.
What is established rather than opinion: a chunk should be a unit that can answer a question on its own. If your chunks routinely need three neighbours to make sense, they are too small. If a chunk covers four unrelated topics, it is too large and will dilute retrieval signal. The honest answer is that chunk size is a thing to measure on your data, not a setting to inherit. Skylar’s recommendation is to look at your actual chunks and check whether they are self-contained, and to filter out low-value content such as copyright footers and boilerplate, which add noise that can be retrieved and crowd out substance.
Metadata
Metadata is the structured information attached to a chunk: source document, date, author, section, document type, access scope. It matters for two reasons. It lets you filter retrieval precisely (only documents from this year, only this customer’s data), and it lets you answer questions that pure similarity cannot, such as “what is the latest policy on X?” Neither BM25 nor embeddings can answer a temporal question without an explicit date to filter on. Jason Liu makes this a recurring theme: “what is the latest X” is unanswerable by text or semantic search alone, and metadata extraction plus query understanding is what makes it work.
But metadata is not free, and it is not always worth it. Skylar Payne offers a useful corrective: in his client work, roughly four in ten implementations had indexes so small that extensive tagging bought little. Metadata pays off when you have both a large corpus and a diverse set of query types. A small, homogeneous corpus may not need it. This is a context-specific judgement, not a rule, and it is worth measuring rather than assuming.
Freshness and deletion
Two operational concerns that ingestion must handle, and that most demos ignore.
Freshness. If your corpus changes, your index must change with it, and you must be able to detect when it has not. Skylar describes a financial summarisation system whose index had not been refreshed for two weeks, so it returned stale earnings data when users asked for the “latest” information. For time-sensitive corpora, monitor index freshness explicitly and consider filtering out documents past a certain age. The failure mode is not a crash; it is a confidently wrong answer built on data that was correct a month ago.
Deletion. When a source document is removed, edited, or access-revoked, the corresponding chunks must be removed too, or the system will keep surfacing content that should not exist. This is both a correctness problem and a compliance one. The clean design is to treat the source of truth as authoritative and rebuild or incrementally sync the index from it, with document-level identifiers so a deletion propagates. An index that can only grow is a liability.
Permissions
Access control must be decided at ingestion, not bolted on at query time. If a chunk does not carry its access scope, you cannot filter by it later without re-deriving permissions from scratch, and you will eventually leak one user’s data into another’s answer. Skylar’s framing of metadata tagging includes this: for B2B systems where customer data is segregated, the tagging may look like overhead, but it is what keeps tenants apart. Record the access scope on every chunk, and enforce it as a retrieval filter, not as a post-hoc check on the generated text. A post-hoc check is too late; the content has already entered the model’s context.
Retrieval: hybrid, reranking, and query transformation
Hybrid search
Hybrid search combines keyword matching (typically BM25) with semantic search (embeddings). The keyword component catches exact terms, product codes, names, and rare tokens that embeddings can blur; the semantic component catches paraphrase and synonymy that keywords miss. Jason Liu recommends hybrid consistently across his series, and Skylar Payne treats the absence of full-text search as one of the common gaps to audit for.
This is close to established practice, but it is worth stating the evidence honestly. Jason’s claim that hybrid outperforms either alone “in every real-world test I have seen” is a practitioner’s strong consensus, not a published benchmark. His own measurements, reported in his systematic-improvement piece, show the picture is corpus-dependent: on a corpus of essays, full-text and embeddings performed about the same, except full-text was roughly ten times faster; on a corpus of repository issues, full-text reached about 55% recall and embeddings about 65%. The lesson is not “always hybrid.” It is that the relative value of each component depends on your corpus, and you should measure both on your data before deciding. For some corpora, plain keyword search is nearly as good and much cheaper.
Reranking
A reranker is a cross-encoder model that re-scores the candidate set after initial retrieval, before the context is built. Initial retrieval is fast and broad; the reranker is slower and more accurate, and it catches cases where embedding similarity ranked irrelevant content highly. The common pattern is to retrieve a larger candidate set (say, the top 50) and rerank down to the handful you actually use.
Reranking is a good default when retrieval recall is decent but precision is poor, that is, when the right passages are being retrieved but buried below noise. It is less useful when the right passages are not being retrieved at all, because a reranker cannot promote what was never fetched. It adds latency and cost, so it should be justified by measurement, not added reflexively. Cohere’s reranker is a frequent choice in Jason’s writing, but the tool is less important than the decision: measure whether reranking moves your retrieval metrics on your queries before committing to it.
Query transformation
Query transformation uses an LLM call to rewrite or expand the user’s question before retrieval. It extracts entities and dates, resolves vague references, and can expand a thin query into several. It is what lets the system handle “what is the latest X” by pulling out the date constraint and turning it into a filter.
The trade-off is latency. Jason puts the cost at roughly 500 to 700 milliseconds for the extra model call. That is an order-of-magnitude figure and depends on the model, but the point stands: query transformation buys retrieval quality at the price of a round trip. It is worth it when users ask vague or temporal questions that raw retrieval mishandles. It is not worth it when queries are already specific and retrieval is performing. As with everything here, the condition matters more than the technique.
A related idea from the context-engineering series is that retrieval should not be one-shot. In agentic systems the model can call search repeatedly, refine, and explore. That shifts query transformation from a single preprocessing step into an ongoing loop the model drives itself. More on that below.
Context construction and the shift to agents
Assembling the context
Once you have a ranked set of passages, you build the context block the model will read. The naive version concatenates the top-k chunks. Better versions deduplicate overlapping fragments, order passages sensibly, and include enough source information for the model to cite. Skylar Payne recommends forcing the model to produce inline citations, then validating that each citation actually exists in the retrieved set and actually supports the claim it is attached to. In sensitive domains such as healthcare, this verification is not optional; a hallucinated citation in a medical answer is a serious failure.
The broader principle, which Jason Liu emphasises in his context-engineering writing, is that the structure of what you hand the model is itself a form of prompt engineering. Wrapping results in clear structure, attaching source metadata, and including short instructions about how to use the results all shape the model’s behaviour. This is cheap to do and often the highest-leverage change available, because it requires reformatting strings rather than rebuilding infrastructure.
From chunks to landscapes
Jason Liu’s context-engineering series argues that the chunk-based model of RAG is incomplete for agentic systems. A one-shot RAG pipeline precomputes which chunks to inject and asks the model to reason over them. An agent, by contrast, makes many tool calls and builds understanding over time. It does not just need the right chunk; it needs some sense of the information landscape so it can decide what to explore next.
His proposed progression has four levels. Level one returns bare chunks. Level two adds source metadata, which enables citation and lets the agent load a whole document when fragments from it keep appearing. Level three handles multimodal content such as tables and images properly. Level four adds facets: metadata aggregations returned alongside results, such as counts by category or by date, that reveal the shape of the result set. A concrete observation from his consulting is that a simple function to load whole pages, rather than returning disconnected fragments, noticeably improved agent reasoning.
Facets deserve a moment because they capture a real insight. Jason describes a support-ticket case where the best-documented, most-retrieved tickets were the resolved ones, so the system kept surfacing old answers and missing the emerging issue. Similarity search had a bias toward whatever was best written up. Facets, showing the distribution of results by status and date, made the gap visible in a way that top-k results hid. The general lesson: returning only the highest-scoring chunks can hide systematic structure in the data, and a little aggregated metadata can expose it.
He reports large improvements from this approach: in his consulting, roughly 90% fewer clarification requests, 75% fewer escalations, 95% fewer timeouts, and about four times faster resolution. These are his numbers from specific engagements, not independently verified benchmarks, and should be read as evidence that context structure matters, not as results to expect.
Subagents and context hygiene
When an agent does messy, token-heavy work, that work can pollute the main thread’s context. Jason’s context-engineering series measures this directly on a coding task: a slash command that dumped raw output left the main thread about 91% noise, while routing the same work through a subagent that returned only a summary left the main thread roughly 76% useful signal, which he describes as about eight times cleaner. The numbers are from one task and should not be generalised, but the principle is sound: keep heavy reads isolated, and return distilled summaries to the thread that has to keep reasoning.
This connects to the next section, because the strongest case for restraint in agent architecture comes from the coding-agent teams, and it cuts against the instinct to add machinery.
Evaluation: the only way to know
Evaluation is the stage that tells you which other stage failed. Without it, every bad answer is a mystery with four plausible suspects, and teams respond by changing things at random and hoping. With it, you can localise the failure and confirm the fix.
The conditional matrix
Jason Liu’s evaluation framework organises metrics around three objects: the question (Q), the retrieved context (C), and the answer (A). Each metric is a conditional relationship between two of them, and the set of relationships tells you where to look.
The first tier needs no model and is the fastest signal. Retrieval precision and recall measure whether the right passages were fetched for the question. Because these are cheap and fast, they are the leading indicator: if retrieval is broken, fix it before spending anything on generation. Jason’s rule of thumb, from his consulting, is that a retrieval system should reach about 97% recall on synthetic questions generated from its own chunks; if it cannot retrieve the chunk a question was generated from, something fundamental is wrong. That 97% is his target, not an industry standard, but the logic is solid: synthetic questions give you a baseline you can compute before you have any real users.
The second tier uses a model as judge and covers the relationships that matter for answer quality. Faithfulness (A given C) asks whether the answer is actually supported by the retrieved context, or whether the model invented details. Answer relevance (A given Q) asks whether the answer addresses the question. Context relevance (C given Q) asks whether the retrieved passages are on-topic. These three catch the common failure modes: hallucination, irrelevance, and noisy retrieval.
The third tier covers subtler relationships: whether the context supports the answer, whether the question is answerable from the corpus at all, and whether the question is self-contained. Jason’s practical guidance is to weight these by domain. A medical system should care most about faithfulness, because an unsupported claim is dangerous. A support system should care most about answer relevance. A documentation system should care about answerability, because an unanswerable question should produce an honest “I don’t know” rather than a guess.
This matrix is one coherent framework, not the only one. Its value is that it forces you to name which relationship you are measuring, instead of reaching for a vague “accuracy” score that hides where the problem is.
Evaluate before you add complexity
The single most important evaluation habit comes from Skylar Payne. He reports that, when properly evaluated, more than 90% of the complex additions teams made to their RAG systems performed worse than the simpler baseline. The additions were not tested against a baseline; they were added on intuition and assumed to help.
The lesson is not that complexity is bad. It is that complexity must be justified by measurement, because the default assumption that a fancier pipeline is better is usually wrong. Before you add a reranker, a second index, a query rewriter, or a fine-tuned embedding, run your evaluation, make the change, and run it again. If the metric did not move, the change did not help, whatever the architecture diagram suggests. This is the cheapest and most ignored discipline in RAG.
Skylar also warns against evaluating only what is easy to measure. If you only check whether retrieved documents are relevant, you are, in his phrase, looking under the lamppost: measuring what is convenient rather than what matters. The end-to-end answer quality is what the user experiences, and it needs to be measured directly, ideally with the citation verification described above for anything sensitive.
Observability in production
Evaluation tells you how the system performs on a test set. Observability tells you how it performs on real users, whose queries are stranger and more varied than anything you generated.
Jason Liu’s monitoring piece, built on conversations with the teams behind Raindrop and Oleve, makes the central point: LLM failures do not raise exceptions. A judge model can flag known failure modes, but it cannot flag the ones you have not imagined. So you need to look at real interactions, especially early.
His practical default for small systems is blunt and useful: if you are below roughly 500 interactions a day, pipe every one of them to a channel and read them. The threshold is a heuristic, not a law, but the reasoning is sound. At low volume, manual review is cheap and catches things no automated metric would. Teams skip this because it does not scale, and then build elaborate monitoring to rediscover what reading a hundred conversations would have shown them.
For larger volume, the Oleve team’s approach, which Jason calls Trellis, is worth summarising. It has three moves. First, discretise: bucket interactions into issue types so you can count them. Second, prioritise: do not fix the most common issue first; fix the one with the best combination of volume, negative sentiment, achievable improvement, and strategic relevance. Volume alone is misleading, because the most frequent complaint may be unfixable or unimportant. Third, refine recursively: fix the top issue, re-measure, and repeat. The fix portfolio he recommends runs from cheapest to most expensive: adjust the prompt, offload work to a tool, adjust retrieval, and only then consider fine-tuning. Most problems are resolved in the first two.
The through-line from evaluation to observability is the same: you cannot improve what you do not measure, and the measurement must reflect what users actually experience, not what is convenient to compute.
Operations: latency, cost, security
These cut across every stage, so they are easy to neglect until they bite.
Latency. Every added model call adds a round trip. Query transformation, reranking, and agentic multi-step retrieval all trade latency for quality. Jason’s framing is that the trade should be explicit and domain-dependent: a medical diagnostic may justify a large latency increase for a small recall gain, while a consumer search product may not, because slow answers cause churn. Measure your latency budget and decide each addition against it, rather than stacking techniques and discovering the system is too slow to use.
Cost. Retrieval is cheap relative to generation, but agentic systems change the arithmetic by making many calls. The coding-agent teams, discussed below, partly abandoned embedding RAG for cost and complexity reasons, and Nik Pash notes that RAG can still be worth it for code when the business model demands minimal token spend, such as a low-priced subscription that cannot afford to load whole files into context. Cost is a legitimate reason to choose a simpler or more expensive design; it should be a conscious decision, not an afterthought.
Security. Two concerns. First, access control, covered above: permissions must be enforced at retrieval, not after generation. Second, the index itself. Nik Pash raises the concern that embeddings can be reverse-engineered to recover source content, so indexing a sensitive codebase creates a copy of it in a form that may be harder to protect. Whether that risk is material depends on your threat model, but it is a reason to think about what you are copying and who can reach it, rather than indexing everything by default.
When RAG is the wrong tool
This is the part of the corpus that disagrees with itself, and the disagreement is the point.
Jason Liu’s RAG series teaches you to build and optimise RAG. His coding-agent series, drawing on Colin Flaherty (Augment), Nik Pash (Cline), Walden Yan (Cognition), and Beyang Liu (Sourcegraph), contains a strong counter-argument: for code, several of the most successful agent teams have abandoned embedding-based RAG in favour of direct exploration. The split is real and should not be smoothed away.
The case against RAG for code
Nik Pash’s position is the sharpest. Having built RAG tooling for years, he became, in his words, actively anti-RAG for coding agents. The argument has several parts. Code is structured and logical, so it is better explored than searched: an agent that reads a directory, opens a file, follows an import, and builds understanding step by step mirrors how a senior engineer learns a codebase, and maintains a coherent narrative that disconnected retrieved fragments destroy. Embedding search hands the agent what he calls a scattered mind-map of unrelated snippets, which distracts more than it helps. Indexing a codebase is a security exposure. And maintaining an embedding pipeline is a resource sink that yields diminishing returns. Colin Flaherty’s experience at Augment points the same way: simple grep and file tools driven by an agentic loop beat embedding-based retrieval on their benchmark work. Boris, of Cloud Code, reportedly tried RAG and abandoned it, as Nik recounts from a podcast appearance.
The honest version of this claim is domain-bounded. It applies to code, which is structured, cross-referenced, and navigable by following explicit links. It does not obviously transfer to prose, policy documents, support tickets, or research papers, where there is no import graph to follow and similarity search genuinely helps. Jason’s own series is largely about those prose-and-knowledge cases, and the two positions coexist in his writing without contradiction once you draw the boundary. The disagreement is less “RAG works” versus “RAG is dead” than “RAG suits unstructured knowledge” versus “direct exploration suits structured code.” If you are building over code, take the anti-RAG case seriously and measure it. If you are building over documents, the classic RAG toolkit still applies.
Even within code, Nik concedes exceptions: RAG can be worth it when token cost dominates, or for very large monorepos where loading whole files is impractical. The retreat from RAG is not absolute.
Multi-agent systems
A second area of hard-won scepticism. Walden Yan’s argument, from building Devin at Cognition, is that multi-agent systems fail through context loss. Splitting a task across agents is a game of telephone: each agent knows only what it was told, and details drop at every handoff. Parallel agents are worse, because they make implicit decisions independently and then conflict, producing incompatible components that have to be reconciled. Cognition’s answer is to favour a single agent with careful context management, and where context overflows, to compress it; they trained a model specifically to preserve the important information from earlier in a long session.
The practical guidance is that if you reach for multiple agents, make them sequential rather than parallel, pass the full context forward, and treat the architecture as a last resort. The default suspicion toward multi-agent designs is one of the most consistent findings across the coding-agent interviews, and it generalises beyond code: coordination cost is real, and a single capable agent with good context is often more reliable than several specialised ones that cannot see each other’s work.
Model coupling
A third shift. Classic RAG advice assumes the model is swappable: build a clean pipeline, and the model is a component you can upgrade. The coding-agent teams push back. Beyang Liu, Nik Pash, and Colin Flaherty all describe systems where the tools and the model are tuned together, so that swapping the model breaks tool behaviour that was carefully calibrated. Different models may be used for different phases, planning versus execution, for instance. The recommendation that follows is that if you are building agents, designing for model agnosticism can work against you, because the value is in the coupling. This is a genuine departure from the orthodoxy, and it is worth weighing even if you disagree: the more agentic your system, the more the model choice is part of the design rather than a detail beneath it.
Alternatives and adjacent patterns
Not everything that looks like RAG needs it. Skylar Payne’s routing point is worth repeating: a question like “what is my billing date?” is a metadata lookup, not a retrieval-and-generation problem, and running it through a full RAG pipeline adds latency, cost, and hallucination risk for no benefit. Classify intent and route simple lookups to direct queries. Reserve RAG for questions that genuinely need synthesis over retrieved text.
Jason’s forward-looking pieces add two ideas. One is that the same retrieval machinery can be reframed from question-answering into report generation: instead of answering one question, the system assembles a structured report from a template or standard operating procedure. He argues the report framing carries more perceived value than a chat answer, and that the templates themselves become the asset. This is opinion and framing rather than established practice, but it points at a real direction: RAG as infrastructure for producing documents, not just answering questions.
The other is the data flywheel: a system that improves itself by collecting real interactions, generating synthetic test data, running fast evaluations, and feeding the results back into retrieval and indexing. The leading metric he recommends is not system quality but experiments per week, on the theory that the teams who iterate fastest win. The flywheel is the long-term answer to the question this article keeps returning to: how do you know which stage is failing? You build the loop that tells you, continuously, on real data.
A closing checklist
Use this when a RAG system misbehaves, in roughly this order.
First, localise the failure. Pull the real query and the real retrieved chunks. If the right content is not retrieved, the problem is retrieval or indexing. If it is retrieved but the answer is wrong, the problem is context or generation. Do not touch the model until you know which.
Second, check ingestion. Are documents being parsed without silent drops? Are chunks self-contained? Is the index fresh? Are deletions propagating? Are permissions recorded on every chunk?
Third, check retrieval. Do you have both keyword and semantic search, measured on your corpus? Is a temporal or entity question being handled, or does it need query transformation? Is a reranker helping, measured, or just adding latency?
Fourth, check context. Are passages deduplicated and ordered? Are citations required and verified? If this is an agent, are tool responses structured, and is heavy work isolated in subagents?
Fifth, check evaluation. Do you have retrieval precision and recall on synthetic data? Do you measure faithfulness and answer relevance on real queries? Did you measure before and after the last change, or did you assume it helped?
Sixth, check operations. Is latency within budget? Is cost understood? Is access control enforced at retrieval?
And finally, check the premise. Is this actually a RAG problem, or a lookup that should be routed elsewhere? If the corpus is code, has direct exploration been tried and measured against retrieval? If the design is multi-agent, is that complexity earning its place, or would a single agent with good context do better?
The recurring lesson across all of this is restraint. The teams who get RAG right are not the ones with the most elaborate pipelines. They are the ones who measure first, change one thing, and confirm it helped, and who treat every added component as a cost to be justified rather than an improvement to be assumed. The pipeline that works is usually simpler than the one on the whiteboard, and the only reliable way to find it is to let the measurements decide.
Sources
Synthesised from Jason Liu’s writing at jxnl.co: the RAG Master Series, the Context Engineering series, and the Coding Agents Speaker Series (compiled July 2026), including interviews with Skylar Payne, Colin Flaherty (Augment), Nik Pash (Cline), Walden Yan (Cognition), and Beyang Liu (Sourcegraph), and the Oleve and Raindrop monitoring conversations. All claims, experiments, and opinions are attributed to their sources above. Starting source map: github.com/bilawalriaz/rag-playbook.