Skip to content

ai-rag — Roadmap & Reading Map

Practice and new theory live here (ai-rag/). Most of the supporting theory already exists elsewhere in this repo — vector index internals, OpenTelemetry, metrics storage, cost attribution, async Python, measurement methodology. This file is the bridge: what to build, in what order, and which existing document to read alongside each thing.

The rung-3 trap applies here too. Reading about rerankers produces confident-sounding knowledge that collapses under one follow-up question. Every chapter in this folder exists to unblock a project in §4. If you are reading something that isn't unblocking a build, stop and go build.


1. Scope

The surface layer of RAG — wire an embedding model to a vector store, stuff results into a prompt — is a two-week ramp and is not what this folder is about. The hard and durable part is everything underneath:

Surface layer What this folder covers
"It returns relevant answers" Golden sets, recall@k regression gates, faithfulness scoring
"It works" OTEL traces per retrieval / rerank / generation / tool call
"It costs something" Per-request, per-tenant, per-model token attribution
A framework call Hybrid retrieval, reranking, index tuning, latency budgets

The organising principle: an LLM pipeline is a data system. Correctness, measurability and unit cost are engineering properties of it, not afterthoughts — and they are where the difficulty actually lives.


2. What changed since 2023

RAG practice from 2023–24 is stale in specific, closable ways:

2023 practice 2026 practice Where it's covered
Eyeball the outputs Golden sets, recall@k / MRR / nDCG, LLM-as-judge with calibration, CI regression gates 08, 09
Fixed-size chunks Semantic, contextual, late chunking; parent-document retrieval 02
Cosine similarity only Hybrid BM25 + dense with RRF, cross-encoder reranking 04
Query goes straight to the index Rewriting, decomposition, multi-query, HyDE 05
Stuff everything in the prompt Context budgeting, compaction, citation, memory 06
Single-shot retrieve→generate Agentic multi-hop retrieval, tool calling 13, 14
No instrumentation OTEL GenAI semantic conventions, span per LLM call, trace↔eval linkage 10
Cost noticed on the invoice Per-request, per-tenant, per-model token accounting and budgets 11
"It's slow" Prompt caching, batching, model routing, streaming, fallback 12

Long context also changed the calculus — sometimes the answer is less RAG, and knowing when is part of the job (06).


3. Chapter plan

Written as they're needed, not up front. Each one is a build-log with theory attached, which keeps every claim honest by construction.

Task sheet: LABS.md turns every written chapter and appendices D–H into predict, build, break and measure tasks on a laptop.

Doc Topic Status
00-mental-models.md The retrieval→generation pipeline as a data system; where correctness actually lives written — labs unrun
01-embeddings-and-representation.md Model choice, dimensionality, domain adaptation, multilingual, drift written — labs unrun
02-chunking-and-document-processing.md Parsing, chunk strategies, contextual/late chunking, parent-doc written — labs unrun
03-indexing-and-vector-stores.md How HNSW/IVF/PQ work (with the math), recall and ef_search defined, quantization, filtered search, pgvector vs dedicated, interview questions, real-world cases written — labs unrun
04-retrieval-hybrid-and-reranking.md BM25 + dense, RRF fusion, cross-encoder rerankers, latency budget written — labs unrun
05-query-understanding.md Rewriting, decomposition, multi-query, HyDE, routing written — labs unrun
06-context-engineering.md Window budgeting, compaction, citation, memory, long-context tradeoffs written — labs unrun
07-generation-and-structured-output.md Structured outputs, schema validation, retries, determinism written — labs unrun
08-evaluation-methodology.md Golden sets, recall@k / MRR / nDCG, faithfulness, LLM-as-judge calibration, significance written — labs unrun
09-eval-infrastructure-and-ci.md Eval as a pipeline: datasets, versioning, regression gates, dashboards planned
10-llm-observability-and-tracing.md OTEL GenAI semconv, span design, trace↔eval linkage, sampling planned
11-token-accounting-and-cost.md Unit economics: cost per answer, session, tenant and feature; the attribution join (usage × trace × eval); showback vs chargeback; forecasting; commitment and batch-lane planning. Budget enforcement moved to appendix L (§2's envelope, §4's ceilings) planned, rescoped
12-serving-latency-and-caching.md The serving path: streaming, TTFT and inter-token budgets, concurrency and batching, timeouts and hedging, response and semantic cache placement, warm-up and cold start. Prompt-caching mechanics stay in 06 §12; the cost arithmetic and cache-TTL break-even are in appendix L §5; routing and fallback in 23 §4–§5 planned, rescoped
13-agents-and-tool-calling.md Tool schemas, multi-hop retrieval, planning, idempotency, failure handling planned
14-agent-evaluation.md Trajectory and environment-state grading, pass^k reliability, harness vs model, public benchmark map (τ²-bench, Terminal-Bench, SWE-bench, GAIA, BrowseComp, OSWorld, BFCL), online eval written
15-ingestion-pipelines-and-freshness.md Parsing at volume, dedup, incremental index updates, backfill, staleness SLOs planned
16-multi-tenancy-and-isolation.md Data isolation: per-tenant indexes vs one filtered index, noisy neighbours at the index and reranker layer, residency, per-tenant deletion and GDPR, per-tenant eval slices. Quotas, fairness and the two-ceiling problem are in 23 §6 and appendix L §4 planned, rescoped
17-safety-guardrails-and-prompt-injection.md Injection defence, output filtering, PII, tool-call authorization written — labs unrun
18-failure-modes-and-incident-walkthrough.md End-to-end: a retrieval regression found, diagnosed, fixed planned
19-build-vs-buy.md Langfuse / LangSmith / Braintrust / Arize / Helicone landscape and when to build planned
20-langchain-architecture-and-internals.md Runnable protocol, LCEL, retrievers, tools, callbacks, package split, lock-in criticism written
21-langgraph-deep-dive.md StateGraph, reducers, checkpointing, interrupts, streaming, subgraphs, LangGraph vs Temporal written
22-agent-orchestration-patterns.md Workflows vs agents, ReAct / Plan-and-Execute / Reflexion / LATS, multi-agent topologies, failure handling written
23-multi-llm-model-gateway.md Provider abstraction, routing, fallback, quotas, cost attribution, gateway caching written
24-tool-calling-and-enterprise-integration.md Tool protocol, registries, authz, idempotency, long-running tools, enterprise wrappers written
25-memory-and-state-management.md History, agent state, long-term memory, multi-tenant memory, GDPR deletion written
26-mcp-and-agent-protocols.md MCP in production: primitives, spec revisions through 2026-07-28, transports and scaling, OAuth authorization, registry and gateway written
27-a2a-and-agent-interoperability.md A2A (Agent Cards, task lifecycle, streaming, push), MCP vs A2A vs AGENTS.md, engineering a remote-agent boundary written
28-agentic-security-owasp-and-mcp-threats.md OWASP Top 10 for Agentic Applications (ASI01–10), MCP tool poisoning / rug-pull / confused deputy, memory poisoning, sandboxing written
29-agent-identity-and-delegated-authorization.md Agent as principal, token exchange and act chains, down-scoping, DPoP, CIBA, workload identity, PEP/PDP, audit written
30-context-engineering-for-agents.md Tool search, writing tools for agents, code execution over tools, sub-agents, Agent Skills, compaction, cache-aware loops written
31-mcp-tool-design-best-practices.md MCP tool design as an API style guide for agents: REST vs MCP, granularity, names and descriptions, field cookbook (e-mail, display names, BCP 47 languages, phone, dates, money, ids), output design, error taxonomy and messages, idempotency and confirmation, RBAC inside tools, REST wrapping, versioning, tool evals; lab labs/mcp-tool-design/ written — lab passes
appendix-a-glossary.md planned
appendix-b-metric-definitions.md Every eval and cost metric, with its exact formula planned
appendix-c-eval-recipe-book.md Copy-pasteable eval setups planned
appendix-d-doc-processing-benchmarks.md Parser, chunker, embedding, vector store benchmarks and best choices by use case written
appendix-e-deployment-and-compute.md Compose vs Kubernetes, model serving, GPU sizing and scheduling, pipeline orchestration, compute cost written
appendix-f-recall-at-every-layer.md Every "recall" in the pipeline (span survival, token, embedding, ANN, candidate, context, citation, abstention, judge) on one page: what each counts, its answer key, a worked example, and a per-stage leak report written
appendix-g-ranking-and-answer-metrics.md Precision, hit rate, MRR, MAP, nDCG variants, faithfulness vs correctness, RAGAS names, scores vs metrics, pp vs %, κ — one worked list, every metric computed, and a per-stage cheat sheet written
appendix-h-graphrag.md GraphRAG, LazyGraphRAG, LightRAG, HippoRAG, RAPTOR: the two query types top-k can't answer, index-cost arithmetic (~1,000× vs vector), what GraphRAG-Bench shows, ACL leakage through summaries, adoption order written
appendix-i-langchain-v1-in-practice.md LangChain 1.x in practice, read from source: what create_agent compiles into, middleware hook order and node-vs-wrapper semantics, the defaults that bite (recursion_limit=9_999, retry on_failure, summarization trigger=None), ToolStrategy vs ProviderStrategy failure modes, ToolRuntime context, best and worst practices, runnable probes written
appendix-j-production-ai-stack.md The production AI stack by layer: the usual senior-AI-engineer roadmap mapped to these chapters, the LangGraph package map with internals (checkpoint tables and quadratic growth, MCP adapter sessions, third-party look-alike packages), reference stacks for enterprise RAG, coding agents (edit formats, search, sandbox layers) and the agent backend (double texting, failure modes), eval/observability tools, and frameworks beyond LangChain written
appendix-k-langgraph-runtime-and-platform-in-practice.md LangGraph runtime and platform in practice, read from source and probed: the Functional API (@entrypoint/@task replay rules), durability modes under a hard kill, RetryPolicy defaults (no retry on TimeoutError), timeouts, CachePolicy, rate limiting, streaming v2, LangSmith (batching, head sampling, redaction, evaluate), Deep Agents backends and sub-agents, the store, MCP adapter interceptors and name collisions written
appendix-l-control-plane-guardrails-budgets-and-gateways.md The control plane as one system: the five chokepoints, the request envelope (absolute deadlines, one shared ledger, sub-agent inheritance), admission control and deadline-aware shedding, priority classes mapped to provider service tiers (priority, flex, batch), the 429 contract and the two ceilings, token optimization ordered by payoff with the cache-TTL break-even, guardrail placement and the 2026 managed products, tool governance as a budget control, AI vs MCP vs agentic gateways (Agent Router, agentgateway, ContextForge, Docker MCP Gateway, LiteLLM), and two reference configurations — three engineers and an enterprise written
appendix-m-how-the-labs-do-it.md How the labs and the AI products actually do it, read out of shipped systems: the five-rung ladder (window → cache → retrieve → compact → persist) and Claude Projects' hysteretic switch at the boundary; the four paths for a large document; caching compared as priced systems (explicit vs implicit breakpoints, write/read multipliers, TTLs, the break-even, and paged attention / RadixAttention underneath); clearing vs compaction vs memory now that both Anthropic and OpenAI ship all three server-side; prompt compression and the five reasons no provider ships it; product memory (ChatGPT dreaming, Claude entries, Gemini personal context) and the governance surface a B2B buyer asks for; agent-memory frameworks and the LoCoMo benchmark war; the labs' own RAG defaults and Cursor's Merkle-tree index; the inference gateway as a product surface and the two-tier model split; the Anthropic/Cognition orchestration disagreement reconciled as reads-vs-writes; and long context vs retrieval computed rather than argued written
appendix-n-multimodal-retrieval-and-generation.md Multimodal, input and output: the modality ratio that picks your architecture; visual document retrieval (ColPali-style patch grids and MaxSim, the storage arithmetic that kills prototypes, int8 and pooling and two-stage as the mitigations, and why single-vector multimodal is now the first thing to try); the modality gap, measured, and why rank fusion rather than score blending fixes it; tables as a retrieval problem — route aggregates to SQL, index the row and return the table, and why oracle-context table scores are generation scores; audio where the timecode is the citation, keyword recall instead of WER, and transcription-free retrieval's real use cases; video as two pipelines on one timeline; output that isn't text — native speech-to-speech vs the pipeline's inspectable middle, latency budgets, render-vs-rows, and C2PA's metadata-stripping hole; the cost arithmetic per modality; and the pixel-borne injection surface every text guardrail misses written
appendix-o-training-agents-rlvr-and-the-alternatives.md Training agents, and the five cheaper levers above it: the lever table ordered by cost and by whether the fix survives a model upgrade; RLVR and GRPO mechanically (critic-free, group-relative, and why a group with zero variance teaches nothing); the verifier as the actual product, with its structure and the final-state-not-transcript rule; reward hacking with the documented numbers (~43× when the grader is reachable, 99.4% acceptance at 2.2% correctness) and the detection harness; where RLVR stops working and the verify-at-inference move that cashes in the verifier without training; SFT vs DPO vs RLVR; trajectories, credit assignment and the environment problem; the break-even arithmetic with the re-training term people omit; and the four-arm baseline that ends most training projects written

Decision models (Jev, TypeSafe AI, 2026-09). These are models that return typed, calibrated probabilities instead of text. They are not a separate chapter. They are covered where they are useful: guardrail classifier tier and tool-call risk scoring in 17 §7.6, first judge tier in eval cascades in 08 §11.8, boundary detection and metadata tagging at ingest in 02 §6.5.2, and relevance filtering after the reranker in 04 §8.3.

Agentic protocol and runtime layer (2026-10). MCP (26), A2A and interop (27), agentic security per the OWASP agentic Top 10 (28), agent identity (29), context engineering for agent loops (30), MCP tool design best practices (31), and agent evaluation (14).

2026 additions. Defense by design against prompt injection (adaptive attacks, the Agents Rule of Two, the six agent design patterns, CaMeL-style capabilities) in 17 §4.7; agentic search vs a vector index in 05 §5.6; reasoning rerankers and BRIGHT in 04 §8.4; nugget-based answer evaluation in 08 §10.9; GraphRAG in appendix H; the LangChain 1.x practice layer (create_agent, middleware, structured output strategies) in appendix I; the whole production stack by layer, with reference stacks for enterprise RAG, coding agents and agent backends, in appendix J; the LangGraph runtime controls, Functional API, LangSmith, Deep Agents, store and MCP adapters in practice in appendix K; the control plane — envelope, admission control, degradation ladder, 429 semantics, token-cost arithmetic, guardrail products and the AI/MCP/agentic gateway layer — in appendix L; and how the labs and AI products actually handle large documents, memory, caching, their own RAG, gateways and agent orchestration — with the evidence graded primary / vendor-published / secondary — in appendix M; multimodal retrieval and non-text output (visual documents, tables, images, audio, voice, and the pixel injection surface) in appendix N; and whether to train an agent at all — RLVR, GRPO, the verifier problem and the five cheaper levers — in appendix O.

Rescope note (2026-10). Appendix L absorbed the control-plane half of three planned chapters, so 11, 12 and 16 were narrowed to what is left and is still worth its own page: unit economics and chargeback (11), the serving path and cache placement (12), and data isolation per tenant (16). Existing cross-references to 12 for caching still hold — it keeps caching as a serving-layer concern and delegates the prompt-cache mechanics to 06 §12.


4. Cross-reference map — theory already in this repo

Do not rewrite any of this here. Read it in place, then write only the AI-specific delta. Roughly 60% of the depth this roadmap needs already exists in other folders.

Retrieval & indexing → ../databases/

Need Read
HNSW internals — graph construction, M / efConstruction / efSearch, recall-vs-latency ../databases/11-hnsw-vector-search-internals.md
Vector search broadly — IVF, PQ, quantization, ANN tradeoffs ../databases/11-vector-search-internals.md
Index structures generally — B-tree, inverted, bitmap; why an index is a tradeoff not a win ../databases/06-indexing-internals.md
Scans, selectivity, access-method choice — the theory under hybrid retrieval ../databases/03-access-methods-and-table-scans.md
Columnar formats & encoding — for eval datasets and trace storage in Parquet ../databases/02-data-storage-formats-and-encoding.md
Query engine internals — pushdown, joins; needed for filtered vector search ../databases/04-query-engine-internals.md
DuckDB / chDB in-process OLAP — the right tool for eval-result and cost analytics ../databases/21-in-process-olap-duckdb-chdb.md
OLAP fundamentals — for the cost/usage aggregation layer ../databases/08-olap-databases.md
SQL performance — for the query layer over eval and cost data ../databases/15-sql-performance-deep-dive.md
Storage engine + WAL — if a project persists its own index ../databases/01-storage-engine-fundamentals.md, ../databases/14-write-ahead-log-internals.md
Replication — when the index shards ../databases/12-replication-and-distributed-storage.md

Observability & tracing → ../sre-observability/

Need Read
LLM & AI observability — the single most on-target document in the repo ../sre-observability/26-llm-and-ai-observability.md
OpenTelemetry deep dive — the substrate for all LLM tracing ../sre-observability/02-opentelemetry-deep-dive.md
Instrumentation practice ../sre-observability/03-instrumentation.md
Trace storage — where spans go and what that costs ../sre-observability/08-traces-storage.md
Metrics storage — for token/latency/quality time series ../sre-observability/06-metrics-storage.md
Query layer — serving the dashboards over all of it ../sre-observability/10-query-layer.md
Semantic conventions governance — directly applicable to GenAI semconv ../sre-observability/34-schema-and-semantic-conventions-governance.md
Cardinality & cost — per-tenant/per-model labels explode fast ../sre-observability/18-cardinality-and-cost.md
FinOps for observability — the cost-attribution pattern, reused for tokens ../sre-observability/31-finops-for-observability.md
Telemetry lakehouse — where eval + trace + cost data lands for analysis ../sre-observability/35-telemetry-lakehouse.md
Pipeline reliability — the ingestion side must not lose data ../sre-observability/28-telemetry-pipeline-reliability.md
Python observability ../sre-observability/42-python-observability.md
Alerting & SLOs — for retrieval-quality SLOs, which almost nobody defines ../sre-observability/12-alerting.md, ../sre-observability/13-slo-engineering.md
Multi-tenancy in telemetry ../sre-observability/19-multi-tenancy.md

Inference & GPU economics → ../gpu-observability/

Relevant only for self-hosted inference; skip for API-based work.

Need Read
LLM inference observability — TTFT, ITL, batching, queueing ../gpu-observability/14-llm-inference-observability.md
Capacity planning & cost optimization ../gpu-observability/12-capacity-planning-and-cost-optimization.md
Allocation vs utilization semantics ../gpu-observability/05-gpu-allocation-and-utilization-efficiency.md
Prometheus metric design & cardinality ../gpu-observability/08-prometheus-metrics-design-and-cardinality.md
Telemetry lakehouse + SQL analytics ../gpu-observability/17-telemetry-lakehouse-and-sql-analytics.md
Multi-tenant GPU observability ../gpu-observability/13-multi-tenant-gpu-observability.md

Engineering craft → ../python-mastery/

Need Read
Measurement methodology — the rigour that makes eval numbers defensible ../python-mastery/31-measurement-methodology.md
Testing strategy — evals are tests; this is how they're structured ../python-mastery/43-testing-strategy.md
asyncio internals — every LLM pipeline is I/O-bound and concurrent ../python-mastery/28-asyncio-internals.md
Async patterns and pitfalls — bounded concurrency, backpressure, cancellation ../python-mastery/29-async-patterns-and-pitfalls.md
Concurrency correctness ../python-mastery/30-concurrency-correctness.md
Profiling — when the pipeline is slow and it isn't the model ../python-mastery/32-profiling.md

31-measurement-methodology.md is the highest-leverage file in the repo for this track. Eval credibility is measurement methodology: an accuracy number you cannot explain the derivation of is worth less than no number at all.

Distribution & deployment

Need Read
Distributed systems foundations — for a sharded/replicated index ../distributed-systems/README.md
Failure detection & leader election ../databases/16-failure-detection-and-leader-election.md
Working reference implementations ../databases/failure_detection_phi_accrual.py, ../databases/failure_detection_gossip.py
Scale-tiered service patterns to imitate ../implementation/distributed-counter/, ../implementation/fastapi-rbac/
Deployment, when it's time ../k8s-learn/README.md (Track A only — do not detour into Track C)
The RAG-specific deployment delta — written, so read it before the two rows above appendix-e-deployment-and-compute.md (§15 maps every substrate doc worth reading, so you enter ../kubernetes/ with a question rather than a reading list)

5. Project ladder

Build-driven. Each project defines what to read next; nothing is read speculatively.

P0 — Eval harness (do this first)

A golden query set over a real corpus, with recall@k, MRR, nDCG, and a faithfulness check. Versioned datasets, results in DuckDB, a regression gate that fails CI when retrieval quality drops.

Why first: every later project needs it to prove anything at all. Without it, each subsequent change is a guess. It is also the part of the stack most people skip, which is why most RAG systems cannot answer "did that change help?"

Reads: 08-evaluation-methodology.md (written — it is the spec for this project; §19's labs are P0's task list), ../python-mastery/31-measurement-methodology.md, ../python-mastery/43-testing-strategy.md, ../databases/21-in-process-olap-duckdb-chdb.md.

P1 — Retrieval service

Hybrid BM25 + dense with RRF fusion and a cross-encoder reranker, behind an API. Every change benchmarked against P0. Keep a written log of what moved the numbers and what didn't — the negative results are the valuable half.

Reads: ../databases/11-hnsw-vector-search-internals.md, ../databases/06-indexing-internals.md, ../databases/03-access-methods-and-table-scans.md.

P2 — Traced and costed

OTEL instrumentation of P1: span per retrieval / rerank / generation / tool call, GenAI semantic conventions, token counts as metrics, per-tenant and per-model cost attribution, a dashboard showing quality and cost on the same screen.

Why it matters: quality and cost are a single tradeoff surface, and almost no RAG system exposes both. This is the layer that turns a demo into something operable.

Reads: ../sre-observability/26-llm-and-ai-observability.md, ../sre-observability/02-opentelemetry-deep-dive.md, ../sre-observability/31-finops-for-observability.md, ../sre-observability/18-cardinality-and-cost.md.

P3 — Agentic retrieval

One agent with real tool calls, multi-hop retrieval, retries, structured failure handling — plus a trajectory eval suite measuring task success, tool-call correctness, and cost per resolved task. Keep it small and make it reliable, not impressive.

Reads: ../python-mastery/29-async-patterns-and-pitfalls.md, ../python-mastery/30-concurrency-correctness.md.

P4 — Flagship: retrieval & eval data platform

P0–P3 assembled into a substrate other engineers could build on: multi-tenant ingestion → incremental indexing → serving → measurement → cost attribution. Sharded and replicated once single-node works.

Why one flagship rather than five small projects: only a flagship survives follow-up questions. Small projects demonstrate that you can follow a tutorial; a system with tenants, failure modes and a cost model demonstrates that you can design one.

Reads: the distributed and storage rows in §4, pulled in as each becomes load-bearing.


6. Rung ledger

Each artifact sits on exactly one rung, and each rung gets its own verb. Blurring them is the only thing here that would make a claim dishonest.

Artifact Rung How to describe it
P0–P4 code 2 — implemented "A project I built." No caveat needed.
Numbers produced by P0 1 — measured Always state the dataset, its size, and what counted as a hit.
Docs in this folder 3 — studied, unless written as build-logs Build-logs — "here's what I built and what the numbers did" — are rung 2 by construction

Any accuracy or latency figure you quote must come with a one-sentence account of how it was measured. If that sentence doesn't exist, cut the number.


7. Deliberately out of scope

Named so they don't creep in:

  • Fine-tuning, LoRA, RLHF — a different job family. Learn the decision boundary (when fine-tuning beats retrieval) in 01; skip the practice.
  • Training infrastructure, distributed training — requires cluster access this roadmap assumes you don't have.
  • CUDA, kernels, quantization implementation — adjacent-sounding, entirely different skill set.
  • Model architecture and research — not the job this folder trains for.
  • ../kubernetes/ beyond deployment basics — 46 files at ~130KB each, the most expensive directory in the repo and the least load-bearing here. If it becomes relevant, enter via ../k8s-learn/controller-tasks.md and operator-tasks.md only.