Appendix J — The production AI stack: layers, reference stacks, and what is inside each tool¶
Why this appendix exists. Most production LLM systems use the same few stacks. An enterprise RAG service, a LangGraph agent backend and a coding agent like Claude Code, Codex or Cursor are built from the same layers: a model client, a loop, tools, state, retrieval, a sandbox, evals and traces. Many "senior AI engineer" roadmaps list 80+ tool names. A list of names does not tell you which layer each tool belongs to, what it does inside, or how it fails.
This page does three things:
- It maps the usual roadmap checklist to the chapters in this folder that already cover each item (§1). Most of it is already written, so read it there.
- It fills the gaps that no chapter covers: the LangGraph package ecosystem and its internals (§2), the coding-agent harness (§4), the agent-backend runtime (§5), the eval and observability tools (§6), and the frameworks other than LangChain (§7).
- It gives three reference stacks (§3, §4, §5): what a staff engineer would pick for each layer by default, when to deviate, and what breaks.
Read from source. Package facts were checked against PyPI and the installed source code in October 2026:
langgraph1.2.15,langgraph-checkpoint-postgres3.1.3,langchain-mcp-adapters0.3.2,langgraph-supervisor0.0.31,deepagents0.7.25,openai-agents0.23.1,pydantic-ai2.56.0,claude-agent-sdk0.2.166,google-adk2.12.0,agent-framework1.21.0. Marked (source) or (probe) when checked by reading code or running it. Claims about closed products (Cursor, Claude Code internals) come from their public engineering posts and docs and are marked (public).Companions: appendix I goes deep on
create_agent, middleware and structured output; appendix K on the Functional API, runtime controls, LangSmith, Deep Agents, the store and MCP adapters. This page is the wider map.
Contents¶
- The roadmap checklist, mapped to this folder
- The LangChain / LangGraph package map, with internals
- Reference stack A: enterprise RAG
- Reference stack B: coding and code-execution agents
- Reference stack C: the agent backend
- Evaluation and observability tools
- Frameworks other than LangChain, by their core idea
- Providers and serving: the facts that leak through abstractions
- What to learn first, and what to skip
- Interview questions
- Sources
1. The roadmap checklist, mapped to this folder¶
In plain words. Before learning a new tool, find out whether its layer is already covered. The left column is the usual roadmap item. The right column says where to read it.
Real-world example. "Learn LangGraph checkpointing" and "learn Temporal" look like two topics. They are one question, how to survive a crash in the middle of a run.
21§5 and §14 answer it once.
| Roadmap item | Covered in | Gap filled here |
|---|---|---|
| Models, messages, prompts, tool binding, output parsers | 20 §2–§8 |
— |
init_chat_model, create_agent, middleware, retries, fallbacks |
appendix I | — |
| Structured output (provider vs tool strategy, Pydantic, Instructor) | 07 §3–§7, §12; appendix I §5 |
— |
StateGraph, reducers, MessagesState, Command, Send, ToolNode |
21 §2–§4, §9 |
— |
| Checkpointing, interrupts, resume, streaming, subgraphs | 21 §5–§8 |
storage internals and growth (§2.2); durability modes and streaming v2 in appendix K §2 |
| Retry policies, loop limits, durable execution, LangGraph vs Temporal | 21 §11–§14 |
Functional API, retry/timeout/cache defaults in appendix K §1–§2 |
| Agent patterns: ReAct, router, supervisor, plan-and-execute, reflection | 22 §3, §5 |
supervisor library status (§2.1) |
| Packages: checkpoint-postgres/sqlite, store, supervisor, swarm, sdk, cli, mcp-adapters, classic | 20 §12 (0.x split) |
§2 |
| Document parsing (PyMuPDF, Unstructured, Docling, OCR) and chunking | 02, appendix D |
— |
| Embeddings, rerankers, bi- vs cross-encoder | 01, 04 §4–§8 |
— |
| Vector DBs (pgvector, Qdrant, Pinecone, Milvus, OpenSearch), HNSW/IVF/PQ, filtering | 03, appendix D |
— |
| Hybrid search, RRF, query rewriting, multi-hop, GraphRAG | 04, 05, appendix H |
— |
| Measuring retrieval separately from answers | 08, appendices F, G |
— |
| Providers, LiteLLM, routing, fallback, quotas, cost | 23 |
leaking provider differences (§8) |
| vLLM, Ollama, GPUs, Compose vs Kubernetes, pipeline orchestration | appendix E | — |
| MCP, OpenAPI → tools, JSON Schema | 24, 26 |
langchain-mcp-adapters sessions (§2.3) |
| A2A | 27 |
— |
| OAuth/OIDC, delegated identity, tool-boundary authz | 29, 24 §6 |
— |
| Prompt injection, untrusted tool output, sandboxing | 17, 28 |
coding-agent sandbox layers (§4.5) |
| Memory: short-term, long-term, semantic, episodic, deletion | 25 |
— |
| Context engineering, compaction, sub-agents, skills | 30 |
— |
| Agent evaluation, trajectory grading, SWE-bench / Terminal-Bench | 14 |
— |
| LangSmith, Langfuse, Phoenix, Ragas, DeepEval, promptfoo | planned 19 |
§6; LangSmith in practice in appendix K §3 |
| OpenTelemetry GenAI conventions | ../sre-observability/26-llm-and-ai-observability.md |
— |
| LlamaIndex, PydanticAI, OpenAI Agents SDK, ADK, CrewAI, SK/AutoGen, Haystack, DSPy | 22 §11 (partial, and SK/AutoGen are now merged) |
§7 |
| Coding agents (Claude Code, Codex, Cursor, Aider) | 05 §5.6 (search only) |
§4 |
| FastAPI + queue + Postgres agent backend, double-texting, failure modes | 21 §15–§16 (partial) |
§5 |
2. The LangChain / LangGraph package map, with internals¶
In plain words. LangChain is many separately released packages. Some are core, some are optional, some are not recommended any more, and at least one package with an official-looking name is not from LangChain at all.
Real-world example. A roadmap tells you to install
langgraph-store-postgresfor long-term memory. That PyPI package is published by an independent developer, not by LangChain. The official Postgres store ships insidelanggraph-checkpoint-postgresaslanggraph.store.postgres. Installing a look-alike into a service that holds user memory is a supply-chain decision. Check the publisher before youpip install.
2.1 The map¶
| Package (Oct 2026) | What it is | Use it? | Staff note |
|---|---|---|---|
langchain 1.x |
create_agent, middleware, init_chat_model, tool and message re-exports |
yes | appendix I |
langchain-core |
messages, Runnable, tools, BaseChatModel, callbacks |
always (dependency) | the only package partner integrations depend on |
langchain-<provider> (-openai, -anthropic, -google-genai, -aws, -ollama, …) |
one chat/embedding class per provider | yes, pinned | provider features (caching, thinking, built-in tools) arrive here first, and behave differently per package |
langchain-text-splitters |
splitters, including code-aware ones | optional | 02 covers what to use instead for hard documents |
langchain-classic |
the old chains, AgentExecutor, memory classes |
only for legacy code | new code should not import it |
langchain-tests |
standard test suites for integration authors | only if you write an integration | not an app-testing tool |
langgraph |
StateGraph, Pregel runtime, channels, Command, Send, interrupt |
yes | 21 |
langgraph-prebuilt |
ToolNode, ToolRuntime, the deprecated create_react_agent |
indirectly | used by create_agent |
langgraph-checkpoint |
BaseCheckpointSaver, InMemorySaver, serializers, BaseStore |
dependency | InMemorySaver is for tests only |
langgraph-checkpoint-postgres |
PostgresSaver, ShallowPostgresSaver, and PostgresStore (langgraph.store.postgres) |
yes, for production | §2.2 |
langgraph-checkpoint-sqlite |
SqliteSaver, SqliteStore |
local dev, single-process tools | one writer; not for multi-replica services |
langgraph-store-postgres |
third-party (independent author, its own SQLAlchemy/psycopg2 schema) | no | name squatting risk; use the official store above |
langgraph-supervisor |
prebuilt supervisor multi-agent graph | not for new code | its own README now recommends the supervisor pattern "directly via tools", and it still imports the deprecated create_react_agent (source). Write the supervisor as an agent whose tools call sub-agents |
langgraph-swarm |
prebuilt peer handoff graph | rarely | handoffs are one Command(goto=...) tool; write them yourself (21 §13) |
langchain-mcp-adapters |
MCP servers → LangChain tools; MultiServerMCPClient |
yes, with care | §2.3 |
deepagents |
create_deep_agent: create_agent + planning, filesystem, sub-agent, summarization, skills, memory and prompt-caching middleware (source) |
for long tasks | the open version of the coding-agent harness (§4) |
langgraph-cli, langgraph-api, langgraph-sdk |
dev server (langgraph dev), the Agent Server runtime, and its client |
if you deploy on LangChain's platform | §5.3 |
langsmith |
tracing, datasets, experiments SDK | if you use LangSmith | §6 |
2.2 Checkpointer internals: what is stored, and how fast it grows¶
PostgresSaver.setup() creates four tables (source):
| Table | Key | Holds |
|---|---|---|
checkpoints |
(thread_id, checkpoint_ns, checkpoint_id) |
one row per super-step: channel versions, parent_checkpoint_id, metadata (JSONB) |
checkpoint_blobs |
(thread_id, checkpoint_ns, channel, version) |
the serialized value of each channel, stored again whenever its version changes |
checkpoint_writes |
(thread_id, checkpoint_ns, checkpoint_id, task_id, idx) |
pending writes of each task in a super-step |
checkpoint_migrations |
v |
schema version, applied by setup() |
Three consequences follow.
1. Partial completion is recoverable at task level. When three parallel tool tasks run in one
super-step and one fails, the two successful tasks' writes are already in checkpoint_writes. On
resume, only the failed task runs again. The side effects of the two finished tasks are not repeated
by LangGraph, but anything a failed task did before it raised is repeated. Idempotency keys
are still needed (appendix I §6.3).
2. Storage grows with the square of the conversation length. The messages channel changes on
every step, and each new version stores the whole list again. (probe: a one-node graph,
MessagesState, SqliteSaver, one human message plus one 1,000-character AI message per turn)
| Turns | Checkpoint rows | Checkpoint bytes | Growth vs 10 turns |
|---|---|---|---|
| 10 | 30 | 220 KB | 1× |
| 20 | 60 | 857 KB | 3.9× |
| 40 | 120 | 3.38 MB | 15.4× |
Doubling the turns multiplies storage by about four. Forty turns holding about 41 KB of actual text take 3.4 MB. At a million threads that is the difference between a small table and a database migration project. The levers:
ShallowPostgresSaverkeeps only the latest checkpoint. You lose time travel; you keep resume and interrupts.- A retention job that deletes old checkpoints per thread (
21§5.7), anddelete_thread(thread_id)for user deletion requests (GDPR erasure,25§10). DeltaChannel(beta inlanggraph1.2) stores only writes and rebuilds the value by replaying them through the reducer (source). It is the real fix, but its on-disk format is marked "not yet stable". Do not adopt it for data you must keep for years without a migration test.
3. Connection settings are not optional. from_conn_string opens the connection with
autocommit=True, prepare_threshold=0 and row_factory=dict_row (source). If you build your own
psycopg pool, use the same settings. prepare_threshold=0 is also what makes it work behind
PgBouncer in transaction mode.
PostgresStore (long-term memory) is a separate table pair, store and store_vectors. It has
optional pgvector semantic search via an index config, and TTL with ttl_minutes and refresh on
read (source). Namespaces are tuples such as ("tenant-42", "user-7", "prefs"). Namespaces are not
access control. Any code with the store handle can read any namespace, so derive the namespace from
runtime.context, never from model output (25 §10).
2.3 langchain-mcp-adapters: one session per tool call by default¶
MultiServerMCPClient(...).get_tools() returns tools that open a new MCP session for every tool
call (source: the get_tools docstring and tools.py). For a stdio server that means
starting a new process per call. For a remote server it means a new initialize handshake per call.
It also means a stateful server (a browser session, an open database transaction) loses its state
between calls.
| Need | Do |
|---|---|
| Stateless remote tools, low call volume | default get_tools() is fine |
stdio servers, or many calls per run |
async with client.session("name") as s: tools = await load_mcp_tools(s) and keep the session for the run |
| Stateful server | one explicit session per agent run, closed in after_agent or a finally |
| Many servers, many tenants | an MCP gateway in front (26 §8), not one client per tenant per pod |
3. Reference stack A: enterprise RAG¶
In plain words. An enterprise RAG service is a search engine with a model on top. The hard parts are permissions, freshness and measurement, not the model call.
Real-world example. An HR assistant passes its demo. In month two, an employee who moved teams still gets answers from the old team's documents. The vector store kept the old ACLs, because permission changes were never synced. Nothing in the model layer could have prevented it.
3.1 The default stack¶
| Layer | Default pick | Deviate when | Inside / what breaks | Read |
|---|---|---|---|---|
| Connectors and sync | your own sync jobs per source (SharePoint, Confluence, Drive, DBs), with change feeds | a managed connector platform is already licensed | ACL sync is a separate pipeline from content sync. Permission changes must reach the index faster than content changes | 04 §12, appendix E §8 |
| Parsing | Docling or PyMuPDF for born-digital PDFs; OCR / vision models only for scans | heavy tables or forms | parser upgrades silently change chunk boundaries and IDs; version the parser | 02, appendix D |
| Chunking | structure-aware (headings, tables kept whole), ~300–800 tokens, contextual prefix | very short FAQ-style documents | chunk IDs must be stable across re-ingests, or citations and eval labels break | 02 |
| Embeddings | one hosted or open model, version stored on every vector | multilingual or domain gap measured on your data | changing the model means re-embedding everything; plan dual-write and a cutover | 01 |
| Index | Postgres + pgvector (HNSW) up to roughly tens of millions of vectors; OpenSearch/Elasticsearch when you also need mature BM25 | >100M vectors, or strict latency at high QPS → a dedicated engine (Qdrant, Milvus, Vespa…) | filtered ANN with very selective filters loses recall; ACL filter inside the search, not after | 03, 04 §12 |
| Lexical | Postgres FTS or OpenSearch BM25 | — | analyzer choice (stemming, tokenization of codes like ERR-4102) matters more than the engine |
04 §2 |
| Fusion and rerank | RRF over dense + BM25, then a cross-encoder over the top 50 | latency budget below ~150 ms → smaller reranker or none | reranker is usually the largest latency item; measure p95 | 04 |
| Query understanding | rewrite only when measured to help; routing for distinct corpora | — | every extra LLM call adds latency and a new failure mode | 05 |
| Generation | structured output with citations to chunk IDs; abstain path | — | citation validity must be checked in code, not trusted | 07 |
| Guardrails | injection defenses on retrieved text, PII handling, output checks | regulated data → DLP | retrieved documents are untrusted input | 17 |
| Gateway | one model gateway for keys, quotas, routing, fallback, cost | single provider and single team | duplicated retries across layers multiply outage load | 23 |
| Eval | golden set + retrieval metrics + judged faithfulness, gated in CI | — | eval sets rot when documents change; refresh labels on schedule | 08, F, G |
| Observability | OTel spans per stage + an LLM tracing tool | — | traces contain document text and PII; set retention and redaction | §6, ../sre-observability/26-llm-and-ai-observability.md |
3.2 What breaks in year one¶
| Failure | Mechanism | Prevention |
|---|---|---|
| Stale permissions | ACL changes not propagated; summaries or caches built from restricted docs | ACL sync SLO; derived data inherits the intersection of source readers (appendix H §6) |
| Silent quality drop after a parser or embedding upgrade | chunk boundaries and vectors change; no one re-ran evals | version every stage; CI eval gate on any ingest-pipeline change |
| Re-embedding project nobody planned | the model was changed or deprecated | model version on every vector, dual index during migration |
| Freshness complaints | batch nightly ingest | change-feed ingestion with a staleness SLO |
| Cost per tenant unknown | no attribution | tenant ID on every span and token count (23 §7) |
| "Search is fine, answers are wrong" debates | no stage-level metrics | measure each stage separately (appendix F §5) |
4. Reference stack B: coding and code-execution agents¶
In plain words. Claude Code, Codex CLI, Cursor's agent, Aider and LangChain's
deepagentsare the same machine: a model in a tool loop, with file tools, a shell, a way to search the code, a way to keep the context small, and a sandbox with permissions. They differ mostly in how they edit files, how they search, and how they isolate commands.Real-world example. A team builds an internal "fix the failing test" agent on a generic agent framework, with a
write_file(path, content)tool. On a 2,000-line file the model rewrites the whole file, drops 40 lines it did not mean to touch, and the diff review misses it. Coding agents use targeted edit formats for exactly this reason (§4.2).
4.1 Anatomy of the harness¶
| Component | What it does | Common choices (public) | Why it matters |
|---|---|---|---|
| Loop | model ↔ tools until no tool calls | a plain tool loop; no planner graph | simple loops with good tools beat complex graphs here (22 §1) |
| Read tools | read file ranges, list, glob | Read with line ranges and a size cap |
full-file reads of huge files fill the context |
| Search | find code | grep/ripgrep + glob in a loop (Claude Code, Codex); embeddings index plus grep (Cursor); a tree-sitter "repo map" ranked by references (Aider) |
05 §5.6: exact identifiers favour grep; conceptual questions favour embeddings; layered is the consensus |
| Edit tools | change files | exact string replace (Claude Code Edit), patch format (apply_patch in Codex), search/replace blocks (Aider), a separate fast "apply" model (Cursor) |
§4.2 |
| Shell | run tests, builds, git | Bash with timeout, output truncation, background processes |
the most powerful and most dangerous tool |
| Context files | project rules loaded at start | CLAUDE.md, AGENTS.md, .cursor/rules |
cheap, high-leverage steering; also an injection surface if the repo is untrusted |
| Context management | stay inside the window | automatic compaction near the limit, tool-result clearing, sub-agents with their own context, skills loaded on demand | 30 §6–§8 |
| Planning | keep track of multi-step work | a todo-list tool | gives the model a scratchpad and the user a progress view |
| Permissions | decide which actions need approval | modes (ask / allow edits / plan only / bypass), allow and deny lists per tool and command pattern | §4.5 |
| Sandbox | limit what commands can touch | OS-level (macOS Seatbelt, Linux bubblewrap/Landlock), containers, microVMs for cloud agents | §4.5 |
| Hooks | run your code at loop events | pre/post tool use, prompt submit, stop | deterministic policy (lint after edit, block a path) outside the model |
| Extensibility | more tools | MCP servers, plugins | each server adds tool definitions to every prompt (30 §2) |
| Checkpoints / undo | roll back file changes | snapshot before edits; git | lets users trust larger autonomous steps |
| Headless / SDK | run the same harness from code | Claude Agent SDK (drives the Claude Code CLI as a subprocess, source), codex exec, deepagents |
CI bots and internal agents reuse a tested harness instead of a new one |
4.2 Edit formats are the core design choice¶
| Format | How it works | Fails how | Notes |
|---|---|---|---|
| Whole file | model writes the full new file | drops or changes unrelated lines; slow and costly on big files | fine for new and small files only |
Exact string replace (old_string → new_string) |
tool finds old_string, requires it to be unique, replaces it |
fails loudly when the text is not found or appears more than once | loud failure is the point: the model re-reads and retries. Requires a "read before edit" rule so the model edits what is really on disk |
Unified diff / patch (apply_patch) |
model writes a patch with context lines | wrong line context → the patch does not apply | compact for many small changes; needs a tolerant applier |
| Search/replace blocks | like string replace, several blocks per message | fuzzy matching can apply to the wrong place | Aider published benchmarks showing edit format changes success rates noticeably for the same model |
| Draft + apply model | big model writes a sketch; a small fast model merges it into the file | the apply model can mis-merge | lower latency on big files; one more model to evaluate |
Staff rule: choose a format that fails loudly and locally. A silent wrong edit costs more than a failed tool call, because the model sees the failure and fixes it, while nobody sees the silent one.
4.3 Search: agentic grep vs an index¶
Covered in 05 §5.6. Summary for design reviews:
- Agentic grep has no index to build, keep fresh or secure. It is exact on identifiers, and it costs more turns on vague questions.
- An embeddings index answers conceptual questions ("where do we handle retries?") in one call. It needs incremental sync. Cursor describes hashing files into a Merkle tree so only changed files are re-chunked and re-embedded, and obfuscating file paths before they leave the machine (public). It is also a second copy of the source code that you must secure.
- Default for an internal agent: grep, glob and read first; add LSP or symbol tools for precision; add an index only when your evals show conceptual queries failing.
4.4 Building your own coding or code-execution agent¶
| Best practice | Why |
|---|---|
Start from an existing harness (Claude Agent SDK, codex exec, deepagents) |
the hard parts (edit tools, compaction, truncation, permissions) are already tuned; your value is in tools and policies |
| Tool outputs truncated with a hint ("output truncated; use a narrower command") | one cat of a log file can fill the window |
| Tests as the stop condition | "done" means tests pass, checked by the harness, not claimed by the model |
| A per-repo context file with build and test commands | most failed runs start with the agent guessing how to run tests |
| Separate read-only exploration sub-agent | keeps exploration output out of the main context (30 §6) |
| Every run in a fresh sandbox, with network egress allow-listed | §4.5 |
| Evaluate on your own repo tasks, not only SWE-bench | public benchmark scores do not transfer to your codebase (14) |
| Worst practice | What happens |
|---|---|
write_file as the only edit tool |
silent, large, unreviewable diffs |
| Shell with the developer's credentials and no sandbox | one injected instruction in a README or issue can run curl … \| sh with production credentials (28) |
Trusting AGENTS.md / CLAUDE.md from an untrusted repo or PR |
the context file is a prompt; a malicious repo controls your agent |
| Unlimited turns and tokens in CI | one confused run burns a day of budget; set turn and cost caps |
| Auto-approve all commands "because it is in Docker" | the container often has the repo's secrets, the Docker socket or open network |
4.5 Sandbox and permissions, layered¶
| Layer | Stops | Does not stop |
|---|---|---|
| Permission prompts / allow lists | unexpected commands, when a human reads them | approval fatigue; humans approve what they do not read |
| Hooks with deterministic policy | known-bad patterns (paths, commands) | anything not in the pattern list |
| OS sandbox (Seatbelt, bubblewrap, Landlock) | writes outside the workspace; with a network proxy, unapproved hosts | kernel exploits; data the process may read legitimately |
| Container | most host access, if configured strictly | shared kernel; mounted secrets; Docker socket |
| gVisor / microVM (Firecracker) | kernel-level escapes, for multi-tenant cloud agents | anything reachable over the network you allowed |
| Egress control | exfiltration to unknown hosts | exfiltration through allowed hosts (GitHub gists, package registries) |
The two that matter most: filesystem isolation and network isolation together. Either alone is
weak. Without network limits, a sandboxed agent can still send your code out. Without filesystem
limits, it can read your SSH keys and send them out through any allowed host (28 §7).
5. Reference stack C: the agent backend¶
In plain words. An agent in production is a backend service that runs long, stateful jobs. It needs an API, a database for state, a queue for long runs, a stream to the browser, and rules for what happens when a user sends a second message while the first run is still going.
Real-world example. A user sends "refund my order", then, after 5 seconds of no reply, sends "hello?". Both requests start a run on the same thread. Two runs append to the same message list, and the agent issues the refund twice. The fix is a per-thread run policy (§5.2), not a better prompt.
5.1 Topology¶
browser ──SSE/WebSocket──► API (FastAPI, stateless, N replicas)
│ auth → tenant/user/scopes → context
├─ short runs: run in-process, stream events
└─ long runs: enqueue ──► queue (Redis/RabbitMQ/SQS)
│
workers (N replicas)
│ LangGraph run
┌──────────────────────────────────┼──────────────────────────┐
▼ ▼ ▼
Postgres: checkpoints + store model gateway (23) tools / MCP gateway (26)
(+ app tables, outbox) keys, quotas, fallback authz per call (29)
│
OTel traces + LLM tracing (6) ── token & cost metrics per tenant (23 §7)
| Decision | Default | Why |
|---|---|---|
| Where runs execute | in the API for runs under ~30 s; workers for longer runs | HTTP timeouts, deploys and autoscaling kill long in-request runs |
| State | Postgres checkpointer, one DB for checkpoints, store and app data | one backup, one consistency model; transactional outbox for side effects |
| Streaming | SSE from the API; workers publish events through Redis pub/sub or the DB | the client reconnects with a cursor; runs must not depend on the socket |
| Durable multi-day work | LangGraph checkpoints for agent state; Temporal when you need timers, cross-system sagas, and strict exactly-once activities | 21 §14 |
| Background jobs (ingest, eval) | the same queue, separate worker pools | an eval backlog must not starve user runs |
5.2 Concurrent messages on one thread ("double texting")¶
LangChain's Agent Server names four strategies for a new run on a thread that already has one running. They are the right vocabulary even if you build your own:
| Strategy | Behaviour | Use when |
|---|---|---|
| reject | refuse the new run | side-effecting agents; the UI disables input while running |
| enqueue | run the new input after the current run | chat where every message must be answered in order |
| interrupt | stop the current run, keep its progress, start the new one | the user corrects themselves mid-run |
| rollback | stop the current run, discard its progress, start the new one | the first message was a mistake |
Implement it with a per-thread lock or lease (a row lock or a Redis lock with a fencing token). Without it, two workers write checkpoints for the same thread concurrently, and the checkpoint chain forks.
5.3 Build vs LangChain's platform¶
The LangGraph platform (Agent Server, run by langgraph-api, with langgraph-sdk as client and
langgraph-cli for local dev) provides threads, runs, a task queue, cron, double-texting strategies,
streaming and a store as an HTTP API. It is the fastest path if you are already on LangSmith and
its hosting or licensing fits. Build your own when you need your own auth model, data residency, or
one platform for many frameworks. The design in §5.1 is the minimum you would build.
5.4 Failure modes and their fixes¶
| Failure | Mechanism | Fix | Read |
|---|---|---|---|
| Duplicate tool execution | retries, resumes, parallel Sends, double texting |
idempotency keys from thread_id + tool_call_id; per-tool call limits |
appendix I §6.3, 24 §8 |
| Partial graph completion | crash mid-super-step | task-level pending writes (§2.2); make side effects idempotent | 21 §5 |
| Checkpoint migration | state schema or node names changed while threads are paused | versioned state; test resuming old checkpoints in CI | appendix I §9.2 |
| Concurrent state updates | two writers to one key in one super-step, or two runs on one thread | reducers; per-thread lease | 21 §3.3, §5.2 here |
| Provider throttling | 429s under load | gateway quotas, backoff with jitter, fallback, admission control | 23 §5–§6 |
| Queue backlog | slow runs, retry storms | per-tenant concurrency limits, separate pools, backlog-age alerts | ../python-mastery/29-async-patterns-and-pitfalls.md |
| Latency budget exceeded | too many steps | step and token budgets per request class; stream early | 22 §12 |
| Storage growth | quadratic checkpoint growth | shallow saver, retention, delete_thread |
§2.2 |
6. Evaluation and observability tools¶
In plain words. Two jobs: see what happened in a run (traces) and decide whether a change made things better (evals). Some tools do both. The real choice is where your trace data is allowed to live and whether the tool fits your existing telemetry.
Real-world example. A team adds the OpenAI Agents SDK to a service under EU data-residency rules. Its default trace exporter sends spans to OpenAI's trace ingest endpoint (source:
agents/tracing/processors.py). The spans include prompts and tool outputs unlesstrace_include_sensitive_datais turned off. Nobody approved that data flow.
| Tool | How it hooks in | Strength | Watch out for |
|---|---|---|---|
| LangSmith | LangChain callbacks / @traceable, env-var switch; OTel ingest too |
deepest LangChain/LangGraph view (graph nodes, middleware spans), datasets, experiments, online evaluators | SaaS by default (self-host is an enterprise plan); per-trace pricing at high volume |
| Langfuse | SDK built on OpenTelemetry; LangChain callback handler | open source, self-hostable, prompt management, LLM-as-judge | you run ClickHouse + Postgres + Redis + object storage when self-hosting |
| Arize Phoenix | OpenTelemetry + OpenInference instrumentors | open source, retrieval-focused views, runs locally | different semantic conventions (OpenInference) from OTel GenAI; map them |
| Braintrust | SDK + proxy | experiment-first workflow, fast diffing between runs | SaaS |
| Ragas | library: metrics over (question, contexts, answer, reference) | standard RAG metric names | judge-based scores need calibration (08 §11); names collide with other metrics (appendix G) |
| DeepEval | pytest-style LLM test cases | fits CI as unit-test-shaped checks | same judge-calibration caveat |
| promptfoo | YAML/CLI test matrix over prompts × models × assertions, plus red-team suites | regression and security testing in CI without code | assertions can drift into "vibes"; keep deterministic checks first |
| OpenTelemetry GenAI semconv | standard span and metric names for model and tool calls | vendor-neutral; joins LLM spans with the rest of your traces | still evolving; content capture is opt-in |
Staff defaults:
- Instrument with OpenTelemetry first, and export to whatever tool you choose. Changing vendors is then an exporter change, not a re-instrumentation.
- Traces are a data store with PII. Set redaction (LangSmith
hide_inputs/hide_outputs, appendix I'sTracePolicy), retention, and region before the first production trace. - Sample traces, not evals. Keep 100% of errors and a sample of successes; run evals on fixed datasets.
- Deterministic checks first, judges second, humans third (
08§11). A judge score without a measured agreement with human labels is a rumor.
7. Frameworks other than LangChain, by their core idea¶
In plain words. Every agent framework has the same loop. They differ in their one big idea: typed dependency injection, handoffs, event-driven workflows, roles, or prompt optimization. Learn the idea; you can read the API docs when you need them.
Real-world example. A Python backend team needs one typed extraction agent with tests. A LangGraph graph is more structure than they need, and a typed agent library with a test model fits in a day. A month later they need a 3-day approval workflow. That is a graph or a workflow engine, whatever the agent library.
| Framework | Core idea | Inside (source where marked) | Pick when |
|---|---|---|---|
| PydanticAI | typed agents: Agent[Deps, Output], dependency injection via RunContext |
structured output defaults to a tool named final_result; NativeOutput and PromptedOutput are the other modes (source); TestModel/FunctionModel for tests; durable execution through Temporal, DBOS or Prefect integrations (source: durable_exec/) |
Python backends that value types and tests; small to medium agents |
| OpenAI Agents SDK | agents + handoffs + guardrails + sessions | a handoff is a tool named transfer_to_<agent> (source); guardrails raise when tripwire_triggered (source); tracing on by default to OpenAI |
OpenAI-centric stacks; simple multi-agent handoffs |
| Claude Agent SDK | the Claude Code harness as a library | runs the Claude Code CLI as a subprocess (source); hooks, permission callbacks, in-process MCP tools, sub-agents, skills | coding and file/shell-heavy agents (§4) |
| Google ADK | composable agent types | LlmAgent plus workflow agents SequentialAgent, ParallelAgent, LoopAgent (source); session services; A2A support |
Gemini / Vertex AI stacks; workflow-shaped multi-agent systems |
| Microsoft Agent Framework | successor that merges Semantic Kernel and AutoGen | agents plus graph workflows with checkpointing (source: _workflows/_checkpoint.py); .NET and Python |
Microsoft / Azure / .NET organisations. 22 §11's separate SK and AutoGen rows predate the merge |
| LlamaIndex | data framework: loaders, indexes, query engines; event-driven Workflows | steps triggered by typed events | RAG-heavy work where its ingestion and index abstractions save time |
| Haystack | typed component pipelines | components with declared inputs and outputs, wired into a graph | search-centric pipelines; teams that want explicit pipeline graphs |
| CrewAI | roles, goals and tasks | sequential or hierarchical process over role agents | prototyping the shape of a multi-agent workflow (22 §11) |
| DSPy | programs, not prompts: signatures + modules, optimized against a metric | optimizers such as MIPROv2 and GEPA rewrite instructions and pick few-shot examples (source: teleprompt/) |
you have a metric and labelled data, and a pipeline worth tuning |
| Vercel AI SDK / Mastra | TypeScript agents and streaming UI | — | TypeScript full-stack teams |
How to choose in one rule: pick by the hardest requirement, not the first demo. Durable
approvals and resumable state → LangGraph, a workflow engine, or Microsoft Agent Framework workflows.
Strict typing and testability → PydanticAI. A coding or shell agent → Claude Agent SDK, codex, or
deepagents. A metric to optimize against → DSPy. Many frameworks can be combined: a LangGraph node
can call a PydanticAI agent, and everything can export OTel.
8. Providers and serving: the facts that leak through abstractions¶
In plain words.
init_chat_model("provider:model")makes switching providers a one-line change. It does not make the providers behave the same.
| Difference | Example of the leak | Mitigation |
|---|---|---|
| Structured output support | native JSON-schema on some models, tool-call fallback on others; supported schema features differ | appendix I §5; pin the strategy; eval per provider |
| Prompt caching | automatic on some providers, explicit cache markers on others; different TTLs and minimum lengths | 30 §10; measure cache hit rate per provider |
| Parallel tool calls | some models emit several per turn, some one; some allow turning it off | idempotent tools; do not depend on either behaviour |
| Reasoning / thinking | separate content blocks, separate token budgets, sometimes incompatible with forced tool choice | use content_blocks; test with your strategy |
| Limits | context window, max output tokens, rate limits per org and per model | gateway quotas (23 §6); model profiles |
| Errors | different status codes and retryable conditions | classify errors in one place (the gateway) |
Serving your own models (appendix E §6): vLLM's throughput comes from paged KV-cache memory,
continuous batching and prefix caching. Its structured output uses grammar-constrained decoding
(07 §4). Ollama is for local development, not for multi-tenant serving. LiteLLM translates many
provider APIs into the OpenAI format as a library or a proxy. Treat that proxy as your gateway (23),
with the same reliability requirements.
9. What to learn first, and what to skip¶
In plain words. Learn layers, not logos. One tool per layer, learned deeply, beats ten learned from their README.
| Order | Layer | Learn deeply | Recognize only |
|---|---|---|---|
| 1 | Loop and state | LangGraph (21), create_agent + middleware (appendix I) |
swarm/supervisor libraries |
| 2 | Retrieval | one vector store (pgvector), BM25, RRF, one reranker, the eval harness (03, 04, 08) |
every other vector DB |
| 3 | Measurement | golden sets, retrieval metrics, judge calibration, traces (08, 14, §6) |
the 5 eval SaaS products you are not using |
| 4 | Tools and integration | typed tools, authz at the boundary, MCP, idempotency (24, 26, 29) |
A2A, until you need cross-org agents (27) |
| 5 | Production runtime | the agent backend (§5), gateway (23), sandboxing (28, §4.5) |
Kubernetes details beyond Track A |
| 6 | One alternative framework | PydanticAI or the Agents SDK, enough to argue a choice (§7) | the rest |
| 7 | Specialisations | GraphRAG (appendix H), coding agents (§4), DSPy | — |
Skip, unless the job requires it: memorizing integration class names, comparing ten vector DBs on blog benchmarks, multi-agent frameworks before a single agent works, and fine-tuning (README §7).
The readiness checklist, as builds in this repo. Each item is something to build and measure, not something to read:
| Checklist item | Build |
|---|---|
| Custom graph with reducers, branches, bounded loop, checkpoint + resume, idempotency | 21 §19 labs; capstone C3 in LABS.md |
| Ingestion, hybrid retrieval, reranking, metadata filter, retrieval eval | capstone C1 |
| Typed tools with authz, MCP server, approval workflow, injection handling | 24 and 26 labs, labs/tool-registry/, capstone C2 |
| Compare LangGraph with one alternative | §7: build the same 2-tool agent in both, compare code, traces and failure handling |
| Offline eval dataset, traces of failed runs, latency/token/cost, deploy with recovery | P0–P2 in README.md §5; appendix I §10 probes as tests |
10. Interview questions¶
Q1. Your agent service stores conversations in PostgresSaver. Six months later the checkpoint
table is the largest in the database. Why, and what do you do?
Each super-step stores a new version of every changed channel, and the messages channel holds the
whole history, so storage grows with the square of thread length (§2.2: 4× per doubling). Use
ShallowPostgresSaver if you do not need time travel, add a retention job, delete threads on
request, and consider DeltaChannel once its format is stable.
Q2. Why do coding agents use exact-string-replace or patch edit tools instead of write_file?
Targeted edits fail loudly and locally when the model's view of the file is wrong. Whole-file writes
fail silently by changing lines nobody asked to change, and they cost output tokens proportional to
file size.
Q3. Claude Code searches with grep; Cursor keeps an embeddings index. Which is right?
Both, for different queries. Exact identifiers and call chains favour grep; conceptual questions
favour embeddings. An index costs sync, freshness and a second copy of the code to secure. Default
to grep and add an index when evals show conceptual queries failing (05 §5.6).
Q4. A user sends a second message while the first run is still calling tools. What should happen? Choose a per-thread policy: reject, enqueue, interrupt or rollback (§5.2). Enforce it with a per-thread lease so two runs never write the same thread's checkpoints concurrently.
Q5. You use MultiServerMCPClient.get_tools() with a stdio server, and latency per tool call is
800 ms. Why?
By default each tool call opens a new MCP session, which for stdio starts a new server process.
Open one session per run with client.session(...) and load_mcp_tools(session).
Q6. When would you choose PydanticAI, the OpenAI Agents SDK or the Claude Agent SDK over LangGraph? PydanticAI for typed, testable agents in Python services. The Agents SDK for simple handoff-based multi-agent systems in an OpenAI stack, with its tracing destination checked. The Claude Agent SDK for file and shell agents, where its harness is the product. LangGraph when you need explicit control flow, durable state and interrupts. Pick by the hardest requirement.
Q7. What is the minimum sandbox for an agent that runs shell commands on untrusted repositories? A fresh isolated environment per run (container with a hardened runtime, gVisor or a microVM), no host credentials, the workspace as the only writable path, and egress limited to an allow-list through a proxy. Filesystem and network isolation together; either alone is weak (§4.5).
Q8. Which observability decision should be made before the first production trace? Where traces are stored, what is redacted, and for how long. Traces contain prompts, retrieved documents and tool outputs, so they are a PII store with its own residency and retention rules.
11. Sources¶
- PyPI metadata and installed source, October 2026:
langgraph(channels/delta.py),langgraph-checkpoint-postgres(checkpoint/postgres/base.py,shallow.py,store/postgres/base.py),langchain-mcp-adapters(client.py,tools.py),langgraph-supervisor(README note,supervisor.py),langgraph-store-postgres(publisher and dependencies),deepagents(graph.py),openai-agents(handoffs/__init__.py,guardrail.py,tracing/processors.py),pydantic-ai-slim(_output.py,output.py,durable_exec/),claude-agent-sdk(_internal/transport/subprocess_cli.py),google-adk(agents/),agent-framework-core(_workflows/),dspy(teleprompt/). - LangChain docs: multi-agent guide and supervisor tutorial; Agent Server double-texting — https://docs.langchain.com/.
- Anthropic, Claude Code documentation (tools, hooks, permissions, sandboxing, sub-agents, skills) and engineering posts on agent tools and sandboxing — https://docs.claude.com/, https://www.anthropic.com/engineering.
- OpenAI, Codex CLI repository and docs (
AGENTS.md,apply_patch, sandbox modes) — https://github.com/openai/codex. - Cursor engineering blog posts on codebase indexing (Merkle-tree sync, path obfuscation) and semantic search for agents — https://cursor.com/blog.
- Aider documentation on repository maps and edit formats — https://aider.chat/docs/.
- OpenTelemetry GenAI semantic conventions — https://opentelemetry.io/docs/specs/semconv/gen-ai/.