Appendix I — LangChain 1.x in practice: create_agent, middleware and structured output, under the hood¶
Why this appendix exists. Chapters
07,20and21teach the theory: constrained decoding, the Runnable protocol, LCEL, StateGraph, checkpointing. They were written around the 0.x API and only mention the 1.x API in version notes. In 2026 almost all new LangChain code is three calls:init_chat_model,create_agentandresponse_format=. Those calls hide a graph, a middleware stack and a structured-output state machine. Their defaults decide whether your agent stops, retries, falls back or leaks data.This page is the practice layer. It shows what each modern call compiles into, read from the source, which defaults bite, and the best and worst practices a staff engineer should know before approving a design. It does not list use cases. It explains how the code behaves when things go wrong.
Read from source, not docs. Everything marked (source) was checked in
langchain==1.4.5,langchain-core==1.6.10,langgraph==1.2.15,langgraph-prebuilt==1.1.0(October 2026). Everything marked (probe) was run against fake chat models. The scripts are in §10, so you can re-check them after an upgrade. Defaults change between minor versions, so pin and re-probe (§9).Where the neighbouring detail lives: the Runnable protocol and the 0.x package split in
20-langchain-architecture-and-internals.md; StateGraph, reducers, checkpointers,interrupt()and stream modes in21-langgraph-deep-dive.md; how constrained decoding works and how to design schemas in07-generation-and-structured-output.md§4–§5; tool authorization and idempotency in24-tool-calling-and-enterprise-integration.md; retries and fallback as a gateway concern in23-multi-llm-model-gateway.md; compaction strategy in30-context-engineering-for-agents.md. The wider stack (packages, reference stacks, other frameworks) is in appendix J. Runtime controls, the Functional API, LangSmith, Deep Agents, the store and MCP adapters are in appendix K.
Contents¶
- The 2026 surface on one page
create_agent: what it compiles into- Middleware: six hooks, two kinds, one ordering rule
- The prebuilt middleware, and the defaults that bite
- Structured output: three paths and the state machine behind them
- Tools in 1.x:
ToolRuntime, context, state and store - Best practices: a production skeleton
- Worst practices: what fails in production, and why
- When not to use
create_agent, and version hygiene - Probe it yourself
- Interview questions
- Sources
1. The 2026 surface on one page¶
In plain words. LangChain 1.0 (October 2025) cut the
langchainpackage down to a few agent-building calls and moved everything else intolangchain-classic. If code importsAgentExecutor,LLMChain,ConversationBufferMemoryorcreate_react_agent, it is legacy code.Real-world example. A team copies a 2024 tutorial with
AgentExecutor(max_iterations=6)andConversationBufferMemory. It works in the demo. It has no persistence, no approval step, and the memory grows without limit. The 1.x version of the same agent is onecreate_agentcall with a checkpointer and two middleware.
| Job | Use in 2026 | Legacy (do not start new code with it) |
|---|---|---|
| Get a chat model | init_chat_model("provider:model") |
Importing a provider class everywhere |
| Agent loop | langchain.agents.create_agent |
AgentExecutor, create_tool_calling_agent, langgraph.prebuilt.create_react_agent (deprecated) |
| Change behaviour inside the loop | middleware=[...] (AgentMiddleware, @before_model, @wrap_model_call, …) |
Subclassing the executor, custom prompt templates with agent_scratchpad |
| Typed final answer from an agent | response_format=Schema / ToolStrategy / ProviderStrategy |
Parsing the last message with an output parser |
| Typed answer from one model call | model.with_structured_output(Schema) |
PydanticOutputParser + format instructions in the prompt |
| Conversation memory | checkpointer= + thread_id |
ConversationBufferMemory, RunnableWithMessageHistory |
| Cross-thread memory | store= (BaseStore) |
Custom vector "memory" classes |
| Per-request facts (tenant, user, scopes) | context_schema= + invoke(..., context=Ctx(...)) |
Stuffing them into config["configurable"] or the prompt |
| Tool access to state, context, store | runtime: ToolRuntime parameter |
InjectedState, InjectedStore (still work) |
| Message content across providers | msg.content_blocks (standard blocks: text, reasoning, tool_call, image, …) |
Branching on provider-specific content lists |
| Long-horizon "deep" agent (planning, files, sub-agents) | deepagents.create_deep_agent, which is create_agent plus a middleware stack (source: deepagents/graph.py returns create_agent(...)) |
Hand-rolled planner loops |
The key point: create_agent is not a new runtime. It is a function that builds a LangGraph
StateGraph and compiles it. Everything in 21 (checkpoints, interrupts, stream modes, subgraphs)
applies unchanged. What is new is the middleware layer and the structured-output logic, and that is
where the surprises are.
2. create_agent: what it compiles into¶
In plain words.
create_agentbuilds a graph with amodelnode, atoolsnode, and one extra node for eachbefore_*/after_*hook in your middleware. Conditional edges decide after every step whether to run tools, call the model again, or stop.Real-world example. You add six middleware and the trace shows only four new nodes. That is expected. Retry and fallback middleware do not become nodes. They run inside the
modelnode, so they never get their own checkpoint (§3.2).
2.1 The signature¶
create_agent(
model, # "provider:model" string → init_chat_model(), or a BaseChatModel
tools=None, # BaseTool | callable | dict (provider built-in tools, e.g. web search)
*,
system_prompt=None, # str | SystemMessage; prepended on every model call, not stored in state
middleware=(), # ordered; order matters (§3.3)
response_format=None, # Schema | ToolStrategy | ProviderStrategy (§5)
state_schema=None, # extend AgentState; prefer middleware.state_schema
context_schema=None, # type of the immutable per-run context (§6)
checkpointer=None, # per-thread persistence; required for interrupts / HITL
store=None, # cross-thread persistence
interrupt_before=None, interrupt_after=None,
debug=False, name=None, cache=None, transformers=None,
) -> CompiledStateGraph
2.2 The graph it builds (probe)¶
With ModelCallLimit, Summarization, HumanInTheLoop, ModelRetry, ModelFallback and
ToolCallLimit middleware (in that order) plus one tool and a response_format, the compiled graph has
exactly these nodes and edges:
__start__ ─► ModelCallLimit.before_model ─► Summarization.before_model ─► model
│ (jump: end) │
▼ ▼
__end__ ◄── ToolCallLimit.after_model ◄───────────────────────┘
│
▼
HumanInTheLoop.after_model ─► ModelCallLimit.after_model ─┬─► tools ─┬─► ModelCallLimit.before_model (loop)
│ └─► __end__ (return_direct / structured output)
├─► ModelCallLimit.before_model (loop)
└─► __end__
Notice three things.
ModelRetryandModelFallbackare not nodes.wrap_model_callhooks are composed into one function that runs inside themodelnode (source:_chain_model_call_handlers). The same holds forwrap_tool_callinsidetools.after_modelhooks run in reverse list order.ToolCallLimitwas last in the list and runs first after the model.before_*hooks run in list order (§3.3).- There is no
max_iterations. The loop exits only when the model stops calling tools, areturn_directtool runs, a structured response is parsed, or a middleware jumps toend.
2.3 The routing rules (source: _make_model_to_tools_edge, _make_tools_to_model_edge)¶
After the model (or after the last after_model hook):
| Order | Condition | Goes to |
|---|---|---|
| 1 | A middleware set jump_to |
model / tools / end as requested |
| 2 | No AIMessage in state (for example, history was cleared) |
end |
| 3 | Last AIMessage has no tool calls |
end (the normal exit) |
| 4 | There are tool calls with no ToolMessage yet, excluding structured-output tools |
one Send("tools", [call]) per call |
| 5 | A structured response was produced in this step | end |
| 6 | Tool calls exist but all are answered (a middleware injected ToolMessages) |
model |
After tools: end if every executed tool has return_direct=True or a structured-output tool
succeeded; otherwise back to the loop entry.
Rule 4 matters most. Each pending tool call becomes its own Send, so parallel tool calls run as
concurrent tasks in one LangGraph super-step. Two consequences:
- A model that emits
charge_cardtwice in one turn charges twice at the same time. Idempotency keys are required, not optional (§6.3). - Tools that return
Command(update=...)to the same state key in one turn conflict unless that key has a reducer (21§3.3).
2.4 Two quiet defaults (source)¶
recursion_limit=9_999.create_agentsets this with.with_config(...). A plain LangGraph graph defaults to 25 steps, which used to stop runaway agents. Here it does not. A runaway loop runs until your budget ends unless you addModelCallLimitMiddleware. Treat a call limit as required in production (§7).- Malformed tool calls are patched. Before every model call,
_patch_invalid_tool_callsadds aToolMessage(status="error", "...arguments were malformed or truncated")for eachinvalid_tool_call. A truncated JSON argument (often caused by hittingmax_tokens) therefore does not crash the run. The model sees the error and tries again. That is good, but it also hides the cause, so alert onstatus="error"tool messages, not only on exceptions.
3. Middleware: six hooks, two kinds, one ordering rule¶
In plain words. Middleware is the 1.x way to change what the agent does without writing a graph. Some hooks are nodes (they run between steps and can save to state). Others are wrappers (they surround the model or tool call and can retry, swap or short-circuit it).
Real-world example. Cutting old tool outputs from the prompt: do it in a wrapper and the checkpoint keeps the full history for audit, while the model sees a short prompt. Do it in a node and the history is deleted for good.
3.1 The six hooks¶
| Hook | Kind | Runs | Can do | Typical job |
|---|---|---|---|---|
before_agent |
node | once per invoke |
update state, jump | load user profile, validate input |
before_model |
node | before every model call | update state (persisted), jump to end |
trimming/summarizing history, call limits |
wrap_model_call |
wrapper | around every model call | change the request (model, tools, prompt, messages, response_format), call the handler 0..n times, return a different response | retry, fallback, dynamic model/tool selection, transient context edits, caching |
after_model |
node | after every model call | inspect tool_calls, update state, jump |
approval (HITL), guardrails on output, tool-call limits |
wrap_tool_call |
wrapper | around every tool call | change args, retry, return ToolMessage or Command without running the tool |
auth checks, tool retry, error-to-message, emulation in tests |
after_agent |
node | once at the end | update state | write-back to memory, audit record |
Decorator forms exist for one-off hooks (@before_model, @after_model, @wrap_model_call,
@wrap_tool_call, @dynamic_prompt). Use a class when the middleware has config, state fields or more
than one hook.
3.2 Node versus wrapper: what persists and what is checkpointed¶
This is the distinction most teams miss.
Node hook (before_*, after_*) |
Wrapper hook (wrap_*) |
|
|---|---|---|
| Return value | a state update (dict) → merged through reducers → checkpointed |
a response → only what the model/tool node returns is saved |
Editing messages |
permanent. SummarizationMiddleware returns RemoveMessage(REMOVE_ALL_MESSAGES) + summary + kept messages (source) |
transient. request.override(messages=...) changes only this call's prompt |
| Crash in the middle | resumes from the previous checkpoint, the hook runs again | the whole model/tools node runs again, including retries already made |
| Visible in the graph | yes, as "<Name>.before_model" |
no, it is inside model / tools |
| Can jump | yes, with @hook_config(can_jump_to=[...]) and {"jump_to": "end"} |
no; it shapes the response instead |
Rule of thumb: do transient edits in wrappers and permanent edits in nodes. If the audit trail must
keep every tool output (regulated domains, 29 §12), do not use a node hook that deletes messages. Use
ContextEditingMiddleware (a wrapper) to hide old tool results from the prompt instead.
Two mechanics to know:
ModelRequestis immutable in practice. Userequest.override(model=..., tools=...). Direct assignment such asrequest.tools = [...]still works but raisesDeprecationWarning(source:ModelRequest.__setattr__).- A wrapper that wants to write state returns
ExtendedModelResponse(model_response, command=Command(update={...})). The command is applied through reducers after the model node.gotois not supported there.
3.3 The ordering rule¶
middleware = [A, B, C]
before_agent / before_model : A → B → C (list order)
wrap_model_call / wrap_tool_call : A( B( C( call ) ) ) (first = outermost)
after_model / after_agent : C → B → A (reverse list order)
It is the same onion model as HTTP middleware. Order changes behaviour, not just style:
| You want | Order | Why |
|---|---|---|
| Retry each model several times, then fall back | [ModelFallback(...), ModelRetry(on_failure="error")] |
Fallback is outer, so it only sees an exception after the inner retries are used up |
| Retry the whole fallback chain | [ModelRetry(...), ModelFallback(...)] |
Retry is outer and re-runs primary + fallbacks |
Approve the final tool calls after other after_model checks |
put HumanInTheLoopMiddleware first in the list |
after_model runs in reverse, so the first entry runs last |
| Count tool calls before they are approved | put ToolCallLimitMiddleware after HITL in the list |
it then runs before HITL |
| Summarize after the call-limit check | [ModelCallLimit, Summarization, ...] |
do not pay for a summary on a call that will be refused |
Also: middleware names must be unique, and two instances of the same class raise
AssertionError("Please remove duplicate middleware instances.") unless you override .name
(source). For example, two ToolCallLimitMiddleware instances for two tools need distinct names.
3.4 State and context from middleware¶
state_schemaon a middleware class adds fields to the agent state. They are merged withAgentStateat compile time. Mark internal countersPrivateStateAttrso they stay out of the public input and output schemas, asModelCallLimitStatedoes.- Per-run counters use
UntrackedValue(not checkpointed). Per-thread counters are normal fields (checkpointed). That is howModelCallLimitMiddlewarekeepsrun_limitandthread_limitapart (source). runtime.contextis the typedcontext_schemaobject passed atinvoke(..., context=...). It is not saved in state or checkpoints. Use it for tenant, user and scopes (§6).
4. The prebuilt middleware, and the defaults that bite¶
In plain words. The prebuilt middleware save weeks of work, but each has a default chosen for demos. Read the defaults below before you ship.
Real-world example. A team adds
ModelRetryMiddlewareandModelFallbackMiddleware. In a provider outage, users get the reply "Model call failed after 3 attempts with APITimeoutError: …" as the agent's final answer, and the fallback model is never called. The cause is one default (on_failure="continue", below). (probe, §10)
| Middleware | Hook(s) | Default that bites (source) | Production setting |
|---|---|---|---|
ModelCallLimitMiddleware(thread_limit, run_limit, exit_behavior) |
before_model, after_model |
exit_behavior="end" stops with an AIMessage like "Model call limits exceeded: run limit (3/3)". structured_response is None (probe) |
always set run_limit; callers must handle a missing structured_response; use "error" in batch jobs so a failure is loud |
ToolCallLimitMiddleware(tool_name, thread_limit, run_limit, exit_behavior) |
after_model |
exit_behavior="continue" blocks extra calls with error ToolMessages and lets the model keep going |
per-tool limits on expensive or side-effecting tools; "end" for those |
ModelRetryMiddleware(max_retries=2, retry_on, on_failure, backoff_factor=2.0, initial_delay=1.0, max_delay=60, jitter=True) |
wrap_model_call |
on_failure="continue" turns the final exception into an AIMessage with the error text. The loop ends there (no tool calls), the user gets the error as an answer, and an outer fallback never runs. retry_on defaults to every Exception (except ModelError with is_retryable=False), so it also retries 400s and context-length errors |
on_failure="error"; retry_on limited to timeouts, 429, 5xx |
ModelFallbackMiddleware(first, *more) |
wrap_model_call |
catches every exception, including invalid requests that will fail on the fallback too | put it outside the retry; prefer one gateway (23) over per-agent fallback when you have more than one agent |
ToolRetryMiddleware(max_retries=2, tools=None, ...) |
wrap_tool_call |
tools=None means all tools, including non-idempotent writes |
list only read-only or idempotent tools |
ToolErrorMiddleware(on_error) |
wrap_tool_call |
opt-in: returning None re-raises |
return a short message naming the error type, not str(exc) (stack details, SQL, internal hosts) |
HumanInTheLoopMiddleware(interrupt_on={tool: True \| InterruptOnConfig}) |
after_model (+ wrap_tool_call) |
uses interrupt(), so without a checkpointer it cannot resume. Decisions: approve, edit, reject, respond |
durable checkpointer (Postgres), approvals keyed by tool_call_id, allowed_decisions as narrow as possible |
SummarizationMiddleware(model, trigger=None, keep=("messages", 20), trim_tokens_to_summarize=4000) |
before_model |
trigger=None never summarizes (it is a no-op). When it runs, it removes all messages from state and writes summary + kept tail (permanent). The rewritten prefix also invalidates the provider prompt cache |
set trigger=("fraction", 0.7) or ("tokens", N); use a cheap model; keep raw history elsewhere if you need audit |
ContextEditingMiddleware(edits=[ClearToolUsesEdit(...)]) |
wrap_model_call |
transient: state keeps everything, the prompt does not | preferred first step against context bloat (30 §8) |
PIIMiddleware(pii_type, strategy="redact", apply_to_input=True, apply_to_output=False, apply_to_tool_results=False) |
before_model, after_model |
only the user input is scanned by default. PII in tool results (CRM lookups, emails) goes to the model provider and the checkpoint unredacted | apply_to_tool_results=True for data-returning tools; regex detectors are a floor, not a DLP (17 §8) |
LLMToolSelectorMiddleware(model, max_tools, always_include) |
wrap_model_call |
one extra LLM call per model call; on_parsing_failure="error" |
only above roughly 30 tools; consider provider tool search (ProviderToolSearchMiddleware, 30 §3) |
TodoListMiddleware() |
wrap_model_call, after_model |
adds a write_todos tool and prompt text on every call |
long multi-step tasks only |
ShellToolMiddleware(execution_policy=...) |
before_agent, after_agent |
execution_policy=None means HostExecutionPolicy: commands run on your host |
DockerExecutionPolicy or a real sandbox (28 §7) |
LLMToolEmulator |
wrap_tool_call |
replaces real tools with LLM-made results | tests and evals only; fail the build if it is configured in prod |
5. Structured output: three paths and the state machine behind them¶
In plain words. There are three ways to get a typed object. Ask the provider to enforce a JSON schema (provider strategy). Give the model a fake tool whose arguments are the schema and force it to call a tool (tool strategy). Or, for one call with no loop, use
with_structured_output. They fail differently, retry differently, and leave different messages in history.Real-world example. A ticket-triage agent returns
Triage(priority: int 1–5). With the tool strategy, the model sendspriority=7. Pydantic rejects it, the agent sends the error back as aToolMessage, and the model fixes it on the next call. With the provider strategy, the provider's schema has nomaximum(some providers drop numeric bounds), the JSON parses, Pydantic rejects it, and the run raises. There is no retry on that path.
5.1 Which path you get¶
| You pass | What happens (source: create_agent, _get_bound_model) |
|---|---|
response_format=MySchema (raw type) |
wrapped in AutoStrategy. At each model call, if the model's profile says structured_output (or the model name matches a fallback regex list), it uses ProviderStrategy; otherwise ToolStrategy. Exception: Gemini models before the 3 series with tools fall back to ToolStrategy |
ToolStrategy(MySchema, handle_errors=True, tool_message_content=None) |
always the tool path |
ProviderStrategy(MySchema, strict=None) |
always the native path |
model.with_structured_output(MySchema, method=..., include_raw=False) |
one call, no loop. Base implementation: bind_tools([schema], tool_choice="any") + a tools parser. Partner packages override it and add method="json_schema" (native) next to "function_calling" |
Because auto-selection runs per call, a fallback middleware can silently switch strategies in the middle of a run: primary model with native output, fallback model with the tool path. The message history then looks different, and evals tuned on one shape can break on the other. Pin the strategy explicitly when you use fallbacks.
5.2 ToolStrategy, step by step (source + probe)¶
- At build time each schema (each member of a
Union) becomes aStructuredToolnamed after the class, with the class docstring as its description. - At each model call those tools are added to your real tools, and
tool_choice="any"is set. The model must call some tool on every turn. It can never answer in plain text. It ends the run by calling the schema tool. - If the model calls the schema tool and Pydantic validation passes, the result is
structured_response, plus aToolMessage("Returning structured response: ..."). The loop ends. - If validation fails and
handle_errorsallows it, aToolMessagewith"Error: {error}\n Please fix your mistakes."is added and the loop goes back to the model. If the model calls two schema tools at once, that is aMultipleStructuredOutputsError, handled the same way. - If the model calls real tools, they run, and the loop continues.
The transcript of a successful retry (probe):
HumanMessage 'rate'
AIMessage tool_calls=[Answer(score=42)]
ToolMessage "Error: Failed to parse structured output for tool 'Answer': ... less than or equal to 10 ... Please fix your mistakes."
AIMessage tool_calls=[Answer(score=7)]
ToolMessage 'Returning structured response: score=7'
→ structured_response = Answer(score=7)
What bites:
- The retries have no limit.
handle_errors=Trueloops until the model gets it right, andrecursion_limitis 9,999. A schema the model cannot satisfy (a contradictory validator, an enum the model never picks) burns tokens until something else stops it. (probe: withModelCallLimitMiddleware(run_limit=3)the run ends after 3 calls withstructured_response=None) - Validation errors are fed back to the model and saved in the checkpoint. Pydantic messages
include the bad
input_value. If a field can hold user PII, pass a callablehandle_errors=lambda e: "priority must be 1-5"that names the rule, not the value. - Raw JSON-schema
dicts are not validated. The tool arguments are returned as they are, andhandle_errorsdoes nothing. Use Pydantic, a dataclass or aTypedDict. Union[A, B]is how you model "answer or refuse". Each member becomes a separate tool, which is clearer to the model than an optional-fields-everywhere schema (07§5.4).- Forced tool choice changes the model's behaviour. Some reasoning models do worse with forced tool use, and some providers do not allow extended thinking together with forced tool choice. Check your provider before you combine them.
5.3 ProviderStrategy, step by step (source)¶
- The schema is sent as the provider's native
response_format(OpenAIjson_schemashape; partner packages translate it for other providers). For OpenAI-compatible models,strict=Trueis also set on the tools. - If the model returns tool calls, they run as normal. The schema applies only to the final text message, the one without tool calls.
- The text is
json.loads-ed and validated. Any failure raisesStructuredOutputValidationError. There is nohandle_errorsand no retry in the loop.
Why it is still the better default when available: constrained decoding makes structural failures
rare (07 §4), the model ends with a normal message instead of a forced tool call, and there is no
extra round trip. What it does not guarantee is semantic validity. Provider schema subsets drop
things like numeric ranges, regex patterns and custom validators, so Pydantic can still reject a
response that the provider accepted. Wrap the agent call in your own bounded retry, or use
ToolStrategy when the schema has strong validators.
5.4 Choosing¶
| Situation | Choose |
|---|---|
| One extraction call, no tools, no loop | with_structured_output(Schema, method="json_schema") |
| Agent with tools, provider has native structured output, schema is mostly structural | ProviderStrategy(Schema), with your own outer retry (max 1–2) |
| Strong validators (ranges, cross-field rules, lookups) or "answer | refuse" unions | ToolStrategy(Schema, handle_errors=<callable>) plus ModelCallLimitMiddleware |
| Fallbacks across providers | pin one strategy explicitly (§5.1) |
| You need the raw message for audit or debugging | include_raw=True on with_structured_output; on agents, the AIMessage is in messages |
| Streaming partial JSON to a UI | stream the messages mode and parse partial tool-call args (07 §10). structured_response arrives only at the end |
with_structured_output(..., include_raw=True) never raises on parse failure. It returns
{"raw": ..., "parsed": None, "parsing_error": exc} (source: .with_fallbacks(..., exception_key="parsing_error")).
Check parsing_error. Code that reads only ["parsed"] treats a failure as None and moves on.
6. Tools in 1.x: ToolRuntime, context, state and store¶
In plain words. A tool can ask for a
runtime: ToolRuntimeparameter. The model never sees it. Through it the tool gets the per-request context (who is calling), the agent state, the long-term store, its owntool_call_id, and a stream writer.Real-world example.
search_kb(query)must only search the caller's tenant. Iftenant_idis a tool argument, a prompt injection can change it. If it comes fromruntime.context, the model cannot touch it.
6.1 Three places for data¶
| Where | Lifetime | Saved in checkpoint | Who can change it | Put here |
|---|---|---|---|---|
runtime.context (context_schema) |
one invoke |
no | only your server code | tenant, user, scopes, request id, feature flags |
state (AgentState + middleware fields) |
one thread | yes | nodes, tools via Command(update=...) |
messages, counters, plan, intermediate results |
store (BaseStore) |
across threads | its own storage | tools, middleware | user preferences, long-term memory (25) |
Never put secrets or auth tokens in state. It is checkpointed, often traced, and sent back to clients
in stream output. Put identity in context and resolve credentials inside the tool (29).
6.2 Error behaviour (source: langgraph.prebuilt.tool_node)¶
The default handle_tool_errors returns only argument-validation errors
(ToolInvocationError) to the model. Any other exception raised by a tool ends the run. That is
a safe default, since internal errors are not shown to the model, but it means one flaky downstream
API kills the conversation. Choose per tool:
- expected business failures ("order not found") →
returna clear string, or raiseToolExceptionand handle it; - transient infrastructure failures →
ToolRetryMiddleware(tools=[...])for idempotent tools; - everything else →
ToolErrorMiddleware(on_error=...)returning a short message that names the error type, with the full error going to logs.
6.3 Idempotency for side-effecting tools¶
runtime.tool_call_id is stable for a given model output, and that output is checkpointed. After a
crash or an approval pause, the resumed run executes the same tool call id. Use it (with the thread
id) as the idempotency key for writes. That is what turns "at least once" into "effectively once"
(24 §8, 21 §5).
7. Best practices: a production skeleton¶
In plain words. The code below is a small, complete agent with each production concern handled once, in the layer where it belongs. Change the names, keep the structure.
from dataclasses import dataclass
from pydantic import BaseModel, Field
from langchain.agents import create_agent
from langchain.agents.structured_output import ToolStrategy
from langchain.agents.middleware import (
ContextEditingMiddleware, ClearToolUsesEdit, HumanInTheLoopMiddleware,
ModelCallLimitMiddleware, ModelFallbackMiddleware, ModelRetryMiddleware,
PIIMiddleware, ToolCallLimitMiddleware, ToolErrorMiddleware,
)
from langchain.tools import ToolRuntime, tool
from langgraph.checkpoint.postgres import PostgresSaver
@dataclass(frozen=True)
class Ctx: # per-request identity: never in state, never a tool arg
tenant_id: str
user_id: str
scopes: frozenset[str]
class Resolution(BaseModel):
"""Final answer to the customer."""
answer: str = Field(description="Reply to the customer, max 120 words.")
cited_doc_ids: list[str] = Field(description="IDs of KB documents used; empty if none.")
class Escalation(BaseModel):
"""Use when policy does not allow an answer or a human is needed."""
reason: str
@tool
def search_kb(query: str, runtime: ToolRuntime[Ctx]) -> str:
"""Search the help-center articles for this customer's product. Use for policy questions."""
hits = kb.search(query, tenant_id=runtime.context.tenant_id, k=8) # tenant from context
return render_hits(hits) # short, ID-tagged text
@tool
def issue_refund(order_id: str, amount_cents: int, runtime: ToolRuntime[Ctx]) -> str:
"""Refund part or all of an order. Requires human approval."""
if "refunds:write" not in runtime.context.scopes:
return "Not permitted for this user." # authz in code, not in prompt
key = f"{runtime.config['configurable']['thread_id']}:{runtime.tool_call_id}"
payments.refund(order_id, amount_cents, idempotency_key=key) # safe on resume and parallel calls
return f"Refund of {amount_cents} cents queued for {order_id}."
def tool_error_message(exc: Exception, request) -> str | None:
if isinstance(exc, (TimeoutError, ConnectionError)):
log.warning("tool failed", tool=request.tool_call["name"], exc_info=exc)
return f"{request.tool_call['name']} is temporarily unavailable ({type(exc).__name__})."
return None # unknown errors still stop the run
checkpointer = PostgresSaver(pool) # pool built from a DSN read from the vault;
checkpointer.setup() # run once at deploy, not per request
agent = create_agent(
model="anthropic:claude-sonnet-5-5",
tools=[search_kb, issue_refund],
system_prompt=SUPPORT_PROMPT, # static → cache-friendly prefix
context_schema=Ctx,
response_format=ToolStrategy(
Resolution | Escalation,
handle_errors=lambda e: "The response did not match the schema. Fix the fields and try again.",
),
middleware=[
HumanInTheLoopMiddleware( # first → its after_model runs last
interrupt_on={"issue_refund": {"allowed_decisions": ["approve", "reject"]}},
),
ModelCallLimitMiddleware(run_limit=12, exit_behavior="end"), # the loop's real stop condition
ToolCallLimitMiddleware(tool_name="issue_refund", run_limit=1, exit_behavior="end"),
ContextEditingMiddleware(edits=[ClearToolUsesEdit(trigger=60_000)]), # transient trim
PIIMiddleware("email", strategy="redact", apply_to_tool_results=True),
ModelFallbackMiddleware("openai:gpt-5.5"), # outer: only sees exhausted retries
ModelRetryMiddleware(max_retries=2, on_failure="error",
retry_on=(TimeoutError, RateLimitError, InternalServerError)),
ToolErrorMiddleware(on_error=tool_error_message),
],
checkpointer=checkpointer,
name="support_agent",
)
result = agent.invoke(
{"messages": [{"role": "user", "content": text}]},
config={"configurable": {"thread_id": f"{tenant_id}:{conversation_id}"}},
context=Ctx(tenant_id=tenant_id, user_id=user_id, scopes=scopes),
)
answer = result.get("structured_response") # None when a limit stopped the run → handle it
Why each line is there:
| Practice | Why (mechanism) |
|---|---|
Always set ModelCallLimitMiddleware(run_limit=...) |
recursion_limit is 9,999 (§2.4) and ToolStrategy retries without limit (§5.2) |
Read structured_response with .get() and handle None |
limits end the run with a plain AIMessage (§4) |
Retry inside, fallback outside, on_failure="error" |
otherwise the error text becomes the user's answer and the fallback never runs (§4, probe) |
Identity in context, not in tool arguments or state |
the model cannot forge it, and it is never checkpointed (§6.1) |
| Authorization inside the tool, approval as middleware | the prompt is not a security boundary (17, 28); HITL adds a human, not authz |
Idempotency key = thread id + tool_call_id |
parallel Sends and resumes repeat calls (§2.3, §6.3) |
| Durable checkpointer whenever HITL or multi-turn | interrupt() needs it; InMemorySaver loses approvals on restart |
thread_id includes the tenant |
thread ids are the only key into checkpoints; a guessable id means a cross-tenant read |
| Transient context edits before summarization | keeps audit history and the prompt cache prefix (§3.2, §4) |
| Static system prompt, dynamic parts at the end | the provider prompt cache matches on prefix (30 §10). @dynamic_prompt that changes per call invalidates it |
Narrow retry_on |
retrying a 400 or a context-length error only multiplies cost |
| Pin strategy when using fallbacks | auto-selection is per model (§5.1) |
| Name the agent | name tags the model output and the LangSmith metadata (lc_agent_name), which you need to tell sub-agents apart in traces |
Test with GenericFakeChatModel + LLMToolEmulator, eval with real models |
unit tests check graph wiring and middleware order cheaply; behaviour needs evals (14) |
8. Worst practices: what fails in production, and why¶
In plain words. Each row is a pattern that works in a demo and fails under load, under attack or during an outage. The "mechanism" column is the reason in the code, so you can argue it in a review.
| Worst practice | Mechanism | Do instead |
|---|---|---|
| No call limit "because the model will stop" | recursion_limit=9_999; a confused model or an unsatisfiable schema loops |
ModelCallLimitMiddleware on every agent; alert when it fires |
ModelRetryMiddleware() with defaults |
on_failure="continue" returns the error as the final AIMessage; it retries all exceptions |
on_failure="error", narrow retry_on |
| Retry and fallback in every agent, and in the gateway too | retries multiply: 3 × 3 × 3 = 27 calls per user request in an outage | one retry layer (the gateway, 23); agents fail fast |
ToolRetryMiddleware() over all tools |
tools=None retries writes; the payment may already have gone through |
list idempotent tools only; idempotency keys for the rest |
response_format=dict_schema |
dicts are not validated; handle_errors does nothing |
Pydantic / dataclass / TypedDict |
| Pydantic validators that need I/O or are very strict | each failure is a full model round trip; with no limit it is unbounded | cheap structural validation in the schema; business checks after the agent, with your own bounded retry |
tenant_id / user_id as tool parameters |
the model fills them, so a prompt injection can change them | ToolRuntime.context |
| Secrets or tokens in state | state is checkpointed, traced and streamed | resolve credentials inside the tool from the vault |
SummarizationMiddleware in a regulated workflow |
deletes original messages from state (REMOVE_ALL_MESSAGES) |
ContextEditingMiddleware (transient) plus an append-only audit log |
SummarizationMiddleware(model) with no trigger |
trigger=None never runs; context grows until the provider rejects it |
set trigger |
PIIMiddleware and assuming the data is clean |
default scans user input only | apply_to_tool_results=True; real DLP for regulated data |
InMemorySaver in production with HITL |
approvals and paused runs are lost on deploy or restart | PostgresSaver / RedisSaver, with a retention policy (21 §5.7) |
| One giant agent with 80 tools | tool definitions fill the context; selection accuracy drops | sub-agents as tools, tool search, or LLMToolSelectorMiddleware (30 §3, §6) |
| Business rules in the system prompt ("never refund over $500") | the prompt is a suggestion; injection overrides it | enforce in the tool or in a wrap_tool_call policy |
| Changing middleware order "for readability" | order is semantics (§3.3) | treat order as config under test: assert the compiled graph in a unit test |
A @dynamic_prompt that adds the current time or user data at the top |
every call misses the provider prompt cache | keep the prefix static, add dynamic parts as the last message |
Reading result["structured_response"] |
KeyError/None when a limit or interrupt stopped the run |
.get(), and check __interrupt__ in the result |
include_raw=True and reading only parsed |
parse failures become None silently |
check parsing_error |
Unpinned langchain / langgraph |
defaults and behaviour change in minor versions (deprecations in §9) | pin exact versions; re-run §10 probes in CI on upgrade |
Using create_agent for a fixed pipeline |
a loop where a sequence would do adds cost, latency and variance | plain code or LCEL for workflows (22 §1) |
9. When not to use create_agent, and version hygiene¶
9.1 Choosing the level¶
| Situation | Use |
|---|---|
| Fixed steps (retrieve → rerank → generate) | plain functions or LCEL. Not an agent |
| One model call that returns a typed object | with_structured_output |
| Tool loop, standard shape, needs limits/approval/retry | create_agent + middleware |
| Custom control flow: per-tool branches, parallel branches with different failure handling, map-reduce, a planner and an executor with different models | your own StateGraph (21), possibly with create_agent sub-graphs as nodes |
| Long tasks with planning, a file system and sub-agents | deepagents (built on create_agent) or your own graph |
| Work that must survive days, with strict exactly-once side effects across systems | a durable workflow engine (Temporal) calling agents as activities (21 §14) |
| You only use one provider and a 40-line loop is enough | the provider SDK directly. The framework is optional (20 §13) |
Rule: reach for middleware first, for a custom graph second. If you need three custom middleware that jump between nodes to express your flow, the flow is a graph. Write it as one.
9.2 Version hygiene¶
- Pin exact versions of
langchain,langchain-core,langgraph,langgraph-checkpoint-*and every partner package. They are released separately and their combinations matter. - Deprecations to grep for on upgrade:
create_react_agent,AgentExecutor, direct assignment onModelRequest,SummarizationMiddleware(max_tokens_before_summary=...)(nowtrigger=),request.system_prompt(nowsystem_message). - Checkpoint compatibility is a data migration. Checkpoints are serialized state. Adding a required field to the state schema or renaming middleware (node names change) can break resuming threads that were paused before the deploy. Test "resume a thread paused on the old version" in CI.
- Read the source when a default matters. The code is short:
python -c "import langchain, pathlib; print(pathlib.Path(langchain.__file__).parent)"
# then read: agents/factory.py (create_agent, edges), agents/structured_output.py,
# agents/middleware/types.py (hooks), agents/middleware/<name>.py (defaults)
10. Probe it yourself¶
In plain words. These scripts use fake chat models, so they cost nothing and need no API key. Run them after every upgrade. If an output changes, a default changed.
Probe A: the compiled graph and its config.
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage
from langchain.agents import create_agent
from langchain.agents.middleware import (HumanInTheLoopMiddleware, ModelCallLimitMiddleware,
ModelFallbackMiddleware, ModelRetryMiddleware, SummarizationMiddleware, ToolCallLimitMiddleware)
from langchain.tools import tool
from pydantic import BaseModel
class Fake(GenericFakeChatModel):
def bind_tools(self, tools, **kw):
return self
@tool
def search(q: str) -> str:
"""Search docs."""
return "result"
class Answer(BaseModel):
text: str
m = Fake(messages=iter([AIMessage("hi")]))
agent = create_agent(m, [search], response_format=Answer, middleware=[
ModelCallLimitMiddleware(run_limit=8), SummarizationMiddleware(m, trigger=("tokens", 4000)),
HumanInTheLoopMiddleware({"search": True}), ModelRetryMiddleware(), ModelFallbackMiddleware(m),
ToolCallLimitMiddleware(run_limit=5),
])
for e in agent.get_graph().edges:
print(e.source, "->", e.target, "(cond)" if e.conditional else "")
print(agent.config) # {'recursion_limit': 9999, ...}
Expected: no node for ModelRetry or ModelFallback; model -> ToolCallLimitMiddleware.after_model
(reverse order); recursion_limit: 9999.
Probe B: ToolStrategy retry and a limit stopping it.
from langchain.agents.structured_output import ToolStrategy
from pydantic import Field
class Score(BaseModel):
"""Final answer."""
score: int = Field(ge=0, le=10)
bad = AIMessage("", tool_calls=[{"name": "Score", "args": {"score": 42}, "id": "c1"}])
good = AIMessage("", tool_calls=[{"name": "Score", "args": {"score": 7}, "id": "c2"}])
a = create_agent(Fake(messages=iter([bad, good])), [], response_format=ToolStrategy(Score))
print(a.invoke({"messages": [{"role": "user", "content": "rate"}]})["structured_response"]) # score=7
b = create_agent(Fake(messages=iter([bad] * 50)), [], response_format=ToolStrategy(Score),
middleware=[ModelCallLimitMiddleware(run_limit=3)])
out = b.invoke({"messages": [{"role": "user", "content": "rate"}]})
print(out["messages"][-1].content, out.get("structured_response"))
# 'Model call limits exceeded: run limit (3/3)' None
Probe C: retry default hides the fallback.
class Down(GenericFakeChatModel):
def _generate(self, *a, **k):
raise TimeoutError("primary down")
backup = GenericFakeChatModel(messages=iter([AIMessage("from backup")] * 5))
for on_failure in ("continue", "error"):
agent = create_agent(Down(messages=iter([])), [], middleware=[
ModelFallbackMiddleware(backup),
ModelRetryMiddleware(max_retries=1, initial_delay=0, jitter=False, on_failure=on_failure),
])
print(on_failure, agent.invoke({"messages": [{"role": "user", "content": "hi"}]})["messages"][-1].content)
# continue → 'Model call failed after 2 attempts with TimeoutError: primary down'
# error → 'from backup'
Turn the probes into tests. Put probe A's edge list in a snapshot test. Middleware order then becomes a reviewed change, not an accident.
11. Interview questions¶
Q1. What does create_agent return, and how is it different from AgentExecutor?
A compiled LangGraph StateGraph: a model node, a tools node, a node for each before_*/after_*
middleware hook, and conditional edges. Wrapper hooks run inside the model and tools nodes.
Unlike AgentExecutor, it is checkpointable, can be interrupted, streams, and can be extended
through middleware, and it can be used as a sub-graph.
Q2. Your agent ran 400 model calls on one request. Why didn't LangGraph stop it?
create_agent sets recursion_limit=9_999. The loop only ends when the model stops calling tools, a
structured response parses, or middleware jumps to end. Fix: ModelCallLimitMiddleware(run_limit=...),
an alert when it fires, and check for an unsatisfiable ToolStrategy schema.
Q3. During a provider outage users received "Model call failed after 3 attempts…" as an answer, and
the fallback model was never used. Explain.
ModelRetryMiddleware defaults to on_failure="continue". It turns the final exception into an
AIMessage, which has no tool calls, so the loop ends and the fallback (outer wrapper) sees a success.
Use on_failure="error" with fallback outside retry.
Q4. ToolStrategy vs ProviderStrategy: how does each fail?
ToolStrategy forces tool_choice="any", validates the schema-tool arguments, and on failure sends
the error back as a ToolMessage and retries without limit. ProviderStrategy uses native
constrained decoding on the final message and raises StructuredOutputValidationError on any parse or
validation failure, with no in-loop retry. A raw schema chooses between them per model call.
Q5. Where should you trim conversation history: before_model or wrap_model_call?
before_model changes are saved to state (permanent, checkpointed). wrap_model_call changes affect
only that call's prompt. Trim transiently in a wrapper when you need the full history for audit or
for later turns. Rewrite state in a node only when you want the history itself to shrink.
Q6. Three middleware [A, B, C] all implement before_model, wrap_model_call and after_model.
In what order do they run?
before_model: A, B, C. wrap_model_call: A wraps B wraps C wraps the call. after_model: C, B, A.
Q7. How do you stop a prompt injection from making search_kb read another tenant's data?
Do not let the model supply the tenant. Pass it in context_schema at invoke, read
runtime.context.tenant_id in the tool, and enforce the filter in the retrieval layer (04 §12).
Q8. A model emits the same transfer_funds call twice in one turn. What happens, and what protects
you?
Each pending tool call becomes a separate Send, so both run concurrently in one super-step. Protect
with an idempotency key (thread id + tool_call_id or a business key), a
ToolCallLimitMiddleware(tool_name="transfer_funds", run_limit=1), and HITL approval.
Q9. Why can adding ModelFallbackMiddleware change the shape of your message history?
With a raw schema, the strategy is chosen per model. A primary model with native structured output and
a fallback without it produce different transcripts (final text vs a forced tool call plus a
ToolMessage). Pin the strategy.
Q10. What breaks when you deploy a middleware rename while threads are paused on approval? Node names come from middleware names. The checkpoint records the next node to run. A paused thread may point to a node that no longer exists, or state may lack a newly required field. Treat state schema and middleware names as a versioned contract, and test resuming old checkpoints.
12. Sources¶
langchain1.4.5 source:langchain/agents/factory.py(create_agent,_get_bound_model,_handle_model_output,_chain_model_call_handlers, edge builders,recursion_limit),langchain/agents/structured_output.py,langchain/agents/middleware/types.py, and each file underlangchain/agents/middleware/. Repository: https://github.com/langchain-ai/langchain (libs/langchain_v1).langgraph1.2.15 /langgraph-prebuilt1.1.0 source:langgraph/prebuilt/tool_node.py(ToolNode,ToolRuntime,_default_handle_tool_errors). Repository: https://github.com/langchain-ai/langgraph. Therecursion_limitchange references langgraph issue 7313.langchain-core1.6.10:BaseChatModel.with_structured_outputinlangchain_core/language_models/chat_models.py.deepagents0.7.25:deepagents/graph.py(create_deep_agentreturnscreate_agent(...)).- LangChain docs: Agents, Middleware, Structured output, Tools, Short-term memory — https://docs.langchain.com/oss/python/langchain/.
- LangChain 1.0 release notes and migration guide (October 2025): https://docs.langchain.com/oss/python/migrate/langchain-v1.