Skip to content

Agent orchestration

System Design Task: Autonomous Agent Orchestration Platform

Problem Statement

Design a multi-agent orchestration platform that enables autonomous AI agents to plan, execute multi-step tasks, use tools, collaborate with other agents, and self-correct on failure — reliably, at scale, and with human oversight.

The current state of AI agents is fragile: a single LLM call with a list of tools works for simple tasks, but falls apart on complex, multi-step workflows. The agent gets stuck in loops, forgets what it already tried, burns through tokens re-deriving context, fails silently when a tool errors, and has no mechanism for a human to inspect or redirect it mid-task. When you chain multiple agents together (a planner that decomposes a task, a coder that writes code, a reviewer that checks it), the coordination is ad-hoc: passing a string between Python functions, with no shared memory, no structured handoff, and no way to recover when one agent's output is garbage.

The platform fixes this by providing a structured runtime for agent execution that handles: task decomposition and planning, tool execution with error recovery, memory management (short-term and long-term), multi-agent coordination, human-in-the-loop checkpoints, cost and resource budgets, and observability — so that builders focus on what their agent should do, not how to keep it from failing.

This is infrastructure for a company where 50+ teams are building agent-powered features: customer support agents, coding assistants, data analysis agents, workflow automation agents, and research agents. Each runs on the same orchestration platform, sharing tool registries, memory systems, and observability.

Functional Requirements

  1. Agent Definition and Configuration

  2. An agent is defined by:

    • System prompt: the agent's identity, role, constraints, and behavioral guidelines.
    • Model configuration: which LLM (model ID, provider, temperature, max tokens), with fallback options.
    • Tool set: the tools the agent is allowed to use (from the tool registry), with per-tool permissions and rate limits.
    • Memory configuration: what memory stores the agent has access to (conversation history, working memory, long-term knowledge base).
    • Budget: maximum tokens (input + output), maximum tool calls, maximum wall-clock time, and maximum dollar cost per task.
    • Human-in-the-loop policy: which actions require human approval before execution (e.g., sending emails, modifying production data, spending money), and what happens when the human is unavailable (queue, timeout, abort).

  3. Agent definitions are versioned and deployable — like model versions in a model registry.

  4. Task Planning and Decomposition

  5. For complex tasks, the agent (or a dedicated planner agent) produces a structured plan: an ordered or partially-ordered set of steps, each with an objective, expected output, and success criteria.

  6. Dynamic replanning: when a step fails or produces unexpected output, the agent revises the plan rather than blindly continuing.
  7. Plan approval: for high-stakes tasks, the plan is presented to a human for approval before execution begins.

  8. Sub-task delegation: a step in the plan can be delegated to a different, specialized agent (e.g., the planner delegates "write the database migration" to a coding agent).

  9. Tool Execution Framework

  10. Tool registry: a catalog of available tools (APIs, code execution sandboxes, database queries, file system operations, web search, browser automation), each with:

    • A typed schema (input/output JSON schema).
    • A description (consumed by the LLM for tool selection).
    • Execution mode: synchronous (< 30s), asynchronous (minutes to hours), or streaming.
    • Side-effect classification: read-only, write (reversible), write (irreversible).
    • Rate limits and cost.
  11. Sandboxed execution: tool calls run in isolated environments (containers, VMs) so a misbehaving tool or agent-generated code cannot affect other agents or the host system.
  12. Error handling: when a tool call fails, the platform provides the agent with the error type, message, and suggested retry strategy — not just an opaque failure. The agent can retry with modified input, use an alternative tool, or escalate to the human.
  13. Tool result caching: identical tool calls (same tool + same input) within a configurable window return cached results to save cost and latency.

  14. Confirmation gates: for tools classified as "write (irreversible)" or above a cost threshold, require human confirmation before execution.

  15. Memory and Context Management

  16. Conversation history: the full sequence of messages (user, assistant, tool calls/results) for the current task, managed as a sliding window with summarization to stay within the model's context window.

  17. Working memory: structured, mutable state for the current task — a scratchpad where the agent stores intermediate results, extracted data, and current plan state. Persisted across LLM calls but scoped to the task.
  18. Long-term memory: a persistent knowledge base (vector store + keyword index) where the agent stores and retrieves:
    • Facts learned from previous tasks.
    • User preferences and context.
    • Tool usage patterns and past mistakes (to avoid repeating them).

  19. Context window management: when the conversation + working memory + tool results exceed the model's context window, the platform automatically:

    • Summarizes older conversation turns.
    • Compresses tool results (keep conclusions, drop raw data).
    • Pages out less-relevant working memory to the long-term store.
    • Maintains a "pinned" section for critical context that must never be evicted (system prompt, current plan, active constraints).

  20. Multi-Agent Coordination

  21. Agent-to-agent communication: agents can send structured messages to each other (request, response, notification) through a message bus, not by embedding one agent's output into another's prompt as a string.

  22. Coordination patterns:
    • Sequential pipeline: agent A's output feeds agent B's input (planner → coder → reviewer).
    • Parallel fan-out: a coordinator agent dispatches sub-tasks to multiple specialist agents in parallel, collects results, and synthesizes.
    • Hierarchical delegation: a manager agent delegates to worker agents, monitors their progress, and intervenes if they're stuck.
    • Debate / adversarial: two agents argue opposing positions, and a judge agent (or human) selects the better output.
  23. Shared state: agents on the same task can read/write a shared working memory (with conflict resolution for concurrent writes).

  24. Handoff protocol: when one agent hands off to another, the handoff includes: task context summary, what's been accomplished, what's remaining, and any constraints or warnings — not the entire conversation history.

  25. Human-in-the-Loop

  26. Approval gates: configurable points where execution pauses for human review (before irreversible actions, after planning, at task completion).

  27. Intervention: a human can pause a running agent, inspect its state (current plan, working memory, conversation history), modify the plan or context, and resume — or abort and take over manually.
  28. Escalation: the agent can explicitly escalate to a human when it's stuck, uncertain, or the task exceeds its capabilities, providing a structured summary of what it tried and where it's blocked.

  29. Feedback loop: human corrections (rejecting a tool call, modifying an output, choosing between options) are fed back into the agent's context and optionally into long-term memory for future improvement.

  30. Reliability and Error Recovery

  31. Retry with backoff: transient tool failures are retried automatically with exponential backoff and jitter.

  32. Fallback strategies: if a tool is unavailable, the agent can use an alternative tool or approach (defined in the agent config or decided by the LLM).
  33. Loop detection: detect when the agent is stuck in a retry loop or repeating the same failed approach, and force a strategy change or escalate.
  34. Checkpoint and resume: long-running agent tasks are checkpointed periodically (conversation state, working memory, plan progress) so they can resume after platform restarts or model provider outages.
  35. Timeout enforcement: per-step and per-task timeouts, with graceful shutdown (save state, notify human) rather than hard kill.

  36. Idempotency tracking: track which tool calls have been executed successfully, so a resumed task doesn't re-execute side-effecting operations.

  37. Cost and Resource Management

  38. Token budgets: per-task and per-agent limits on input + output tokens, enforced in real time. When approaching the limit, the platform compresses context aggressively; when exceeded, the task is paused and escalated.

  39. Tool call budgets: maximum number of tool calls per task (to prevent runaway agents).
  40. Dollar cost tracking: real-time cost accumulation per task (LLM tokens + tool execution costs), with alerts at configurable thresholds.
  41. Concurrency limits: maximum concurrent agents per team, to prevent resource starvation.

  42. Priority queues: high-priority tasks (customer-facing, time- sensitive) get resources before batch/background tasks.

  43. Observability

  44. Execution trace: a complete, structured log of every LLM call (input/ output, latency, tokens, cost), tool call (input/output, latency, success/failure), plan update, and memory operation — viewable as a timeline or tree.

  45. Metrics: task success rate, average steps to completion, token efficiency (tokens per successful task), tool call success rate, human intervention rate, cost per task.
  46. Live inspection: view a running agent's current state (conversation, plan, working memory) in real time — the debugging equivalent of a debugger attached to a running process.
  47. Replay and debugging: replay a failed task step-by-step, modifying inputs at any point to see how the agent would have behaved differently.
  48. Alerting: stuck agents (no progress for > N minutes), runaway agents (cost exceeding budget), and high error rates.

Non-Functional Requirements

  1. Scale

  2. 10,000 concurrent agent tasks across the platform.

  3. 50,000 tool calls/minute aggregate.
  4. 100+ agent definitions deployed, serving 50+ teams.

  5. Long-term memory: 100M+ entries across all agents.

  6. Latency

  7. Orchestration overhead (routing, context assembly, tool dispatch): ≤ 50 ms per step, excluding LLM and tool execution time.

  8. Tool dispatch (from agent deciding to call a tool to the tool beginning execution): ≤ 100 ms.
  9. Memory retrieval (long-term memory lookup): P99 ≤ 50 ms.

  10. Human notification (from agent requesting approval to human seeing the request): ≤ 5 seconds (push notification + webhook).

  11. Reliability

  12. Task completion rate ≥ 85% for well-defined tasks with appropriate tools (the rest are escalated to humans, not silently failed).

  13. Zero lost tasks: every task either completes, is explicitly failed with a reason, or is escalated to a human. No task silently disappears.

  14. Checkpoint durability: a platform crash must not lose more than the last 60 seconds of agent progress.

  15. Safety

  16. Blast radius containment: a misbehaving agent (infinite loop, excessive tool calls, attempting unauthorized actions) must not affect other agents or the platform itself.

  17. Permission enforcement: an agent cannot use tools or access data beyond what its definition and the task's authorization scope allow, regardless of what the LLM generates.
  18. Prompt injection resistance: tool results and external data are treated as untrusted input; the platform's control logic (plan execution, budget enforcement, permission checks) is not influenced by content in LLM responses or tool outputs.

Constraints and Assumptions

  • LLMs are accessed via an internal LLM Gateway (your existing design); the orchestration platform calls the gateway, not providers directly.
  • Tools are registered by individual teams and may have their own availability and latency characteristics; the orchestration platform does not control tool uptime.
  • Assume a Kubernetes-based deployment with standard cloud infrastructure.
  • Human-in-the-loop operates via a web UI and push notifications; assume humans respond within minutes for synchronous approval, hours for async review.
  • Not in scope: building the LLM itself or the tools. The platform orchestrates existing LLMs and existing tools.
  • Not in scope: fine-tuning or RLHF from agent interactions (but the observability data should be exportable for offline analysis and model improvement).

What You Should Deliver

  1. Requirement clarification and explicit assumptions.
  2. High-level architecture: orchestration engine, tool execution layer, memory system, multi-agent coordination bus, and human-in-the-loop interface.
  3. Agent definition language and versioning model.
  4. Task planning and decomposition: how plans are generated, validated, executed, and dynamically revised.
  5. Tool execution framework: sandboxing, error handling, confirmation gates, caching, and async tool support.
  6. Memory architecture: conversation management, working memory, long-term memory, and context window management with eviction/summarization.
  7. Multi-agent coordination: message bus, coordination patterns, shared state, and handoff protocol.
  8. Human-in-the-loop: approval flow, intervention, escalation, and feedback integration.
  9. Reliability: checkpoint/resume, loop detection, idempotency, and timeout enforcement.
  10. Cost management: token budgets, real-time cost tracking, and resource allocation.
  11. Observability: execution traces, live inspection, replay/debugging, and alerting.
  12. Safety and security: permission model, blast radius containment, prompt injection defenses.
  13. Capacity estimates with arithmetic: memory store sizing, message bus throughput, concurrent agent resource requirements.
  14. Failure walkthroughs: LLM provider outage mid-task, tool returning malicious content, agent stuck in a loop, and a multi-agent deadlock.
  15. Trade-offs: autonomy vs. safety, context window utilization vs. cost, plan rigidity vs. adaptability, and agent specialization vs. generality.

Expectations

  • The memory/context management is the hard part. Show exactly how a 20-step task stays coherent when the conversation exceeds the context window — the summarization strategy, what's pinned, what's evicted, and how the agent "remembers" step 3's result at step 18.
  • Multi-agent coordination must be concrete. Don't just say "agents communicate" — show the message format, the routing, the shared state model, and how a dead agent is detected and replaced.
  • Safety is not optional. Show how a compromised tool result (prompt injection in tool output) cannot escalate the agent's permissions or bypass budget limits.
  • Do the arithmetic. Token costs per multi-step task, memory store sizes, concurrent agent resource consumption.
  • Name concrete mechanisms — ReAct loop, chain-of-thought planning, tool- use structured outputs, vector store for long-term memory, dead-letter queue for failed tasks, circuit breaker for flaky tools.
  • Prefer a design that makes agents debuggable and predictable over one that maximizes autonomy.

Interview Kit

Read first: Solution · Agent patterns · LangGraph · Guardrails §4.7

Curveballs. The interviewer changes one thing mid-design. The hint in italics is what a strong answer reaches for:

  1. An agent reads an email containing injected instructions and has a "send payment" tool. (Rule of Two, human approval for irreversible actions, capability limits.)
  2. A 40-step run crashes at step 31. What is replayed, and what must not run twice? (Durable execution, idempotent tools.)
  3. Two agents deadlock waiting on each other's output. (Timeouts, supervisor, step budgets.)
  4. Costs per task vary 50×. How do you bound them? (Step, token and time budgets per run.)

Must answer (security, privacy, operations):

  • Per-run credentials scoped to the task, not the agent's owner
  • Audit trail of every tool call with inputs and outputs, and its retention

Phase it (MVP → Growth → Scale): MVP: a single agent with a fixed tool list and step budget, with runs stored in Postgres. Growth: durable workflows, human-in-the-loop approvals, evals. Scale: multi-agent supervision, cost controls, a tool marketplace.

Score yourself with the rubric.