Multi-agent AI systems promise to handle complex business workflows autonomously — routing tasks between specialized agents, recovering from failures, and adapting to new inputs without human intervention. Most production implementations fail not because the AI models are wrong, but because the orchestration architecture wasn't designed for real-world conditions: partial failures, LLM latency variance, token budget management, and state consistency.
Here's how we structure multi-agent systems that actually run in production.
The Architecture Problem Nobody Talks About
Single-agent prototypes work fine. You send a prompt, get a response, maybe chain a few tool calls. The moment you need agents coordinating across a workflow — a classifier routing to specialized processors, a reviewer checking outputs before committing — you hit a new class of problems:
- Which agent's output is authoritative when they disagree?
- What happens when agent B times out mid-workflow?
- How do you replay from a checkpoint without re-running expensive API calls?
- How do you explain what happened when a workflow fails at step 7 of 12?
These are distributed systems problems dressed in AI clothing. Solving them requires familiar distributed systems patterns applied to an unfamiliar substrate.
Orchestrator-Worker vs. Peer-to-Peer Architectures
Two primary patterns dominate production multi-agent systems:
Orchestrator-Worker (Recommended for most use cases)
A central orchestrator receives the workflow trigger, maintains workflow state, and delegates specific subtasks to specialized worker agents. Workers report back to the orchestrator; they don't communicate directly with each other.
class WorkflowOrchestrator:
def __init__(self, state_store: StateStore):
self.state = state_store
self.agents = {
"classifier": ClassifierAgent(),
"extractor": ExtractionAgent(),
"validator": ValidationAgent(),
"writer": OutputAgent(),
}
async def run(self, workflow_id: str, payload: dict) -> WorkflowResult:
state = await self.state.get_or_create(workflow_id, payload)
while state.current_step != "complete":
agent = self.agents[state.current_step]
try:
result = await agent.execute(state.context)
state = await self.state.advance(workflow_id, result)
except AgentError as e:
state = await self.state.handle_failure(workflow_id, e)
if state.retry_count > MAX_RETRIES:
raise WorkflowFailed(workflow_id, state)
return state.final_result
This pattern gives you a single source of truth (the orchestrator's state store), clear audit trail, and simple replay logic.
Peer-to-Peer (for high-throughput parallel processing)
Agents consume from a shared queue and publish results to downstream queues. Better throughput, harder to reason about and debug. Use this when you have genuinely parallel processing requirements, not just because it sounds more sophisticated.
State Management Is the Hard Part
Every production multi-agent system needs durable state management. "Durable" means the workflow survives: - LLM API failures and timeouts - Your application restarting - Individual agent crashes mid-execution
Minimum viable state schema:
@dataclass
class WorkflowState:
workflow_id: str
status: Literal["pending", "running", "failed", "complete"]
current_step: str
retry_count: int
context: dict # accumulated outputs from each step
error_log: list[StepError]
created_at: datetime
updated_at: datetime
# Store in Postgres, Redis with AOF, or DynamoDB — not in memory
The context dict accumulates outputs from each step and is what you checkpoint. When replaying a failed workflow, you restore context from the last successful checkpoint rather than re-running everything from step 1.
Handling LLM Failure Modes
LLMs fail in ways that differ from typical service failures. You need specific handling for:
Malformed output (LLM returns invalid JSON/format): Use structured output validation with retry logic. Retry with an increasingly explicit prompt clarification — second attempt says "Your previous response was malformed JSON. Respond ONLY with valid JSON matching this schema: [schema]."
Hallucinated tool calls: LLMs sometimes invoke tools with fabricated parameters. Validate all tool call arguments before execution. Return structured errors back to the LLM with specific correction instructions rather than silently failing.
Context window overflow: Long-running workflows accumulate context. Implement context summarization strategies for workflows with many steps — compress early steps into summaries once they're confirmed complete.
Rate limit handling:
async def call_llm_with_backoff(
client: AsyncAnthropic,
messages: list,
max_retries: int = 3,
) -> Message:
for attempt in range(max_retries):
try:
return await client.messages.create(
model="claude-opus-4-5",
max_tokens=4096,
messages=messages,
)
except anthropic.RateLimitError:
wait = (2 ** attempt) + random.random()
await asyncio.sleep(wait)
raise MaxRetriesExceeded()
Observability: What You Can't Skip
Multi-agent workflows are opaque without deliberate observability. Minimum instrumentation:
Structured logging per step:
logger.info("agent_step_complete", extra={
"workflow_id": workflow_id,
"step": step_name,
"agent": agent_class,
"input_tokens": usage.input_tokens,
"output_tokens": usage.output_tokens,
"latency_ms": elapsed_ms,
"retry_count": retries,
})
Token budget tracking: Track tokens per step, per workflow, and per workflow type. Token costs compound across multi-agent workflows — a 12-step workflow might use 50,000 tokens per run at scale. Cost surprises are a common reason production AI workflows get shut down.
Step-level latency traces: Which agent is your bottleneck? Without per-step latency, you're guessing. Use OpenTelemetry or LangSmith traces when running LangChain-based workflows.
Human escalation path: Every multi-agent workflow needs a defined escalation path when confidence thresholds aren't met. Low-confidence routing to a human review queue is not a failure — it's the system working correctly.
Practical Patterns We Use
Skeptical reviewer pattern: After every AI-generated output, route through a lightweight reviewer agent that checks plausibility. The reviewer isn't re-doing the full task — it's doing a quick sanity check ("Does this output make sense given the input?"). Costs a few hundred tokens per step but catches a significant fraction of hallucinations.
Idempotency keys on all agent calls: Pass a deterministic idempotency key with every LLM call when your infrastructure supports it. On retry, you get the same response without additional cost. Some LLM providers support this natively; others require your own caching layer.
Canary deployment for new agents: When deploying a new agent version, route 5% of production traffic to the new version while monitoring quality metrics against the existing version. Multi-agent system quality regressions are harder to catch than traditional software regressions — small behavioral changes compound across a workflow.
When to Use Multi-Agent vs. Single Agent
Multi-agent architecture is worth the additional complexity when: - The task genuinely benefits from specialization (different context windows, different prompts for different steps) - Parallel execution of independent subtasks would meaningfully reduce latency - Quality improves from review stages between steps
Don't add multi-agent complexity for: - Tasks where a single well-prompted agent performs comparably - Early prototypes where you're still validating the approach - Low-volume workflows where operational simplicity matters more than performance
Partner With Rutagon for Production AI Systems
Architecting AI workflow systems that survive real production conditions requires more than prompt engineering. Rutagon builds and debugs multi-agent AI infrastructure for companies that need this working reliably. Reach out at rutagon.com/contact.
See also: AI capabilities overview and production AI deployment patterns.
FAQ
How do you prevent infinite loops in multi-agent systems?
Set a maximum step count on every workflow execution and a maximum retry count per step. Workflows that exceed the step limit should fail explicitly rather than continuing indefinitely. Track retry counts in durable state, not in-memory — so restarts don't reset retry counters and allow infinite retries across restarts.
How do multi-agent systems handle partial workflow completion on crash?
The orchestrator's state store is the source of truth. On restart, the orchestrator queries the state store for in-progress workflows and resumes from the last successful checkpoint. This requires that each step is idempotent — running the same step twice produces the same result. Design agent steps with idempotency in mind from the start.
What's the best way to test multi-agent workflows?
Unit test each agent with fixed inputs and expected outputs. Integration test the full workflow with deterministic mock agents (agents that return fixed responses) to test the orchestration logic separately from the LLM behavior. Run a shadow pipeline in production that processes real inputs with a new agent version and compares outputs to the current version before switching traffic.
How do you control costs in multi-agent production workflows?
Set per-workflow token budgets enforced by the orchestrator. If a workflow is approaching its token budget, either compress context or escalate to human review rather than continuing to burn tokens. Log token usage per step and workflow type — cost anomalies in multi-agent systems usually trace to a specific agent or workflow type that's consuming far more than expected.
Should each agent be a separate service or all in one process?
Start with a single process, separate execution classes. Extract to separate services only when you have a specific reason: different scaling requirements for different agents, different team ownership, or the need for different runtime environments. Premature service decomposition in multi-agent systems creates deployment complexity before you understand the actual scaling profile.