Arun Baby

Becoming Unlabelable

19 minute read

“If you want to go fast, go alone. If you want to go far, go together.”

1. Introduction: Beyond the “Solo” Agent

In our journey so far, we have built specialized agents for voice, vision, screen reading, and database querying. These are powerful on their own, but they are limited by their Domain Focus. A SQL agent can’t see a graph, and a Vision agent can’t execute a financial transaction.

To solve complex, real-world problems (like “Research a company’s competitors and write a 10-page investment memo”), you need a Multi-Agent System (MAS).

Multi-Agent Architectures allow us to break a massive task into smaller, manageable sub-tasks, each handled by a “Specialist” agent. This approach mirrors human organizations: you don’t expect the CEO to also be the lead developer, the HR manager, and the janitor. You hire experts and coordinate them.

In this post, we will explore the different ways to build and coordinate teams of AI agents.

2. Why Multi-Agent? (The Engineering Rationale)

Why bother with the complexity of multiple agents? Why not just use one “Super-Agent” with access to 100 tools?

  1. Context Management: Every tool you give an agent consumes tokens and increases the chance of “Parameter Hallucination.” By splitting agents, each one only needs to know the tools for its specific task.
  2. Specialized Prompting: A “Research Agent” needs a prompt focused on source verification, while a “Writing Agent” needs a prompt focused on tone and grammar. One single prompt cannot do both perfectly without being excessively long and confusing.
  3. Parallelism (Throughput): Specialist agents can work on different parts of a project simultaneously. While Agent A is scraping data, Agent B can be analyzing the data that Agent A already finished.
  4. Resilience: If the “API Agent” crashes due to a rate limit, the “Planner Agent” can detect this and decide to skip that step or try a different approach, without the whole system failing.

2.1 The “Silo” Effect (A Warning)

While splitting agents is powerful, it creates “Silos.” If Agent A doesn’t know what Agent B is doing, they might repeat work. As an engineer, your job is to design the State Schema that ensures critical info flows between silos without creating token bloat.

3. Architecture 1: The “Hierarchical” (Supervisor) Pattern

This is the most common pattern for enterprise agents.

3.1 The Supervisor (The Router)

A high-level Supervisor Agent acts as the “Manager.”

  • Role: It receives the user’s goal, breaks it into a list of sub-tasks, and assigns those tasks to specialized “Workers.”
  • Routing Logic: The supervisor doesn’t just “talk”—it “routes.” It outputs a JSON like {"worker": "Researcher", "task": "Find top AI trends 2024"}.
  • Review: After a worker finishes, the Supervisor reviews the output. If it’s not good enough, it sends it back with a revision_request.

3.2 The Workers (The Specialists)

These are specialized agents (e.g., Code Specialist, Web Searcher, QA Tester).

  • Constraint: Workers usually do not talk to each other. They only report back to the Supervisor.
  • Junior Tip: Limit each worker to a maximum of 5 tools. If a worker needs 10 tools, it’s probably not a “worker”—it’s a sub-supervisor.

4. Conflict Resolution: When Agents Disagree

In collaborative or hierarchical systems, two agents might produce conflicting results.

  • Scenario: The “Finance Agent” says the budget is $10k, but the “Sales Agent” says it’s $15k.

Patterns for Resolution:

  1. Highest Confidence Wins: Every agent must output a confidence_score (0-1). The orchestrator picks the result with the highest score.
  2. The Tie-Breaker (Senior Agent): If a conflict is detected in the shared state, the system triggers a specialized “Mediator Agent” (using a high-end model like GPT-4o) to review both inputs and make a final call.
  3. Consensus (Voting): In a “Peer-to-Peer” system (Section 4), you can use an odd number of agents (e.g., 3) and have them vote on the correct output.

4. Architecture 2: The “Collaborative” (Peer-to-Peer) Pattern

In this model, there is no boss. Agents talk to each other directly in a “Round Robin” or “Blackboard” fashion.

4.1 The Blackboard Pattern

All information is posted to a shared State Object (the “Blackboard”). Any agent can read from the board and write new insights to it.

  • Workflow:
    1. Agent A (Data Fetcher) writes a list of facts.
    2. Agent B (Analyst) sees the facts and writes a summary.
    3. Agent C (Fact Checker) sees the summary and flags a mistake.
    4. Agent B sees the flag and rewrites the summary.

The Risk: Infinite Loops. Without a supervisor, agents might argue forever (e.g., Agent B and Agent C disagreeing on a fact).

  • Mitigation: Always implement a “Recursion Limit” (e.g., max 10 message exchanges) or a “Final Decision Maker” node.

5. Architecture 3: The “Sequential” (assembly line) Pattern

This is the simplest multi-agent pattern. The output of Agent A is the input to Agent B, which is the input to Agent C.

Example: A Content Creation Pipeline

  1. Researcher: Finds 10 facts about AI Agents.
  2. Writer: Turns the facts into a blog post.
  3. Editor: Corrects the spelling and formatting.
  4. Translator: Converts the finished post into Spanish.

Benefit: Predictable and easy to debug. You know exactly where a “bad” output came from.

5. Dynamic Team Formation: The “Agentic HR” Pattern

In static systems, you define your agents once: Supervisor, Researcher, Writer. But in a Dynamic Multi-Agent System, the system creates agents on the fly.

The Architecture:

  1. Orchestrator Node: Takes the user request.
  2. Tool: create_specialist(domain, prompt): If the request involves a domain you don’t have a pre-trained agent for (e.g., “Analyze the legal implications of this contract”), the Orchestrator uses a high-end LLM to generate a custom system prompt and spins up a temporary agent.
  3. Tear-down: Once the legal analysis is done, the temporary agent is destroyed to save resources.

6. Agent Discovery: The “Service Mesh” for AI

How does the Supervisor know which workers are available? You need an Agent Registry (similar to a tool registry).

Registry Metadata:

  • Model ID: (e.g., GPT-3.5-Turbo for speed, GPT-4o for quality).
  • Capabilities: A list of “Skills” (e.g., “Web Scraping”, “Image Generation”).
  • Availability: Is the agent currently overloaded with other tasks?
  • Cost/Token: What is the price of using this agent?

The Discovery Workflow: Instead of hardcoding “Call Agent X,” the Supervisor asks the Registry: “Who is the best available agent for generating 10 variations of an ad copy?”. The Registry returns the ID of the “Copywriter Agent.”

7. Shared Memory: The “Central State”

In a multi-agent system, how do agents know what others have done?

7.1 Ephemeral Memory (The LangGraph State)

Maintain a shared JSON object that stores the “History of Actions.” Every agent, when it finishes its turn, appends its result to this object.

  • Junior Tip: Do not pass the entire history to every agent. Only pass the relevant parts (e.g., the Translator doesn’t need to see the Researcher’s raw data, only the Writer’s final draft).

7.2 Narrative Memory (The “Global Summary”)

If a task is very long, the history will become too large. Use a Summarizer Agent that periodically condenses the work done so far into a 1-page “Status Report” that all agents can read.

8. Cost-Aware Multi-Agent Systems

A MAS can be extremely expensive. If you have 10 agents, all using GPT-4o, and they get into a 10-turn conversation, you’ve just spent $50 on a single request.

Strategies for Economic Stability:

  • Hybrid Models: Use a “cheap” model (Llama 3 or GPT-4o-mini) for the Workers and a “premium” model (Claude 3.5 Sonnet) only for the Supervisor who does the final review.
  • Token Quotas: Each worker is given a token “budget.” If it exceeds the budget without reaching a conclusion, it is terminated and the Supervisor is notified.
  • Compression Agents: Before passing the history from the Researcher to the Writer, use a Compression Agent to summarize the findings into the smallest possible token footprint.

9. Performance: Orchestrating for Latency

In a multi-agent assembly line (Section 5), the user has to wait for Agent A, then B, then C. This results in poor UX.

Optimization Patterns:

  • Speculative Execution: If the “Planner” agent is 90% sure the “Coder” agent will need a specific library, have the “Environment Agent” start installing that library while the “Coder” is still thinking.
  • Streaming Ticks: As workers finish sub-tasks, stream them back to the user immediately instead of waiting for the final “Supervisor Review.” Label them as “Work in Progress.”

11. Security: Agent Isolation and the “Least Privilege” MAS

In a multi-agent system, not every agent should have the same level of access.

The Isolation Pattern:

  • The Researcher Agent: Has access to the Search API but NO access to the user’s local files or financial database.
  • The Finance Agent: Has access to the local database but NO access to the external internet.
  • The Air-Gap Pattern: Use the Supervisor as a “Proxy.” The Researcher writes its findings to the Supervisor. The Supervisor “Sanitizes” the findings (removing any potential prompt injections) and only then passes the sanitized text to the Finance Agent.
  • Why? This prevents an external website from “hacking” your Researcher and using it to steal data from your Finance Agent.

12. Self-Healing Agent Teams: The “Reviewer-Actor” Loop

One of the most powerful features of MAS is Automated Error Correction.

The Workflow:

  1. Actor Agent: Generates code to solve a problem.
  2. Linter/Tester Tool: Runs the code and finds a bug.
  3. Reviewer Agent: Analyzes the bug and the Actor’s code. It provides high-level feedback: “You forgot to handle the case where the list is empty.”
  4. Actor Agent: Receives the feedback and regenerates the code.
  5. Target: This loop continues until the Tester Tool returns “Green.”

13. Benchmarking MAS: The “AgentBench” Standard

Testing a single agent is hard. Testing 10 agents coordinating is exponentially harder.

Metrics to Track:

  • Coordination Score: How many turns did the Supervisor need to reach a conclusion? (Lower is better).
  • Information Decay: Did the Writer Agent lose a critical fact that the Researcher found earlier in the chain?
  • Instruction Adherence: Did the Workers follow the Supervisor’s specific constraints?
  • Frameworks: Use AgentBench or GAIA (General AI Assistants) benchmarks to evaluate your system against industry standards.

14. Architecture: The “Multi-Agent Sidecar”

Just as we used sidecars for APIs, we can use them for Specialized Workers.

  • Run each agent in its own isolated container.
  • Use a Message Broker (like RabbitMQ or Redis Pub/Sub) for communication.
  • Benefit: If your Code Agent requires heavy Python dependencies and your Vision Agent requires heavy PyTorch dependencies, you don’t have to build one massive, bloated Docker image. You build two small ones.

15. Pattern: Human-in-the-Loop Coordination

In a MAS, the Human is often an “Agent” themselves.

Coordination Patterns:

  1. The Interject Pattern: The system runs autonomously until an agent hits a specific confidence threshold or encounters an ambiguous task. It then “interrupts” the flow and asks the human for a decision. Once the human provides input, the MAS resumes its cycle.
  2. The Shadowing Pattern: The human performs a task (e.g., browsing a website), and the MAS “watches” and learns. The MAS then generates a “Proposed Next Step” for the human to approve.
  3. The Red-Teaming Pattern: A team of agents works together to solve a problem, and a separate team of “Red-Team” agents tries to poke holes in the solution. The human acts as the final judge between the two teams.

16. Ethics: The “Responsibility Gap” in MAS

Who is responsible when a MAS fails?

  • If Agent A provides wrong data, which causes Agent B to make a bad decision, which causes Agent C to execute a harmful action—who is to blame?

Engineering Solutions:

  • Audit Logging: Every handoff between agents must be signed (using cryptographic hashes). This creates a verifiable “Chain of Causality.”
  • Explainable Traceability: Use tools like LangSmith to visualize the reasoning path that led to a specific outcome. As an engineer, you must be able to “Explain the organizational failure” just as a manager explains a team failure.

17. The Future: Agent-to-Agent Economies

Soon, agents won’t just coordinate within one company; they will coordinate Between Companies.

  • Scenario: Your personal “Shopping Agent” talks to Amazon’s “Sales Agent” and UPS’s “Logistics Agent” to negotiate a price and delivery time without any human involvement.
  • Infrastructure needed: Decentralized Identity (DID) for agents, Micropayment protocols (Lightning Network), and standardized Negotiation Protocols.

18. Pattern: Domain-Specific Languages (DSLs) for Multi-Agent Systems

Sometimes, natural language is too “fuzzy” for agent-to-agent talk. In high-precision environments (like medical or aerospace), agents use DSLs.

Why use a DSL?

  • Zero Ambiguity: A command in a DSL has exactly one meaning.
  • Formal Verification: You can use mathematical tools to prove that a sequence of DSL commands won’t lead to a system crash.
  • Junior Tip: You don’t need to invent a new language. You can use a constrained version of Python or a specialized JSON schema as your “Internal Agent Language.”

19. Case Study: The “Autonomous Software Engineering” Team

Imagine a team of agents tasked with fixing a bug in a GitHub repository.

  1. Issue Triage Agent: Reads the bug report and labels it.
  2. Repo Cartographer: Indexes the codebase and finds the relevant files (Section 2.2 of the Tool Design post).
  3. Debugger Agent: Generates 3 different theories on why the bug is happening.
  4. Coder Agent: Writes a patch for the most likely theory.
  5. QA Runner: Executes the test suite. If it fails, it sends the logs back to the Debugger.
  6. Reviewer Agent: Performs a “Code Review” for security and style.
  7. PR Agent: Submits the final Pull Request once all checks pass.

The Result: A 24/7 autonomous maintenance team that only bothers the senior architect when it hits a “Deep Logic” problem it can’t resolve after 5 attempts.

21. Pattern: Recursive Planning and Sub-Tasking

Sometimes, a task is so complex that even the Supervisor cannot break it down in one go. You need Recursive Planning.

The Process:

  1. Level 1 Supervisor: Receives “Build a SaaS.” It breaks it into “Frontend,” “Backend,” and “Ops.”
  2. Level 2 Specialized Supervisor (Frontend): Receives “Build Frontend.” It breaks it into “Auth UI,” “Dashboard UI,” and “API Integration.”
  3. Level 3 Workers: Receive the granular tasks (e.g., “Write the Login component in React”).
    • Junior Tip: This is essentially a Tree of Agents. The advantage is that if the “Auth UI” fails, the “Frontend Supervisor” can fix it without bothering the “CEO Supervisor” at the top.

22. Architecture: Agent Environments (The Sandbox)

Multi-agent systems should not run raw code on your laptop. They need a Shared Sandbox.

The Setup:

  • The Shared Drive: A Docker volume that all agents in the team can read and write to.
  • The Execution Shell: A secure, isolated environment (like E2B or Docker) where the Code Agent can run tests and the Results Agent can read the logs.
  • Network Isolation: Ensure that while the “Web Scraper Agent” can reach the internet, the “Code Runner Agent” is completely air-gapped from your local network.

23. Case Study: The AI Research Lab

How does a team of agents discover new scientific insights?

  1. The Hypothesis Agent: Reads 100 recent papers and proposes a new experiment.
  2. The Simulation Agent: Writes a Python simulation to test the hypothesis.
  3. The Critic Agent: Looks for “Data Leakage” or statistical errors in the simulation.
  4. The Librarian Agent: Manages the bibliography and ensures all citations are correct.
  5. The Final Writer: Compiles the findings into a LaTeX document.

The Achievement: By coordinating these specialists, you can run 10,000 “Research Cycles” in parallel, discovering patterns that a human team might take years to notice.

25. Architecture: Agent Swarms vs. Orchestrated Teams

How does a 100-agent system differ from a 3-agent system?

25.1 The Orchestrated Team (Hierarchical)

  • Structure: One “Leader” agent assigns tasks to “Worker” agents.
  • Benefit: High controllability and transparency.
  • Scale: Limited. The Leader agent becomes a bottleneck (Section 2.2).
  • Usage: Enterprise workflows (e.g., Document processing, Software development).

25.2 The Agent Swarm (Emergent)

  • Structure: No leader. Agents follow simple local rules (like ants or bees). They “Broadcast” their state to a shared bus.
  • Benefit: Massive scalability and resilience. If one agent dies, the swarm barely notices.
  • Scale: Thousands of agents.
  • Usage: Large-scale simulations, autonomous logistics, and decentralized computing.

26. Conflict Resolution in Shared Environments

When two agents try to edit the same file or access the same hardware, we have a Race Condition.

The MAS Solutions:

  1. Distributed Locking (Mutex): The orchestrator maintains a “Key” for every resource. An agent must “Check out” the key before acting.
  2. Turn-Based Execution: In high-risk environments, agents are strictly sequential. Agent B cannot start until Agent A explicitly broadcasts a RELEASE_LOCK signal.
  3. Conflict-Free Replicated Data Types (CRDTs): A mathematical way to allow both agents to edit the same data and automatically “Merge” their changes without a central server.

27. Observability: Debugging the “Meeting”

Debugging one agent is hard. Debugging 10 agents talking to each other is a nightmare.

Essential Tools for Juniors:

  • Trace Context: Every request must carry a trace_id. If the User asks a question, and 5 agents are involved in the answer, all their logs must share that ID so you can see the sequence.
  • Gantt Charts for Agents: Visualize the “Time” each agent spent on a sub-task. If Agent A is waiting 10 seconds for Agent B, you have a performance bottleneck.
  • Sentiment Monitoring: Is Agent C being “Aggressive” or “Unhelpful” to Agent D? By tracking the sentiment of agent-to-agent talk, you can identify “Personality Clashes” in your system prompt design.

29. Logic Link: Mixture of Experts (MoE) & Sliding Windows

In the ML track, we discuss Mixture of Experts (MoE). Multi-Agent Systems are effectively “MoE at the Macro level.” In a model like Mixtral, a “Router” sends a token to the best 2 of 8 experts. In a Multi-Agent system (Section 2), an “Orchestrator Agent” sends a task to the best 2 of 8 specialized agents. The math is differnet, but the philosophy—Sparse Activation of Specialized Workers—is identical.

In DSA we solve Longest Substring Without Repeating Characters. This uses the Sliding Window technique. When a set of agents are processing a long document (Section 11), they often use a “Sliding Window of Context” where they share a moving window of recent thoughts to maintain coherence without overwhelming the context limit.

30. Summary & Junior Engineer Roadmap

Multi-agent architecture is about moving from “Simple Software” to “Digital Organizations.”

Your Roadmap to Mastery:

  1. Orchestration Frameworks: Master LangGraph or CrewAI. These handle the “State” and “Routing” logic so you don’t have to write it yourself.
  2. State Management: Learn how to design minimal JSON schemas for agent handoffs. Every token in your state is a token that costs money on every turn.
  3. Human-in-the-Loop: Implement “Breakpoints” where a human can intervene in the multi-agent flow to correct a mistake before it cascades down the pipeline.
  4. Debugging MAS: Use tools like LangFuse to visualize the “Graph” of your agent interactions. If a task fails, you need to know exactly which specialist dropped the ball.

In the next and final post of this series, we will look at Agent Communication Protocols: the specific “Language” and “Schemas” agents use to talk to each other reliably without human intervention.

Related across topics