Chapter 36: AI Coding Agents and Autonomous Workflows

Learning Objectives

By the end of this chapter, you will be able to:

1. Analyze the architectural differences between conversational AI assistants and autonomous coding agents (Bloom's: Analysis)
2. Evaluate the plan-act-observe loop and its role in enabling agent autonomy (Bloom's: Evaluation)
3. Design tool-use frameworks that allow agents to interact with codebases, terminals, and external services (Bloom's: Synthesis)
4. Implement guardrails and safety mechanisms that prevent destructive agent actions (Bloom's: Application)
5. Create human-in-the-loop patterns that balance automation with oversight (Bloom's: Synthesis)
6. Assess agent memory strategies for maintaining context across long-running tasks (Bloom's: Evaluation)
7. Construct error recovery mechanisms that enable graceful degradation in agent workflows (Bloom's: Synthesis)
8. Evaluate agent performance using systematic metrics and benchmarks (Bloom's: Evaluation)
9. Build a simple coding agent from scratch that demonstrates core agent principles (Bloom's: Creation)

Introduction

Throughout this book, you have used AI as a coding assistant: you ask a question, it responds, you refine, and you iterate. This conversational paradigm has transformed how developers write software. But a far more profound shift is underway. AI systems are evolving from assistants that respond to individual prompts into agents that autonomously plan, execute, and complete multi-step software engineering tasks.

Imagine describing a feature request in a single sentence and watching as an AI agent reads the relevant source files, plans an implementation strategy, writes the code across multiple files, creates tests, runs them, fixes any failures, and opens a pull request---all without further human intervention. This is not science fiction. Tools like Claude Code, GitHub Copilot Workspace, and Devin have begun to demonstrate exactly this capability.

This chapter explores the architecture, patterns, and practices behind AI coding agents. You will learn how agents differ from assistants, how the plan-act-observe loop drives autonomous behavior, how tool use enables agents to interact with the real world, and how guardrails keep these powerful systems safe. By the end of the chapter, you will build a simple coding agent from scratch.

Callout: Why This Matters

Agent-based coding is the next frontier in software development. Understanding how agents work---not just how to use them---gives you the ability to build custom automation, evaluate agent tools critically, and maintain meaningful oversight as these systems grow more capable. Whether you are a solo developer automating repetitive tasks or a team lead evaluating enterprise AI tools, agent literacy is becoming a core professional skill.

36.1 From Assistant to Agent

The Conversational Assistant Paradigm

The AI coding assistants you have used throughout this book operate in a request-response pattern. You provide a prompt, the model generates a response, and the conversation continues. Each exchange is relatively self-contained: the assistant reads your input, generates output, and waits for your next instruction.

This paradigm has clear strengths. It keeps the human in control, produces predictable interactions, and works well for tasks like code explanation, single-function generation, and debugging guidance. But it has fundamental limitations:

• Single-step reasoning: Each response addresses one prompt, even when the underlying task requires many steps.
• No environmental interaction: The assistant cannot read files, run commands, or verify that its suggestions actually work.
• Stateless execution: While the conversation history provides context, the assistant does not maintain persistent state about the project.
• Human bottleneck: Every step requires human input, even when the next action is obvious.

What Makes an Agent?

An AI coding agent differs from an assistant in several critical ways:

An agent is a system that receives a high-level goal and autonomously determines and executes the steps needed to achieve it. The key properties are:

1. Goal-directed behavior: The agent works toward an objective rather than responding to individual prompts.
2. Environmental interaction: The agent can read files, write files, execute commands, make API calls, and observe the results.
3. Iterative refinement: The agent observes the outcomes of its actions and adjusts its approach accordingly.
4. Autonomy: The agent can make decisions about what to do next without requiring human input at every step.

The Spectrum of Autonomy

It is important to recognize that the assistant-to-agent transition is not binary. There is a spectrum of autonomy:

Level 0 --- Pure Assistant: Generates text in response to prompts. No tool use, no autonomous action. Example: a basic chatbot that suggests code snippets.

Level 1 --- Tool-Augmented Assistant: Can use tools when instructed but requires human approval for each action. Example: an IDE extension that can read files when asked.

Level 2 --- Supervised Agent: Plans and executes multi-step tasks but pauses for human approval at key checkpoints. Example: Claude Code in its default mode, which asks permission before writing files or running commands.

Level 3 --- Autonomous Agent: Executes end-to-end workflows with minimal human intervention, subject to predefined guardrails. Example: a CI/CD agent that automatically fixes failing tests.

Level 4 --- Self-Improving Agent: Not only executes tasks but also improves its own capabilities over time. This level remains largely theoretical for production coding agents.

Callout: Claude Code as a Real-World Agent

Claude Code exemplifies the Level 2--3 range on the autonomy spectrum. It operates in a plan-act-observe loop: it reads your codebase, plans changes, executes them using tools (file reading, file writing, terminal commands, web search), and observes the results. By default, it asks for permission before potentially destructive actions, but it can be configured for greater autonomy with appropriate guardrails. Throughout this chapter, we will reference Claude Code's architecture as a concrete example of agent design patterns.

36.2 The Agent Loop: Plan-Act-Observe

The Core Loop

At the heart of every coding agent is a cycle that repeats until the task is complete or a stopping condition is met:

while not done:
    observation = observe(environment)
    plan = plan(goal, observation, memory)
    action = select_action(plan)
    result = execute(action)
    memory = update_memory(result)
    done = check_completion(goal, memory)

This plan-act-observe loop (sometimes called the "reason-act" loop or "ReAct" pattern) is the fundamental architecture of agent systems. Let us examine each phase.

Phase 1: Observe

The agent gathers information about the current state of its environment. For a coding agent, this might include:

• Reading the contents of source files
• Checking the output of recent terminal commands
• Examining test results
• Reviewing error messages and stack traces
• Inspecting project structure and configuration files

Observation is not passive. The agent must decide what to observe based on its current goal and what it already knows. An effective agent observes strategically, gathering only the information relevant to its current task rather than trying to read every file in the repository.

Phase 2: Plan

Based on its observations and goal, the agent formulates a plan. Planning in coding agents typically involves:

• Task decomposition: Breaking a high-level goal into smaller, actionable steps
• Dependency analysis: Determining the order in which steps must be executed
• Strategy selection: Choosing between alternative approaches
• Risk assessment: Identifying steps that might fail and planning contingencies

The plan is not necessarily fixed. Sophisticated agents use adaptive planning, where the plan is revised after each action based on new observations. This is crucial because software development is inherently unpredictable---a file might not be where you expect it, a test might fail for unexpected reasons, or a dependency might have changed.

Phase 3: Act

The agent executes one step of its plan. Actions available to a coding agent typically include:

• File operations: Reading, writing, creating, and deleting files
• Terminal commands: Running builds, tests, linters, and other command-line tools
• Search operations: Finding files by pattern, searching file contents, searching the web
• Code generation: Writing new code or modifying existing code
• Communication: Reporting progress, asking for clarification, or requesting approval

Phase 4: Update and Iterate

After each action, the agent observes the result, updates its internal state (memory), and decides whether to continue, revise its plan, or stop. This feedback loop is what gives agents their power: they can detect and recover from errors, adapt to unexpected situations, and verify that their work is correct.

Callout: The ReAct Pattern

The plan-act-observe loop is closely related to the ReAct (Reasoning + Acting) pattern described in the research literature. In ReAct, the agent alternates between reasoning steps (where it thinks through the problem in natural language) and action steps (where it interacts with tools). The reasoning steps are made explicit in the agent's output, creating a chain of thought that is both interpretable and useful for debugging. When you see Claude Code "thinking" before taking an action, you are observing the ReAct pattern in practice.

Loop Termination

An agent loop must have clear termination conditions. Without them, the agent might loop indefinitely, wasting resources and potentially causing damage. Common termination conditions include:

• Goal achieved: The task is complete (e.g., all tests pass, the PR is created)
• Maximum iterations: A hard limit on the number of loop iterations
• Maximum cost: A budget limit on API calls or compute resources
• Error threshold: Too many consecutive errors indicate the agent is stuck
• Human intervention: A human explicitly stops the agent
• Timeout: A wall-clock time limit

36.3 Tool Use and Function Calling

Why Agents Need Tools

A language model by itself can only generate text. To become an agent, it needs to interact with the external world. This is where tool use---also called function calling---becomes essential.

Tools give an agent the ability to:

• Read information: Access file systems, databases, APIs, and web pages
• Take action: Write files, execute commands, make API calls
• Verify results: Run tests, check outputs, validate configurations

How Function Calling Works

Function calling is a mechanism where the model's output includes structured instructions to invoke specific functions. Here is the basic flow:

1. The system prompt defines available tools with their names, descriptions, and parameter schemas.
2. The model generates a response that includes a tool call (a structured request to invoke a specific function with specific arguments).
3. The host system intercepts the tool call, executes the function, and returns the result to the model.
4. The model incorporates the tool result into its reasoning and continues.

# Simplified tool definition
tools = [
    {
        "name": "read_file",
        "description": "Read the contents of a file at the given path",
        "parameters": {
            "type": "object",
            "properties": {
                "path": {
                    "type": "string",
                    "description": "Absolute path to the file"
                }
            },
            "required": ["path"]
        }
    },
    {
        "name": "write_file",
        "description": "Write content to a file at the given path",
        "parameters": {
            "type": "object",
            "properties": {
                "path": {"type": "string"},
                "content": {"type": "string"}
            },
            "required": ["path", "content"]
        }
    },
    {
        "name": "run_command",
        "description": "Execute a shell command and return stdout/stderr",
        "parameters": {
            "type": "object",
            "properties": {
                "command": {"type": "string"}
            },
            "required": ["command"]
        }
    }
]

Essential Tools for Coding Agents

A well-equipped coding agent needs tools in several categories:

File System Tools - read_file(path): Read file contents - write_file(path, content): Write or overwrite a file - edit_file(path, old_text, new_text): Make targeted edits to a file - list_directory(path): List files and directories - search_files(pattern, path): Find files matching a glob pattern

Terminal Tools - run_command(command, timeout): Execute a shell command - run_tests(test_path): Run a test suite and return results - run_linter(path): Run a linter and return findings

Search and Information Tools - grep(pattern, path): Search file contents with regex - web_search(query): Search the web for documentation or solutions - fetch_url(url): Retrieve content from a URL

Version Control Tools - git_status(): Check the status of the repository - git_diff(): View current changes - git_commit(message): Create a commit - git_create_branch(name): Create a new branch

Tool Design Principles

When building tools for agents, follow these principles:

1. Clear descriptions: The model chooses tools based on their descriptions. Vague descriptions lead to poor tool selection.
2. Specific parameters: Use typed, well-documented parameters. Avoid catch-all parameters like options: dict.
3. Informative return values: Return enough information for the model to understand the result and decide what to do next.
4. Error handling: Tools should return structured error information rather than crashing. The agent needs to understand what went wrong to attempt recovery.
5. Idempotency where possible: Tools that can be safely retried reduce the risk of partial failure.

Callout: The Tool Description Is Everything

The quality of tool descriptions directly determines how effectively an agent uses its tools. A description like "Execute command" is far less useful than "Execute a shell command in the project's root directory. Returns stdout, stderr, and the exit code. Times out after 120 seconds. Use for running tests, builds, linters, and other CLI tools." The description is effectively the tool's user manual for the AI.

36.4 Autonomous Code Generation Workflows

End-to-End Automation

The ultimate promise of coding agents is end-to-end automation: give the agent a high-level task, and it handles everything from understanding the requirement to delivering working, tested code. Here are the major workflow patterns.

Pattern 1: Issue-to-Pull-Request

The agent receives a GitHub issue describing a feature or bug fix and autonomously produces a pull request.

Steps: 1. Read the issue description and any linked context 2. Explore the codebase to understand the relevant architecture 3. Plan the implementation approach 4. Create a feature branch 5. Implement the changes across one or more files 6. Write or update tests 7. Run the test suite and fix any failures 8. Run linters and formatters 9. Create a commit with a meaningful message 10. Open a pull request with a clear description

Pattern 2: Test-Driven Development Agent

The agent uses test failures to drive its implementation, mirroring TDD methodology.

Steps: 1. Receive a specification or requirement 2. Write failing tests based on the specification 3. Run the tests to confirm they fail 4. Implement code to make the first test pass 5. Run tests again, observe results 6. Iterate until all tests pass 7. Refactor the implementation while keeping tests green

Pattern 3: Bug Diagnosis and Fix

The agent receives a bug report and autonomously diagnoses and fixes it.

Steps: 1. Read the bug report and reproduce the issue 2. Analyze error messages and stack traces 3. Search the codebase for relevant code 4. Identify the root cause 5. Plan and implement the fix 6. Verify the fix resolves the issue 7. Check for regressions by running the full test suite 8. Document the fix in the commit message

Pattern 4: Code Review Agent

The agent reviews a pull request and provides feedback or automatically fixes issues.

Steps: 1. Read the PR diff and description 2. Understand the context by reading related code 3. Check for bugs, security issues, performance problems, and style violations 4. Generate inline comments or suggestions 5. Optionally, apply fixes directly

Orchestrating Complex Workflows

For complex tasks, agents may need to orchestrate multiple sub-tasks. This is where hierarchical planning becomes valuable:

Main Goal: "Add user authentication to the API"
├── Sub-task 1: Design the authentication schema
│   ├── Research existing patterns in the codebase
│   ├── Design the user model
│   └── Design the token schema
├── Sub-task 2: Implement the user model
│   ├── Create migration
│   ├── Create model class
│   └── Write model tests
├── Sub-task 3: Implement authentication endpoints
│   ├── Create login endpoint
│   ├── Create registration endpoint
│   ├── Create token refresh endpoint
│   └── Write endpoint tests
├── Sub-task 4: Add middleware
│   ├── Create authentication middleware
│   ├── Add to route configuration
│   └── Write middleware tests
└── Sub-task 5: Integration testing
    ├── Write end-to-end tests
    ├── Run full test suite
    └── Fix any failures

Callout: The 80/20 Rule of Agent Autonomy

In practice, agents excel at automating the 80% of development work that is routine: boilerplate code, standard patterns, test writing, documentation updates, and straightforward bug fixes. The remaining 20%---novel architectural decisions, complex business logic, ambiguous requirements---still benefits from human judgment. Effective agent workflows are designed around this reality, automating the routine while flagging the exceptional for human attention.

36.5 Guardrails and Safety Mechanisms

Why Guardrails Are Non-Negotiable

An autonomous agent that can read files, write files, and execute arbitrary commands is extraordinarily powerful---and extraordinarily dangerous. Without proper guardrails, a coding agent could:

• Delete critical files or entire directories
• Execute malicious commands
• Expose sensitive data (API keys, credentials, personal information)
• Make irreversible changes to production systems
• Consume excessive computational resources
• Create an infinite loop of expensive API calls

Guardrails are not optional safety theater. They are essential engineering requirements.

Permission Systems

The first line of defense is a permission system that controls what the agent is allowed to do:

Allowlists and Denylists

ALLOWED_DIRECTORIES = ["/project/src", "/project/tests"]
BLOCKED_DIRECTORIES = ["/etc", "/root", "/project/.env"]
BLOCKED_COMMANDS = ["rm -rf", "sudo", "chmod 777", "curl | bash"]
ALLOWED_FILE_EXTENSIONS = [".py", ".js", ".ts", ".md", ".json", ".yaml"]

Tiered Permissions

Different actions require different levels of authorization:

• Tier 1 --- No approval needed: Reading files, searching code, running read-only commands
• Tier 2 --- Soft approval: Writing files in the project directory, running tests
• Tier 3 --- Explicit approval: Deleting files, running arbitrary commands, accessing the network
• Tier 4 --- Always blocked: Modifying system files, accessing credentials, destructive operations outside the project

Sandboxing

Sandboxing confines the agent to a restricted environment where even if it attempts a destructive action, the damage is contained:

• Container-based sandboxing: Run the agent in a Docker container with limited filesystem access and no network access to production systems
• Virtual environment sandboxing: Use Python virtual environments to isolate dependencies
• chroot jails: Restrict the agent's view of the filesystem to a specific directory tree
• User-level isolation: Run the agent as a user with minimal permissions

Cost Controls

Agent workflows can be expensive. Without cost controls, a bug in the agent loop could generate thousands of API calls:

• Token budgets: Set a maximum number of tokens the agent can consume per task
• Iteration limits: Cap the number of plan-act-observe loops
• Time limits: Set a wall-clock timeout for the entire task
• Rate limiting: Throttle API calls to prevent runaway costs
• Cost monitoring: Track spending in real time and alert on anomalies

Output Validation

Before the agent's output reaches the user or the codebase, validate it:

• Syntax checking: Parse generated code to verify it is syntactically correct
• Linting: Run linters to catch common errors and style violations
• Test execution: Run existing tests to verify no regressions
• Diff review: Review the proposed changes before applying them
• Secret scanning: Check that the output does not contain API keys, passwords, or other secrets

Callout: Defense in Depth

No single guardrail is sufficient. Effective agent safety uses defense in depth: multiple layers of protection, each catching failures that slip through the others. A permission system might prevent the agent from accessing a sensitive directory. If it somehow bypasses that, sandboxing limits the damage. If sandboxing fails, cost controls prevent unlimited resource consumption. If costs spiral, time limits force a shutdown. Each layer reduces risk; together, they make catastrophic failures extremely unlikely.

The Principle of Least Privilege

Agents should be granted the minimum permissions necessary to complete their task. If an agent only needs to modify Python files in the src/ directory, do not give it write access to the entire filesystem. If it only needs to run tests, do not give it access to production deployment commands.

This principle applies dynamically as well. An agent performing a code review needs read-only access. An agent implementing a feature needs write access to specific directories. An agent deploying code needs a different set of permissions entirely. Design your permission system to be task-specific, not one-size-fits-all.

36.6 Human-in-the-Loop Patterns

The Role of Human Oversight

Complete autonomy is rarely appropriate, especially for consequential actions in production codebases. Human-in-the-loop (HITL) patterns strike a balance between automation efficiency and human judgment:

Pattern 1: Approval Gates

The agent pauses at predefined checkpoints and waits for human approval before proceeding.

Agent: "I plan to modify 3 files to add user authentication:
  - src/models/user.py (new file)
  - src/routes/auth.py (new file)
  - src/middleware/auth.py (new file)

  Shall I proceed?"

Human: "Yes, proceed."

Agent: [executes the plan]

When to use: Before any destructive or irreversible action, before large-scale changes, before deploying to production.

Pattern 2: Review Checkpoints

The agent completes a phase of work and presents the results for review before continuing to the next phase.

Phase 1: Agent explores codebase and proposes architecture → Human reviews
Phase 2: Agent implements core functionality → Human reviews
Phase 3: Agent writes tests → Human reviews
Phase 4: Agent creates PR → Human reviews and merges

When to use: For complex, multi-phase tasks where early mistakes compound.

Pattern 3: Confidence-Based Escalation

The agent self-assesses its confidence in each action and only requests human input when confidence is low.

if confidence > 0.9:
    execute_autonomously()
elif confidence > 0.6:
    execute_with_notification()  # Human is notified but doesn't need to approve
else:
    request_human_approval()    # Agent pauses until human responds

When to use: For high-volume workflows where approving every action would be impractical.

Pattern 4: Exception Handling Escalation

The agent operates autonomously but escalates to a human when it encounters an error it cannot resolve after a set number of retries.

Agent: "I've attempted to fix the failing test 3 times but the error persists.
  The test expects a JSON response but the endpoint is returning HTML.
  I suspect there's a middleware configuration issue I don't understand.

  Here's what I've tried:
  1. Checking the Content-Type header in the route handler
  2. Reviewing the response serialization middleware
  3. Checking for response interceptors

  Could you help me understand the response pipeline?"

When to use: When the cost of the agent being stuck exceeds the cost of human intervention.

Pattern 5: Guided Autonomy

The human provides a high-level plan, and the agent executes each step autonomously.

Human: "Here's the implementation plan:
  1. Add a User model with email, hashed_password, and created_at fields
  2. Create POST /register and POST /login endpoints
  3. Add JWT token generation and validation
  4. Add auth middleware to protected routes
  5. Write tests for all new endpoints

  Execute each step and show me the results."

Agent: [executes step by step, reporting after each]

When to use: When the human knows what needs to be done but wants to delegate the implementation details.

Callout: Trust Is Earned, Not Assumed

The right level of autonomy depends on trust, which is built through experience. Start with high oversight (approval gates at every step) and gradually increase autonomy as you verify the agent's reliability. This is exactly how Claude Code works: new users start with default permission prompts, and as they gain confidence, they can configure the tool to allow more autonomous operation. Never start with maximum autonomy.

36.7 Agent Memory and State Management

The Memory Challenge

Effective agents need memory at multiple scales:

• Working memory: The current conversation context and immediate task state
• Short-term memory: Information gathered during the current task that needs to persist across loop iterations
• Long-term memory: Knowledge from previous tasks that might be relevant to future ones

Working Memory: Conversation Context

The most basic form of agent memory is the conversation history. Each iteration of the plan-act-observe loop adds to this history:

[System Prompt]
[User Goal: "Add pagination to the /users endpoint"]
[Agent Thought: "I need to understand the current endpoint implementation"]
[Tool Call: read_file("/src/routes/users.py")]
[Tool Result: "... file contents ..."]
[Agent Thought: "The endpoint returns all users. I need to add limit/offset parameters"]
[Tool Call: edit_file("/src/routes/users.py", ...)]
[Tool Result: "File updated successfully"]
...

The challenge with conversation context is that it grows with every iteration. Language models have finite context windows, and even large windows (100K+ tokens) can be exhausted by complex tasks that require reading many files.

Context Window Management

Strategies for managing the context window include:

Summarization: Periodically summarize older conversation history into a compact form. Instead of keeping every file read in full, summarize: "Read users.py --- it contains a Flask route handler that returns all users from the database with no pagination."

Selective retention: Keep the most relevant information in context and discard the rest. Recent actions and their results are more relevant than early exploration steps.

Sliding window: Maintain the most recent N turns of conversation and summarize or drop earlier turns.

Hierarchical context: Keep a high-level summary always in context, with detailed information loaded on demand.

Short-Term Memory: Task State

Beyond conversation history, agents benefit from structured task state:

class TaskState:
    """Structured state for the current task."""

    goal: str                          # The original task description
    plan: list[str]                    # Current plan steps
    completed_steps: list[str]         # Steps already completed
    current_step: int                  # Index of the current step
    files_modified: list[str]          # Files changed during this task
    errors_encountered: list[str]      # Errors seen and how they were resolved
    key_findings: dict[str, str]       # Important discoveries about the codebase
    test_results: list[TestResult]     # Results of test runs

This structured state is more efficient than raw conversation history because it distills information into a compact, queryable format.

Long-Term Memory: Knowledge Bases

For agents that work on the same codebase repeatedly, long-term memory provides significant benefits:

Project knowledge base: A persistent store of information about the project---its architecture, conventions, key files, and common patterns. This prevents the agent from re-exploring the same codebase on every task.

Decision log: A record of past decisions and their outcomes. If the agent previously tried approach A and it failed, it should not try the same approach again.

User preferences: The developer's coding style, preferred libraries, naming conventions, and other preferences that should be consistent across tasks.

# Example: Project knowledge base entry
{
    "project": "my-web-app",
    "architecture": "Flask REST API with SQLAlchemy ORM",
    "test_framework": "pytest",
    "key_directories": {
        "routes": "src/routes/",
        "models": "src/models/",
        "tests": "tests/"
    },
    "conventions": {
        "naming": "snake_case for files and functions",
        "imports": "absolute imports preferred",
        "docstrings": "Google style"
    },
    "known_issues": [
        "The database connection pool sometimes exhausts under load"
    ]
}

Callout: Memory in Claude Code

Claude Code uses several memory mechanisms. The conversation history within a session provides working memory. The CLAUDE.md file (sometimes called a "project memory" file) serves as long-term memory, storing project-specific context, conventions, and instructions that persist across sessions. This file is read at the start of every interaction, giving the agent immediate access to accumulated project knowledge without consuming context window space on re-exploration. You can create your own CLAUDE.md files to preconfigure agent behavior for your projects.

36.8 Error Recovery in Agent Workflows

Why Errors Are Inevitable

In agent workflows, errors are not exceptions---they are the norm. Agents interact with complex environments where things frequently go wrong:

• Files have moved or been renamed
• Tests fail for unexpected reasons
• Dependencies have version conflicts
• APIs return unexpected responses
• The agent's own generated code contains bugs

Robust error recovery is what separates a useful agent from a frustrating one.

Retry Strategies

The simplest error recovery is retrying the failed action:

Immediate retry: Try the same action again immediately. Useful for transient failures like network timeouts.

Retry with modification: Modify the approach slightly before retrying. If a file write fails because the directory does not exist, create the directory first and then retry.

Exponential backoff: Wait increasingly longer between retries. Useful for rate-limited APIs.

async def retry_with_backoff(
    action: Callable,
    max_retries: int = 3,
    base_delay: float = 1.0
) -> Any:
    """Retry an action with exponential backoff."""
    for attempt in range(max_retries):
        try:
            return await action()
        except TransientError as e:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt)
            await asyncio.sleep(delay)

Fallback Plans

When retrying does not work, the agent needs alternative strategies:

Alternative tool: If one approach fails, try a different one. If grep does not find the pattern, try a broader search or look in different directories.

Simplified approach: If the complex approach fails, try a simpler one. If a multi-file refactor causes cascading failures, try a more incremental approach.

Decomposition: If a large action fails, break it into smaller steps that are more likely to succeed individually.

Graceful Degradation

When the agent cannot fully complete a task, it should degrade gracefully rather than failing silently or producing incorrect results:

class AgentResult:
    """Result of an agent task with status information."""

    status: str          # "complete", "partial", "failed"
    completed_steps: list[str]
    remaining_steps: list[str]
    errors: list[str]
    output: str
    suggestions: list[str]  # What the human might do to complete the task

A partially completed task with clear documentation of what was done, what remains, and what went wrong is far more valuable than an opaque failure.

Error Classification

Not all errors are equal. Agents should classify errors to determine the appropriate response:

• Transient errors (network timeouts, rate limits): Retry with backoff
• Configuration errors (missing dependencies, wrong paths): Attempt to fix the configuration
• Logic errors (generated code has bugs): Analyze the error and regenerate
• Permission errors (access denied): Escalate to human
• Fundamental errors (impossible task, contradictory requirements): Report to human with explanation

Callout: The Self-Healing Agent

The most powerful pattern in agent error recovery is the self-healing loop: the agent writes code, runs it, detects errors in the output, analyzes the error message, fixes the code, and reruns it. This is remarkably similar to how human developers work. The key insight is that error messages are information, and a capable agent can use that information to diagnose and fix problems. Claude Code uses exactly this pattern: when a test fails, it reads the failure output, identifies the issue, modifies the code, and reruns the test.

36.9 Evaluating Agent Performance

Why Evaluation Matters

As coding agents become more powerful, it becomes critical to measure their performance systematically. Without evaluation, you cannot:

• Compare different agent implementations or configurations
• Identify regressions when the agent is updated
• Determine whether the agent is ready for a given level of autonomy
• Justify the cost of agent tooling to stakeholders

Metrics for Coding Agents

Task Completion Rate: What percentage of tasks does the agent complete successfully? This is the most fundamental metric but requires careful definition of "success."

Code Quality: Does the generated code meet quality standards? - Passes all existing tests (no regressions) - Passes new tests specific to the task - Meets linting and formatting standards - Follows project conventions - Is free of security vulnerabilities

Efficiency: - Number of loop iterations to complete a task - Number of tokens consumed - Wall-clock time - Number of tool calls - Cost in dollars

Accuracy: - How often does the agent correctly identify the right files to modify? - How often does the agent's first attempt compile and pass tests? - How often does the agent introduce new bugs while fixing existing ones?

Safety: - How often does the agent attempt actions outside its allowed scope? - How often do guardrails trigger? - Has the agent ever caused data loss or security breaches?

Benchmarks

Several benchmarks have been developed to evaluate coding agents:

SWE-bench: A benchmark of real-world GitHub issues paired with their actual fixes. The agent receives the issue description and the full repository and must produce a patch that resolves the issue. This is considered the gold standard for evaluating autonomous coding agents.

HumanEval and MBPP: Benchmarks for code generation from function signatures and docstrings. While originally designed for code completion, they can be adapted for agent evaluation.

Custom benchmarks: For your specific codebase, create a benchmark suite of representative tasks with known correct solutions. This is the most relevant evaluation for your particular use case.

Evaluation Methodology

A rigorous evaluation framework includes:

1. Test suite: A diverse set of tasks spanning different difficulty levels and types (bug fixes, features, refactors, documentation)
2. Ground truth: Known correct solutions for each task, or at minimum, a test suite that the solution must pass
3. Metrics collection: Automated collection of completion rate, code quality, efficiency, and safety metrics
4. Statistical analysis: Run multiple trials to account for the non-determinism of language models
5. Regression testing: Compare new agent versions against previous ones to detect regressions

Callout: The Cost-Quality Tradeoff

Agent evaluation reveals a fundamental tradeoff: more capable agents (using larger models, more iterations, more tools) tend to produce higher-quality results but at greater cost. A simple agent might complete 60% of tasks at $0.10 per task, while a sophisticated agent might complete 90% at $2.00 per task. The right choice depends on the value of each task and the cost tolerance of your organization. This tradeoff is one of the most important practical considerations in agent deployment.

36.10 Building Your First Coding Agent

Architecture Overview

Let us now build a simple but functional coding agent from scratch. Our agent will be able to:

1. Receive a coding task described in natural language
2. Plan an approach
3. Read files from the project
4. Write or modify files
5. Run commands (tests, linters)
6. Observe results and iterate
7. Report completion or failure

The architecture consists of four components:

┌─────────────────────────────────────────┐
│              Agent Controller            │
│  (Plan-Act-Observe Loop)                │
├─────────────────────────────────────────┤
│              LLM Interface              │
│  (API calls to Claude/GPT/etc.)         │
├─────────────────────────────────────────┤
│              Tool Registry              │
│  (File I/O, Terminal, Search)           │
├─────────────────────────────────────────┤
│              Guardrails Layer           │
│  (Permissions, Limits, Validation)      │
└─────────────────────────────────────────┘

Step 1: Define the Tools

First, we define the tools our agent can use:

from dataclasses import dataclass
from typing import Any

@dataclass
class Tool:
    """A tool that the agent can use."""
    name: str
    description: str
    parameters: dict[str, Any]
    function: callable

def read_file(path: str) -> str:
    """Read and return the contents of a file."""
    with open(path, 'r') as f:
        return f.read()

def write_file(path: str, content: str) -> str:
    """Write content to a file."""
    with open(path, 'w') as f:
        f.write(content)
    return f"Successfully wrote {len(content)} characters to {path}"

def run_command(command: str) -> str:
    """Execute a shell command and return the output."""
    import subprocess
    result = subprocess.run(
        command, shell=True, capture_output=True, text=True, timeout=60
    )
    output = result.stdout
    if result.stderr:
        output += f"\nSTDERR:\n{result.stderr}"
    return output

Step 2: Build the Agent Loop

The core agent loop orchestrates the plan-act-observe cycle:

class CodingAgent:
    """A simple coding agent with a plan-act-observe loop."""

    def __init__(self, tools: list[Tool], max_iterations: int = 20):
        self.tools = {tool.name: tool for tool in tools}
        self.max_iterations = max_iterations
        self.history = []

    def run(self, task: str) -> str:
        """Execute a coding task autonomously."""
        self.history.append({"role": "user", "content": task})

        for iteration in range(self.max_iterations):
            # Get the next action from the LLM
            response = self.call_llm(self.history)

            if response.is_complete:
                return response.final_answer

            if response.tool_call:
                # Execute the tool
                result = self.execute_tool(response.tool_call)

                # Add the result to history (observe)
                self.history.append({
                    "role": "tool",
                    "content": result
                })

        return "Maximum iterations reached. Task incomplete."

Step 3: Add Guardrails

Wrap the tool execution with safety checks:

class GuardedAgent(CodingAgent):
    """An agent with safety guardrails."""

    BLOCKED_COMMANDS = ["rm -rf /", "sudo", ":(){:|:&};:"]
    ALLOWED_DIRECTORIES = ["./src", "./tests", "./docs"]

    def execute_tool(self, tool_call):
        """Execute a tool with safety checks."""
        # Check command safety
        if tool_call.name == "run_command":
            for blocked in self.BLOCKED_COMMANDS:
                if blocked in tool_call.args["command"]:
                    return f"BLOCKED: Command contains '{blocked}'"

        # Check file path safety
        if tool_call.name in ("read_file", "write_file"):
            path = tool_call.args["path"]
            if not any(path.startswith(d) for d in self.ALLOWED_DIRECTORIES):
                return f"BLOCKED: Path '{path}' is outside allowed directories"

        # Execute the tool
        return super().execute_tool(tool_call)

Step 4: Run the Agent

# Create tools
tools = [
    Tool("read_file", "Read a file", {"path": "string"}, read_file),
    Tool("write_file", "Write a file", {"path": "string", "content": "string"}, write_file),
    Tool("run_command", "Run a command", {"command": "string"}, run_command),
]

# Create and run the agent
agent = GuardedAgent(tools=tools, max_iterations=20)
result = agent.run(
    "Add a function to src/utils.py that validates email addresses "
    "using a regex pattern. Include type hints and a docstring. "
    "Then write a test in tests/test_utils.py."
)
print(result)

What This Agent Can Do

Even this simple agent demonstrates the core principles:

• Plan-act-observe: It reasons about the task, takes actions, and observes results
• Tool use: It reads files, writes files, and runs commands
• Iterative refinement: It can observe test failures and fix them
• Guardrails: It blocks dangerous commands and restricts file access

What This Agent Cannot Do (Yet)

A production agent would need much more:

• Sophisticated planning and task decomposition
• Context window management for large codebases
• Persistent memory across sessions
• Human-in-the-loop approval workflows
• Robust error recovery with multiple fallback strategies
• Cost monitoring and optimization
• Comprehensive logging and auditability

Building production-quality agents is an active area of development, and tools like Claude Code represent the state of the art. But understanding the fundamentals---the plan-act-observe loop, tool use, guardrails, and memory management---gives you the foundation to use, evaluate, and build agent systems effectively.

Callout: From Prototype to Production

The simple agent we built in this section is a pedagogical tool, not a production system. Production agents like Claude Code represent hundreds of engineering hours of work on reliability, safety, performance, and user experience. But every production agent is built on the same fundamental principles you have learned in this chapter. Understanding these principles makes you a more effective user of agent tools and prepares you for the agent-driven future of software development.

Chapter Summary

In this chapter, you have explored the architecture and practices behind AI coding agents:

• From assistant to agent: You understand the spectrum from simple chatbots to fully autonomous agents, and the architectural differences at each level.
• The agent loop: You can explain and implement the plan-act-observe cycle that drives agent behavior.
• Tool use: You understand how function calling enables agents to interact with codebases and development environments.
• Autonomous workflows: You can identify and design common workflow patterns like issue-to-PR and test-driven development.
• Guardrails: You know how to implement permission systems, sandboxing, cost controls, and output validation to keep agents safe.
• Human-in-the-loop: You can design approval gates, review checkpoints, and escalation patterns that balance automation with oversight.
• Memory management: You understand working memory, short-term memory, and long-term memory strategies for agents.
• Error recovery: You can implement retry strategies, fallback plans, and graceful degradation.
• Evaluation: You know how to measure agent performance using task completion rate, code quality, efficiency, and safety metrics.
• Building agents: You have built a simple coding agent from scratch and understand the gap between prototypes and production systems.

The shift from AI assistants to AI agents represents a fundamental change in how software is developed. By understanding the principles behind this shift, you are prepared to leverage agent technology effectively while maintaining the oversight and safety practices that professional software development demands.

Callout: Looking Ahead

In Chapter 37, you will explore custom tools and MCP (Model Context Protocol) servers, learning how to extend agent capabilities by building your own tools. This directly builds on the tool-use concepts from this chapter and opens the door to creating agents tailored to your specific workflows and codebases.