AI agents are more than chatbots with tools. Before you build one, understand the core components - memory, planning, and tools - the levels of autonomy, and the pitfalls that derail most projects.

Everyone today is building AI agents (or saying they do :) ). According to LangChain's 2025 State of Agent Engineering report, over 57% of surveyed teams already have agents running in production, up from 51% the year before. But behind the hype, the engineering reality is more nuanced: most teams struggle with quality (32% cite it as the top production blocker), and many build agents for problems that don't actually need them. Before you write a single line of agent code, it's worth understanding what agents really are, what makes them different from standard LLM applications, and where things tend to go wrong.

This post covers the core concepts you need before building AI agents - the architecture, the levels of autonomy, and the practical pitfalls that separate a convincing demo from a reliable production system.

Basically, everything I wish we knew a few years back when we started building those to customers.

What Makes an Agent Different from a Chatbot

The fundamental difference between a chatbot and an agent is who controls the workflow. With a standard LLM application, the user sends a prompt, the model returns a response, and you're done. The developer defines the control flow - what happens, in what order, and when. An agent flips this: the LLM itself decides what to do next, which tools to call, and whether to loop back and try again.

Consider a concrete example. You ask a chatbot "write me an essay about renewable energy." It produces the essay in one shot. You review it, notice problems, ask for revisions - you're the one driving the loop. With an agent, you give it the same goal, and it autonomously searches the web for current data, writes a draft, evaluates whether it meets quality criteria, revises weak sections, and iterates until it's satisfied. The key shift is iterative, autonomous problem-solving rather than single-shot generation.

This distinction matters for engineering because it directly impacts cost, latency, and failure modes. A single LLM call costs fractions of a cent and returns in seconds. An agentic workflow might chain dozens of LLM calls, each with tool invocations, running for minutes and costing orders of magnitude more. And because each step carries some probability of error (hallucination, wrong tool selection, misinterpreted results), those errors compound across steps. A 5% error rate per step becomes a 40% chance of at least one error in a 10-step chain. This is why the first engineering question should always be: does this problem actually need an agent, or would a simpler approach work?

The Three Core Components of an Agent

Under the hood, every agent architecture has three core building blocks: memory, planning, and tools. Understanding each is essential to building agents that actually work.

Memory

Memory in agents works at multiple levels. Short-term memory is essentially the LLM's context window - the current conversation and task state. This is what chatbots already have. Long-term memory is external storage (vector databases, knowledge graphs, key-value stores) that persists across sessions, letting the agent recall past interactions, user preferences, or domain knowledge. Working memory - sometimes called a scratchpad - holds intermediate results during multi-step execution. Despite its importance, memory remains a weak point: most agents today function as "temporary chatbots without retention" across sessions unless you explicitly build the persistence layer.

Planning and Reasoning

Planning is what separates an agent from a chatbot with tools bolted on. It's the ability to take a high-level goal, decompose it into subtasks, execute them in the right order, and adapt when something goes wrong. Common patterns include ReAct (interleaving reasoning and action steps), Plan-and-Execute (create a full plan upfront, then run it), and Tree of Thoughts (exploring multiple reasoning paths). This is also where things most often go wrong - if the model goes down a rabbit hole or makes a bad plan, it can derail completely and burn through tokens without ever reaching the goal.

Tools

Tools are what let agents interact with the real world - APIs, databases, code execution environments, file systems, and even other agents. A travel agent that can reason about flights but can't actually query a flight API is just a chatbot with opinions. Tools bridge the gap between reasoning and action. Importantly, tool descriptions are written for the agent, not for you as a developer. If the description is vague or misleading, the agent will call the tool at the wrong time, with the wrong arguments, or not at all. Treat tool definitions as part of your prompt engineering, and test them accordingly.

It is also worth noting that a single agent can use multiple LLM models behind the scenes - a more capable model for planning and reasoning, a faster and cheaper one for tool execution or summarization. This multi-model approach is now common: LangChain's survey found that over 75% of teams use multiple models in their agent systems.

Levels of Autonomy: Not Every Agent Needs Full Control

One of the most important design decisions is how much autonomy to give your agent. Not every use case needs (or should have) a fully autonomous system. A useful framework defines five levels:

  • Level 1 (Operator): The agent assists on demand. The user drives the workflow and invokes the agent for specific subtasks. Think autocomplete or inline code suggestions.
  • Level 2 (Collaborator): Shared control. The agent proposes actions, the user can accept, modify, or override. Most coding assistants today (Copilot, Cursor) operate here.
  • Level 3 (Consultant): The agent plans and executes most tasks independently, consulting the user for expert input or high-risk decisions.
  • Level 4 (Approver): The agent operates independently and only asks the user to approve critical actions.
  • Level 5 (Observer): Fully autonomous. The user monitors logs and has an emergency stop, but doesn't participate in the workflow.

For most production business applications today, Level 2-3 is the sweet spot. To make this concrete, consider a travel assistant agent. A user says: "I want to fly to Italy - find the best time to buy a ticket and buy it for me." The agent plans its steps: check weather patterns, query flight price APIs, compare options. It executes these autonomously at Level 3 - no need to ask permission to hit a weather API. But when it's time to actually purchase the ticket (a financial commitment that's hard to reverse), it drops to Level 2 and asks the user to confirm. This is the right pattern: let the agent handle low-risk exploration autonomously, but require human approval for high-stakes actions. The level of autonomy is a design decision - and more autonomy is not always better. It involves tradeoffs in reliability, accountability, and cost.

What Goes Wrong: Pitfalls That Derail Agent Projects

Building a compelling agent demo takes a few hours. Getting it to production takes months. Here are the most common engineering pitfalls.

Building an agent when you don't need one. This is the most frequent mistake. If your workflow is a deterministic sequence of steps (parse invoice, validate, post to accounting), that's a pipeline, not an agent. Use direct API calls for classification, deterministic workflows for predictable sequences, and reserve agents for problems that genuinely require flexible, adaptive reasoning. Every additional agentic step compounds error rates.

Vague or broken tool definitions. The agent decides which tool to call based on the tool's description and parameter schema. If your description says "gets data" instead of "returns current weather conditions for a given city as JSON with temperature_celsius, humidity_percent, and conditions fields," the agent will misuse it. Treat tool contracts as first-class design work - include clear descriptions, structured return types, and handle edge cases (pagination, rate limits, error messages) gracefully.

No evaluation framework. Without automated regression tests, every prompt tweak or model upgrade risks silently breaking existing use cases. Yet only 52% of teams run offline evaluations, and just 37% do online evaluation in production. Build evaluation infrastructure early: define success criteria per task, track metrics (completion rate, tool selection accuracy, latency, cost per interaction), and review 30-100 real examples regularly. Don't rely solely on "LLM-as-judge" - it lacks the determinism needed for reliable evaluation. We've written in length before about why LLM-as-Judge often fails.

Missing observability. When an agent fails, you need to know why - which step went wrong, what the model was reasoning about, which tool returned unexpected results. Log every LLM call, tool invocation, and intermediate result. Tools like Langfuse, Opik, LangSmith, or Helicone make this tractable. Without observability, debugging agent failures is guesswork.

Key Takeaways

Before building an AI agent, step back and ask whether you actually need one. Many problems are better solved with simpler LLM patterns or deterministic workflows.

If you do need an agent, design with intention. Pick the right level of autonomy for your use case - most production systems should not be fully autonomous. Invest in tool definitions as seriously as you invest in prompts. Build evaluation and observability from day one, not as an afterthought. And remember that the journey from a working demo to a reliable production system is where the real engineering challenge begins.