How Do AI Agents Actually Work? Inside the Loop
There is a moment most people hit when they first use a real AI agent: they type a task, the agent starts doing things, and suddenly they wonder what is actually happening under the hood. It is not just autocomplete. It is not a single model call. Something more complicated is going on, and understanding it makes you much better at getting agents to work for you.
This guide explains how agents actually work, from the basic loop to memory, planning, tool use, and the reflection patterns that separate good agents from flaky ones. No PhD required, but we will not skip the parts that matter.
What makes an agent different from a plain LLM
Before getting into the loop, it helps to fix the distinction. A plain language model does one thing: it takes text in and produces text out. One call, one response, done. An agent wraps that model in a control structure that lets it take actions, observe results, and decide what to do next based on what happened.
The difference is iteration. An agent can run a loop. It can call a tool, see what the tool returned, reconsider its plan, call another tool, and eventually decide the task is done. That loop is the core of every AI agent architecture in production today.
If you want the fuller version of the definition, the what is an AI agent post covers that ground. And if you want to understand why the loop-based agent is categorically different from a one-shot LLM, ai agents vs llms is the comparison to read.
The basic loop
Strip everything away and every agent runs some version of this cycle:
- Receive a goal or task
- Think about what to do next
- Take an action (call a tool, write something, ask a clarifying question)
- Observe the result of that action
- Decide whether the goal is met or another step is needed
- Repeat from step 2 until done
Some agents run this loop invisibly, showing you only the final answer. Others, like Claude Code, show you each step as it happens: the plan, the tool calls, the outputs, and the reasoning behind each next move. Whether the loop is visible or hidden, it is always there.
The loop terminates in one of three ways: the agent decides the task is complete, it hits a predefined step limit, or it hands control back to the user with a question or a problem it cannot resolve on its own.
How planning fits in
Not every agent plans. Some operate entirely reactively, picking the next action based only on the current observation. But the more capable agents do something closer to actual planning before touching any tools.
Planning in this context means the model generates a sequence of intended steps before executing any of them. "First I'll check the current state of the codebase, then I'll identify which files need changing, then I'll write the tests, then I'll implement the code." That sequence can be explicit (literally written out) or implicit (kept in the model's context as a working sketch).
The value of upfront planning is error containment. If the model thinks through the sequence and spots a logical problem at step three, it can revise before wasting API calls on actions that were always going to fail. Agents without planning discover that kind of problem only after the fact.
The limitation is that plans go stale. The real world does not cooperate with step three exactly as anticipated. Good agents combine upfront planning with mid-task re-planning, updating their sequence when new information arrives.
Tool calling
Tools are what give agents reach beyond text. A tool is anything the agent can invoke and get a structured result back from: a web search, a code interpreter, a file system read, an API call, a database query, a browser.
From the model's perspective, tools are described in the system prompt or via the API as a list of available functions with their names, parameters, and expected return types. The model decides when to call a tool by generating a structured output (usually JSON) that names the function and passes the arguments. The agent's scaffolding then executes that call, captures the return value, and feeds it back into the model's context as an observation.
Here is a simplified version of what that exchange looks like at the protocol level:
Model output:
{
"tool": "read_file",
"args": { "path": "src/utils.py" }
}
Tool return:
{
"content": "def parse_date(s):\n ..."
}
Model sees: [tool result: content of utils.py]
Model output: "I see the parse_date function is missing timezone handling. I'll update it."The model never actually reads the file. It generates the instruction to read the file. The framework reads it and gives the content back. That distinction matters when you're debugging why an agent did something unexpected.
Memory: the four kinds
Memory in agents is more complicated than it sounds because there are actually four different things people mean when they say it.
In-context memory is the active conversation window. Everything the agent has seen and done in this session lives here. It is fast and accurate but limited by the context length of the model.
External memory is a database outside the model, usually a vector store or a key-value store, that the agent can query mid-task. The agent calls a retrieval tool, gets relevant chunks back, and adds them to its context. This is how agents appear to "remember" things from past sessions.
Procedural memory is less often discussed but important. It is the set of instructions, examples, and patterns baked into the system prompt or fine-tuned into the model itself. The agent knows how to format a PR description not because it looked it up, but because that knowledge is already part of how it reasons.
Episodic memory covers logs of past runs. Some orchestration systems store a record of what happened in previous sessions and surface relevant episodes when a similar task appears again.
Most production agents today use in-context memory plus an external retrieval store. The other two types are either fixed at setup time or not implemented at all.
Reflection and self-correction
One of the things that separates a capable agent from a brittle one is whether it checks its own work before moving on.
Reflection is the practice of the model reviewing what it just did and deciding whether it was correct. After calling a tool and getting a result, a reflective agent asks: "Does this output look right? Did I get what I expected? Does the result change my plan?" A non-reflective agent just appends the result to context and generates the next step without that internal check.
In practice, reflection is expensive. It adds tokens and latency. So most frameworks make it configurable or trigger it only at certain checkpoints, like after a tool call that returned an error, or after completing a subtask.
The difference in output quality is significant. Agents that reflect catch more of their own mistakes before they compound across multiple steps. A bug introduced at step two gets corrected at step three instead of propagating through to step eight.
The ReAct pattern
ReAct, which stands for Reasoning and Acting, is the most widely cited agent architecture pattern. The paper came out in 2022 and it changed how most practitioners think about the basic loop.
The key insight of ReAct is that the model should interleave its reasoning with its actions rather than separating them. Instead of "plan everything, then act," the model explicitly writes out its current thinking before each action and then after observing the result.
A ReAct trace looks something like this:
Thought: The user wants to know the current price of ETH.
I should use the price API to get a live quote.
Action: get_crypto_price(symbol="ETH")
Observation: {"price": 2847.12, "currency": "USD", "timestamp": "..."}
Thought: I have the current price. I can now answer the question.
Answer: ETH is currently trading at $2,847.What makes this work better than a pure planning approach is that each "Thought" step is informed by the latest observation. The reasoning stays grounded in what actually happened rather than drifting toward what the model expected to happen.
Most production agent frameworks, including LangChain's ReAct agent, LangGraph, and the OpenAI Assistants API, implement some variant of this pattern even when they do not explicitly call it ReAct.
Multi-step tasks in practice
Understanding the loop abstractly is one thing. Seeing it in a concrete example makes it click.
Here is roughly what happens when you tell an agent like Claude Code to "add input validation to the checkout form and write tests for it."
The agent starts by reading the relevant files: the checkout form component, any existing validation utilities, and the existing test file. That is three tool calls. It builds a picture of the current state from those reads.
Next, it plans the changes: a validation function, updates to the form component to call it, and a set of test cases that cover the main edge cases (empty input, invalid email format, too-long strings). It may write this plan out explicitly or hold it in context.
Then it executes: writes the validation function, edits the form component, writes the tests. Each write is a tool call. After writing the tests, it might run them and observe whether they pass. If a test fails because of a typo in the function name, it catches that, fixes it, and re-runs.
When everything passes, it summarizes what it did and stops. The whole sequence might be fifteen to twenty tool calls, but from the user's side it looks like one task completed.
That is the loop in practice: reading, planning, writing, observing, correcting, completing.
Why agents still fail
Knowing the architecture also tells you why agents break.
The most common failure mode is context loss on long tasks. The agent's in-context memory fills up, the earliest parts of the session get compressed or dropped, and the agent forgets what it already tried or what constraints it established early on. Good agent scaffolding actively manages the context window to prevent this.
The second common failure is tool reliability. If a tool returns an ambiguous or malformed result and the agent does not check for it, the error propagates silently until the final output is wrong in a way that is hard to trace back.
The third is underspecified goals. Agents are good at executing clear tasks and bad at inferring what you actually wanted when the instructions were vague. The more specific the task description, the more reliably an agent completes it without needing intervention.
None of this is unfixable. It is just the current state of the technology, and knowing the root causes means you can work around them.
What to read next
If you want to go deeper, a few directions from here. Understanding the frameworks that implement these patterns, LangGraph in particular, will show you how production-grade orchestration actually handles the loop, state management, and multi-agent coordination. Looking at specific agents like Claude Code will show you how these abstract patterns surface in a real tool you can use today.
The architecture is less mysterious than it first appears. It is a loop with a model at the center, tools it can call, memory it can read, and enough reflection to catch its own errors before they compound. Everything else is an implementation detail on top of that.