How to Govern Artificial Intelligence Agents

During the early years of generative artificial intelligence, we became accustomed to interacting with systems capable of summarizing documents, drafting texts, and proposing solutions. The user made a request, the model generated a response, and the interaction ended on the screen.

Agents change that relationship because they do not merely explain what should be done. A chatbot can describe the steps required to cancel an order; an agent can identify the purchase, verify the return conditions, request authorization, carry out the cancellation, and confirm the refund.

The difference may seem small, but it changes the nature of the problem. When the system only produces text, the main risk lies in the quality of the response. When it can act, permissions, identity, security, traceability, cost, and responsibility for the consequences all come into play. This is why the business challenge is not simply to build more capable agents, but to decide how much authority they should be given, under what conditions they may exercise it, and how control can be maintained when they intervene in real-world processes.

What an Artificial Intelligence Agent Really Is

There is no single definition, but there is a practical one: an agent is a software system capable of interpreting a goal, deciding which steps to follow, using tools, and adapting its behavior according to the results. Autonomy does not have to be complete: an agent can prepare a bank transfer and leave the final approval in the hands of an authorized person.

It is useful to distinguish four concepts that are often confused. A foundation model is trained on large amounts of information and can be adapted to a wide range of tasks. A language model is a type of foundation model specialized in processing and generating language. An assistant combines that model with instructions, an interface, and sometimes access to additional information. An agent adds the ability to select tools, execute actions, and continue working until it reaches the objective or encounters a condition that requires it to stop. The difference does not lie in how fluently it writes, but in what it can do beyond the conversation.

And one historical note that is often repeated incorrectly: ELIZA, the program published by Joseph Weizenbaum in 1966, was a pioneer of natural-language conversation, but it was not an agent. It worked through pattern matching and scripted responses. Its real lesson is a different one: it demonstrated how easily we attribute understanding to a machine that imitates conversation.

The Anatomy of an Agent

Although implementations vary, an enterprise agent typically combines six components.

The goal and instructions. “Manage returns” is too broad. An operational definition specifies which products can be returned, within what time frame, which data may be accessed, which exceptions require review, and which actions need human authorization. The goal is not simply an opening sentence: it is a behavioral specification.

The model. It interprets the request and proposes the next step based on statistical patterns and the context it receives. It does not understand the process in the same way a professional would, which is why a plausible response does not guarantee that the proposed action is correct.

The tools. These are functions that allow the agent to interact with other systems: an API, a database, a search engine, or a business application. The agent should not be given unrestricted access to the infrastructure. Someone who only needs to consult invoices should have read access, not permission to modify accounts.

Context and memory. Context contains the information available during a specific execution; memory preserves information across interactions. The more an agent remembers, the greater the exposure to personal, confidential, or outdated information. Memory therefore requires policies for consent, retention, user isolation, and deletion.

The execution loop. Interpret, select an action, execute it, observe the result, and decide on the next step. This is what allows an agent to handle processes that cannot be solved with a single response — and it is also what allows an initial mistake to propagate across several steps.

Evaluation and boundaries. Format checks, business rules, cost limits, maximum numbers of attempts, and conditions for escalating to a person. When the impact is significant, evaluation cannot be entrusted to the same model that generated the action: independent checks and deterministic rules are required. A bank transfer is not valid simply because the model says the amount “looks reasonable”.

Level	Capability	Example	Recommended Control
Consultation	Retrieves and summarizes information	Search for an internal policy	Access control and client review
Recommendation	Proposes a decision	Prioritize an incident	Human validation
Preparation	Completes steps without executing the final action	Prepare a bank transfer	Approval before execution
Limited Execution	Acts within predefined rules	Replenish inventory below a threshold	Limits, logging, and rollback
Extended Autonomy	Plans and coordinates multiple actions	Resolve complete incidents	Continuous monitoring and strict stop conditions

Where the Value Emerges

Three areas currently concentrate the strongest use cases.

Customer experience. Rufus, Amazon’s shopping assistant, combines models, product catalog data, purchase history, memory, and tools. Amazon states that more than 250 million customers used it in 2025 and that shoppers who incorporate it into their purchasing journey are more than 60% more likely to complete a purchase. These are metrics reported by the company itself, not an independent evaluation — but they illustrate where the value is heading: not toward better conversations, but toward reducing steps and completing a specific need.

Automation of complex processes. Agents become especially valuable when the work requires interpreting unstructured information and making decisions across multiple sources. The Formula 1 and AWS assistant for investigating technical incidents reduced initial classification time from more than a day to under twenty minutes and cut overall resolution time by as much as 86%, according to figures published by the organizations involved. The useful lesson is not the percentage itself: the project began by redesigning the diagnostic process and only then introduced the technology.

Software development. Agents can analyze repositories, generate code, run tests, and propose fixes. But generating code is not the same as producing reliable software: human review, testing, vulnerability analysis, and version control remain necessary, and in legacy systems, business knowledge is irreplaceable.

Not Every Problem Needs an Agent

The popularity of the term has led to conventional chatbots and automation systems being marketed as agents. Gartner refers to this practice as agent washing and predicts that more than 40% of agentic AI projects will be canceled by the end of 2027 because of rising costs, unclear value, or insufficient risk controls. These are forecasts, not established facts, but they point to something that can already be verified: routine and predictable processes are often better handled with conventional software. If an operation always follows the same five steps, adding a probabilistic model increases cost and reduces reliability. Agents make sense where there is variability, unstructured information, and decisions that cannot be encoded as a fixed sequence — and even then, critical operations should remain governed by deterministic rules.

In fact, much of what is presented as “agentic” is really a workflow with models embedded inside it: the sequence is defined by the developer, and the model does not choose the path. Anthropic makes exactly this distinction between workflows and agents, and recommends starting with the simplest possible solution and adding autonomy only when it demonstrably improves the outcome. Intermediate patterns — chaining steps, routing requests to specialized processes, parallelizing independent tasks, reviewing outputs with an evaluator, or using decision-and-action loops such as ReAct — cover most real-world cases without requiring full autonomy.

Two clarifications help avoid common misunderstandings. First, an agent’s “reflection” is not learning. Methods such as Reflexion preserve written critiques in memory and reuse them later, but the model’s parameters do not change; a system may correct a step during one session and repeat the same mistake the next day. Second, more agents do not necessarily produce better results. Anthropic measured in its own multi-agent research system that agents consumed roughly four times as many tokens as a standard conversation, while the multi-agent architecture consumed around fifteen times as many. This is a measurement from one specific system, not a universal law — but it shows that coordination has a cost, and that multi-agent architectures should be reserved for problems whose value justifies that overhead.

One final note on protocols, because they are also frequently confused: MCP connects an AI application to external tools, resources, and instructions; A2A, announced by Google in 2025 and later donated to the Linux Foundation, enables agents to communicate with one another. They are complementary, not competing approaches. And no protocol solves security by itself: authentication, permission limits, response validation, and operation logging are still required.

The Gap Between a Demo and Production

A demonstration is prepared with clean data, broad permissions, and a favorable path. Production brings unpredictable users, incomplete data, network failures, and actions with legal consequences. Bridging that gap requires solving five problems.

Identity. Knowing who initiates the operation, which agent is involved, and on whose behalf it is acting. If a person is not allowed to view their colleagues’ salaries, the agent they use should not be able to do so either: authorization must be preserved throughout the entire chain of tools.

Least-privilege permissions. Grant only what is strictly necessary for the task. A travel agent may be allowed to check calendars and prepare bookings, but it should require approval before making a purchase, with spending limits and a list of authorized providers. Platforms exist that help isolate executions and manage identities — Amazon Bedrock AgentCore is one example in this category — but none removes the responsibility of designing access policies properly.

Observability. Being able to reconstruct what the system did: what information led to each decision, which permissions it used, how much it cost, which step failed, and who approved it. A log that stores only the final response is insufficient for investigating incidents or demonstrating compliance. OpenTelemetry is working on conventions for consistently recording model calls, tokens, and tool interactions. Logging also introduces its own risks: traces may contain personal data and credentials, so they require their own access controls and retention policies.

Continuous evaluation. Models change, data is updated, and users discover unexpected ways to use systems, so evaluating only before launch is not enough. Evaluating only the final result is also insufficient: organizations need to examine the outcome, the trajectory — which decisions were made, which tools were used, and whether conditions were respected; a correct result obtained from an unauthorized source is not a success — as well as security, including permissions, data handling, and resistance to malicious instructions, and efficiency, including cost, latency, retries, and human intervention. Testing should include normal cases, edge cases, and adversarial inputs.

Cost control. Every reasoning step and every tool call consumes resources, and an agent trapped in a loop can multiply costs without improving the result. Systems therefore need limits on the number of steps, execution time, budget per task, and clear stopping conditions. The most powerful model is not always the best choice either: simple tasks can often be routed to smaller models or deterministic software.

Security Expands When the Model Can Act

A model can receive malicious instructions either directly from the user or indirectly from the documents it consults. This second form — indirect prompt injection — is particularly dangerous: an email may contain a hidden instruction telling the system to ignore its rules and send information to another address, and what a person sees as part of a document may be interpreted by the model as an instruction. NIST’s Generative AI Profile notes that these attacks can lead to unintended behavior, information disclosure, or code execution, and the risk increases when the model is connected to tools capable of taking action.

There is no single defense. Protection requires a combination of isolation, input and output validation, least-privilege permissions, separation between data and instructions, allowlists of authorized tools, adversarial testing, and human confirmation for sensitive actions. OWASP’s Top 10 for Agentic Applications provides a useful taxonomy of characteristic failures — including memory poisoning, privilege abuse, goal hijacking, and cascading failures — that can help organizations design appropriate tests and controls.

A Prudent Way to Start

Responsible adoption can be organized around five decisions. Choose a real, recurring, and measurable problem, with a baseline that allows performance before and after implementation to be compared. Grant the minimum autonomy necessary: begin with recommendations, move on to preparing operations, and allow execution only when testing demonstrates that the level of control is sufficient. Limit tools and permissions from the beginning, because adding restrictions later is always more difficult. Evaluate complete execution paths, not just answers that “look correct.” And prepare operations before expanding the scope: define accountable owners, supervision, incident response, rollback procedures, and criteria for withdrawing the system.

Intelligence That Acts Requires Accountability

Agents connect the ability to interpret language with the ability to intervene in digital processes. That combination can improve customer service, reduce repetitive work, and accelerate software development. But value does not come from autonomy itself: a more independent agent can also make mistakes across more steps, use more information, and create consequences that are harder to reverse.

The business question is not how many agents an organization can deploy, but which decisions should be delegated, which limits should remain in place, and how responsibility will be assigned when a machine acts on behalf of a person or an institution. Governing intelligence means exactly that: not preventing systems from acting, but designing the conditions under which they can do so with a clear purpose, proportionate permissions, effective oversight, and human accountability.

< Older Post Newer Post >

Contactar

How to Govern Artificial Intelligence Agents

From Conversation to Action: