From Copilots to Autonomous Agents: Designing Safe Enterprise Agent Architecture

Created on 19 July 2026

Key takeaways

An enterprise agent is not merely a chatbot with API access. It is a probabilistic decision-making component embedded within a wider system of deterministic controls, identities, policies and accountable owners.
Use the least autonomous architecture that can solve the business problem. A conventional workflow is usually safer when the process, decisions and exceptions are already understood.
Never give a model raw standing authority. Agent permissions should be scoped to a specific user, purpose, resource, transaction, time window and risk level.
Human approval must be a technical control, not a decorative confirmation screen. Approval should bind to the exact proposed action, parameters, evidence and expiry time.
Production readiness depends on evaluation and operational control. Every agent requires traceability, behavioural evaluation, limits, rollback, incident response and an emergency mechanism that can revoke its ability to act.

Introduction

Enterprise interest in AI agents is shifting from experimentation towards production deployment. Unlike a conventional generative AI assistant, an agent can interpret a goal, gather information, construct or follow a plan, invoke external tools and change the state of business systems.

That distinction matters. Producing an inaccurate summary is undesirable. Sending an inaccurate payment instruction, changing a customer’s entitlement or disabling a production service is an entirely different class of failure.

The architectural challenge is therefore not simply how to connect a language model to more tools. It is how to embed a probabilistic component inside enterprise processes without allowing uncertainty to spread unchecked into security decisions, regulated records, financial transactions and operational systems.

NIST’s Generative AI Profile extends the AI Risk Management Framework to help organisations manage generative AI risks across design, development, use and evaluation. In February 2026, NIST also launched an AI Agent Standards Initiative focused on secure interoperability, agent identity and evaluation. The initiative is significant, but it also confirms that many agent-specific standards remain under development rather than fully mature.

The correct response is not to wait indefinitely. It is to build on mature enterprise architecture disciplines such as least privilege, policy enforcement, workflow management, identity federation, API security, zero trust, auditability and resilience, while treating agent-specific protocols and practices as an emerging layer.

Business context and problem definition

Most enterprise applications are designed around known functions. A customer relationship platform exposes specific operations. A workflow engine follows explicit paths. An integration service transforms and routes defined messages. Humans or deterministic rules decide which operation runs.

An AI agent changes the control model. The organisation provides a goal such as:

Investigate this service incident, identify the likely cause, propose a remediation and apply it if the operational risk is low.

The agent may need to decide:

which monitoring systems to query;
what historical incidents are relevant;
whether it requires another specialist agent;
whether the evidence is sufficient;
which remediation tool to invoke;
whether human approval is mandatory;
how to determine whether the action succeeded;
when to stop or escalate.

This flexibility is the source of both value and risk.

The agent can handle variation that would require a large number of workflow branches. It can interpret unstructured information and adjust its approach as new evidence appears. Yet its behaviour is not guaranteed to be identical across repeated executions. Model responses can vary depending on prompts, retrieved context, tool results and model versions. Traditional uptime and error-rate monitoring is therefore insufficient to assess agent quality or safety.

The enterprise problem has four dimensions:

Control: How can an organisation prevent an agent from exceeding its intended authority?

Evidence: How can it prove what information the agent used, what it decided, what action occurred and who was accountable?

Reliability: How can it manage hallucinations, incomplete plans, tool failures, loops and cascading errors?

Lifecycle governance: How can it approve, deploy, monitor, change and retire agents consistently across business units and platforms?

These are enterprise architecture concerns, not prompt-engineering details.

Core concepts and terminology

Terms such as assistant, copilot and agent are frequently used as marketing labels. An architecture needs more precise working definitions.

Concept	Primary behaviour	Control of execution path	Ability to change systems	Typical oversight
AI assistant	Answers questions or generates content	User controls each interaction	Usually none	User reviews output
Copilot	Assists a user inside a business task	Shared between application and user	Limited, normally user-confirmed	Human remains the operator
Deterministic workflow	Executes predefined process logic	Code, rules or workflow definition	Yes, within explicit steps	Exceptions routed to people
AI-enabled workflow	Uses models within predefined steps	Workflow controls the path	Controlled by workflow	Approval at defined gates
Autonomous agent	Selects steps and tools dynamically to achieve a goal	Model-led within imposed boundaries	Potentially significant	Risk-based supervision
Multi-agent system	Multiple agents collaborate or delegate tasks	Distributed or supervisor-led	Depends on each agent’s authority	Requires system-level governance

A useful architectural distinction is that workflows follow predefined code paths, while agents dynamically direct their processes and tool use. Workflows offer predictability for well-defined tasks; agents are more suitable when the required path cannot be fully anticipated. Agentic designs also tend to increase latency, cost and debugging complexity.

Agent orchestration

Agent orchestration coordinates planning, model calls, retrieval, tool invocation, approvals, state, retries and termination. It may be implemented as code, a state machine, a durable workflow, an agent framework or a managed platform.

Context

Context is the information available during a particular model invocation. It can include the user request, system instructions, retrieved documents, tool outputs, conversation history and policy constraints.

Memory

Memory is persisted information that can influence future executions. It may be short-lived session state, task history, user preferences, learned operational knowledge or summaries of prior interactions.

Memory should not be confused with an authoritative system of record. Model-generated memory is derived data and may be incorrect, outdated or poisoned.

Retrieval and knowledge services

Retrieval-augmented generation, commonly called RAG, selects relevant information from external sources and provides it to a model at runtime. The original RAG research combined model knowledge with an external retrievable knowledge store to improve knowledge-intensive generation and support more updateable information sources.

Enterprise knowledge services usually add access controls, metadata filtering, document provenance, retention, data classification and source citations around this basic idea.

Tools

A tool is a callable capability exposed to an agent. Examples include searching a knowledge base, reading an invoice, creating a support ticket, executing a database query or requesting a payment.

A tool definition is not a security boundary. The underlying service must independently authenticate and authorise every invocation.

Architecture principles

1. Use the least autonomy necessary

Begin with a deterministic process, then introduce model-driven decisions only where variation or unstructured information makes conventional logic impractical.

The burden of proof should sit with autonomy. A team should explain why an agent is safer or materially more effective than a workflow, not merely why an agent is more interesting.

2. Separate reasoning from authority

The model can recommend an action without possessing the permission to perform it. A policy enforcement and action gateway should independently decide whether the action is permitted, requires approval or must be denied.

3. Treat all external content as untrusted

Documents, webpages, emails, retrieved text, tool descriptions and messages from other agents can contain malicious or misleading instructions. Prompt injection occurs when crafted input changes model behaviour in unintended ways, while excessive agency allows manipulated or erroneous model output to produce harmful external actions.

4. Preserve end-to-end identity and delegation

An action should identify:

the human, process or system that initiated the goal;
the agent and runtime executing it;
the application or tool being called;
the authority delegated to the agent;
the policy decision that permitted the action.

The agent must not hide the initiating identity behind a shared technical account.

5. Constrain actions deterministically

Apply explicit controls such as:

allowed tools and operations;
parameter schemas;
transaction and financial limits;
rate limits;
maximum execution steps;
time and cost budgets;
data-access boundaries;
required approval levels;
prohibited destinations;
safe operating hours.

6. Make state explicit and recoverable

Long-running agents require durable state, checkpoints, idempotency and compensating actions. Relying on an in-memory conversation transcript is how one eventually creates a distributed mystery.

7. Design evaluation and observability together

An agent is not production-ready because it completed several demonstrations. It must be evaluated against representative, adversarial and failure scenarios, then monitored using the same success and risk criteria after deployment.

8. Ensure humans retain effective control

Human oversight must provide enough context and time to make a meaningful decision. A stream of vague approval prompts merely transfers liability to an exhausted employee.

Detailed reference architecture

The architecture separates the agent control plane, which interprets goals and coordinates work, from the integration and action plane, which interacts with enterprise systems.

The security and policy plane independently controls identity, delegated authority, approval and action execution. This prevents the model or orchestration framework from becoming its own authorisation authority.

The observability and assurance plane records both conventional telemetry and agent-specific information such as model selection, retrieved sources, tool choices, policy decisions, approvals and evaluation outcomes.

Governance spans all layers. Every production agent should have an accountable business owner, technical owner, data owner, risk classification and operational support model.

Explanation of the architecture components

Users, channels and initiating systems

Agents may receive goals through chat interfaces, embedded application experiences, voice channels, APIs, scheduled triggers or events.

Each request should be normalised into a structured goal containing:

initiating identity;
business purpose;
requested outcome;
permitted scope;
relevant resource identifiers;
sensitivity and risk classification;
completion and escalation criteria.

Natural-language instructions should not be the only record of authority.

Agent access gateway

The access gateway establishes the session and performs initial controls such as authentication, rate limiting, content scanning, request-size limits and tenant isolation.

It should also protect against attempts to smuggle credentials, secrets or prohibited data into prompts.

Agent orchestrator

The orchestrator controls the execution lifecycle. Depending on the design, it may:

load an approved agent definition;
create a task and correlation identifier;
request a plan from a model;
follow a predefined state machine;
select tools;
persist checkpoints;
manage retries and timeouts;
suspend execution for approval;
terminate when the goal is complete or limits are reached.

The orchestrator should not directly execute unrestricted code or call arbitrary endpoints discovered during reasoning.

Model gateway and model services

A model gateway provides a consistent interface to approved models. It can enforce:

model allowlists;
region and residency requirements;
routing based on task and data classification;
token and cost limits;
request and response filtering;
fallback rules;
model-version pinning;
provider logging and retention settings.

Different models may be selected for planning, extraction, classification, summarisation or evaluation. The most capable model is not automatically the correct model for every step.

Prompt and instruction registry

System instructions, tool descriptions and prompt templates should be managed as versioned software artefacts.

The registry should record ownership, approval status, test results, model compatibility and deployment history. Prompts embedded invisibly across application code make impact analysis and incident response unnecessarily painful.

Prompts should describe behaviour, but security controls must remain outside the prompt. OWASP explicitly warns against depending on system prompts to enforce controls because prompts can leak or be altered through other attacks.

Context, memory and knowledge services

The context manager assembles the minimum information required for each step. It should apply data classification, access filtering, redaction, provenance and token budgets.

Memory should be divided into clear categories:

Memory type	Purpose	Recommended controls
Working memory	Current execution state	Short retention, task isolation
Conversation memory	Interaction continuity	User-visible correction and deletion
Episodic memory	Prior task outcomes	Provenance, quality scoring, expiry
Semantic memory	Reusable facts or knowledge	Curated sources, ownership, refresh
Procedural memory	Reusable methods or plans	Version control and formal approval

Never allow unverified model output to become durable organisational knowledge automatically. Memory writes should be filtered, attributed and, for important domains, reviewed.

Tool registry and integration gateway

The tool registry is an approved catalogue of agent-accessible capabilities. Each tool should define:

business owner;
technical owner;
data classification;
input and output schemas;
read or write behaviour;
risk tier;
required scopes;
approval rules;
rate and transaction limits;
reversibility;
support arrangements;
expected audit events.

The integration gateway validates every tool call and isolates agents from underlying credentials. It should support schema validation, safe parameterisation, endpoint allowlists, output sanitisation and idempotency controls.

Model Context Protocol, or MCP, is an emerging open protocol for connecting language-model applications to tools and contextual resources. The November 2025 specification defines an authorisation framework and standardised tool, resource and prompt interfaces. Its own security guidance covers confused-deputy risks, token passthrough, session hijacking, server-side request forgery and scope minimisation. MCP can improve portability, but adopting a protocol does not remove the need for enterprise gateways, trust registries and policy enforcement.

APIs, events and workflow engines

Agents should reuse established integration capabilities rather than create a parallel universe of direct connectors.

API gateways provide synchronous request controls. Event platforms support asynchronous communication and decoupling. CloudEvents offers a mature CNCF specification for representing event metadata consistently across services and platforms.

Workflow engines remain essential when steps, approval paths and compensating transactions must be deterministic. BPMN 2.0.2 is a mature Object Management Group specification for modelling business processes.

A strong design often uses an agent to interpret and investigate, then hands execution to an established workflow.

Identity and delegated authority

Agents require at least two identities:

a workload identity representing the deployed agent runtime; and
an on-behalf-of identity context representing the user or process whose authority is being exercised.

NIST has identified agent identity and authority as a specific area requiring further standards and demonstration work. Its 2026 concept paper highlights the risks created when agents access diverse datasets, tools and applications.

Use short-lived, narrowly scoped credentials rather than static API keys. OAuth 2.0 Token Exchange can support delegated token patterns, while RFC 9700 provides current OAuth 2.0 security best practices. Agent-specific OAuth profiles remain an emerging standards area and should not be mistaken for settled practice.

For service-to-service identity, SPIFFE defines short-lived cryptographic workload identities that can operate across heterogeneous environments and organisational boundaries.

A delegated authority grant should specify:

who delegated the authority;
which agent may use it;
permitted operations and resources;
purpose and transaction context;
maximum value or impact;
expiry;
whether onward delegation is allowed;
approval requirements;
revocation status.

Policy enforcement and human approval

Policy should be expressed and tested as code where practical. A general-purpose policy engine such as Open Policy Agent can separate policy decisions from application enforcement and evaluate structured context at runtime. This is a vendor-neutral pattern, not a requirement to use a particular product.

A policy decision might evaluate:

user role and authentication strength;
agent identity and approved version;
tool risk tier;
target resource;
data sensitivity;
proposed parameters;
transaction amount;
confidence and evidence quality;
current incident state;
jurisdiction and time;
previous actions in the same task.

Human approval should bind to an immutable action proposal. The approver should see the exact action, target, parameters, supporting evidence, expected effect, rollback method and expiry time.

Approval for “the agent to continue” is too vague.

Observability, evaluation and audit

Agent observability should capture:

task and trace identifiers;
initiating identity;
agent, prompt and model versions;
retrieved source identifiers;
model latency and token usage;
tool selections and parameters;
policy decisions;
approval records;
action results;
retries and errors;
safety interventions;
final outcome;
cost.

OpenTelemetry’s generative AI semantic conventions are intended to standardise signals such as model calls, token counts, tool calls and tool results. Full prompt and completion content may also be captured when explicitly enabled, but doing so introduces privacy and security concerns.

Do not place secrets, personal data or confidential documents indiscriminately into observability systems. Content logging should be risk-based, redacted and access-controlled.

Evaluation must include more than final-answer quality. It should assess:

goal completion;
factual grounding;
tool-selection accuracy;
policy compliance;
approval correctness;
resistance to prompt injection;
behaviour under partial failures;
termination behaviour;
repeatability;
unacceptable-action rate;
recovery and rollback success.

Infrastructure and cloud platforms

Agent workloads should use standard production controls including network segmentation, secrets management, encryption, vulnerability management, software supply-chain controls, environment separation and infrastructure policy.

NIST’s zero-trust architecture emphasises resource protection, continuous evaluation and minimum necessary privilege rather than implicit trust based on network location. Those principles apply directly to agent runtimes and tool access.

Emergency shutdown

Every production agent needs controls that can:

stop new executions;
suspend active tasks;
revoke delegated tokens;
disable high-risk tools;
isolate a model or connector version;
block outbound destinations;
switch to read-only operation;
revert to a deterministic workflow;
preserve evidence for investigation.

A kill switch implemented only in the chat interface is not a kill switch. Revocation must occur at the identity, policy or action-enforcement layer.

End-to-end goal and action flow

The process begins with authentication and creation of a bounded authority context. The agent retrieves only information the initiating identity and business purpose are permitted to access.

The model proposes a plan or next action. Policy enforcement evaluates the proposed operation independently. High-risk actions are suspended until an authorised person approves the exact transaction.

After execution, the agent verifies the result rather than trusting a successful API response blindly. It then records the evidence, decisions, approvals and system transaction identifiers required to reconstruct the outcome.

Security, privacy and governance considerations

Risk	How it appears	Architectural response
Prompt injection	Malicious instructions in user input, documents, webpages or tool responses	Treat content as data; isolate instructions; apply source trust, content filtering and tool policy
Tool abuse	Legitimate tool used with harmful parameters or frequency	Schema validation, allowlists, transaction limits, anomaly detection and approvals
Excessive agency	Agent has more functions or permissions than required	Least privilege, short-lived scopes, read/write separation and step limits
Data leakage	Sensitive data included in prompts, logs, memory or external requests	Classification, DLP, redaction, egress control, retention limits and residency enforcement
Hallucination	Agent invents facts, identifiers, policies or action results	Authoritative retrieval, citations, structured validation and post-action verification
Cascading failure	One agent’s error becomes input to other agents or systems	Trust boundaries, validation between stages, circuit breakers and bounded fan-out
Memory poisoning	Incorrect or malicious content becomes persistent memory	Provenance, write controls, review, expiry and separation from systems of record
Unclear accountability	No identifiable owner for an agent decision or action	Named business and technical owners, delegation records and decision logs
Identity confusion	Agent uses a shared credential or loses initiating-user context	Workload identity, on-behalf-of delegation and per-action authorisation
Uncontrolled cost	Loops, excessive retrieval or multi-agent chatter	Token, time, step, concurrency and financial budgets
Supply-chain compromise	Untrusted model, plugin, tool server or framework update	Approved registries, signing, dependency scanning, version pinning and isolation
Unsafe output handling	Model output is executed as code, query or command	Strict parsing, parameterisation, sandboxing and deterministic validation

OWASP’s 2025 LLM risk guidance covers prompt injection, sensitive-information disclosure, improper output handling, excessive agency and other risks directly relevant to agents. Its Agentic Security Initiative extends threat modelling towards autonomous systems. MITRE ATLAS provides a living knowledge base of adversary tactics and techniques for AI-enabled systems.

Governance alignment

Organisations should integrate agents into existing AI, security, privacy, data, model-risk and technology governance rather than inventing an isolated “agent committee”.

ISO/IEC 42001:2023 specifies requirements for establishing and continually improving an AI management system. ISO/IEC 23894:2023 provides AI-specific risk-management guidance. These standards establish governance processes, but organisations must translate their objectives into enforceable architecture controls and operating procedures.

Where the EU AI Act applies, an agent may form part of an AI system subject to obligations based on its intended purpose and risk classification. The Act includes requirements for areas such as risk management, logging, transparency and human oversight for high-risk systems. Most provisions apply from 2 August 2026, with some requirements applying earlier or under separate transition periods. Legal classification should therefore be performed for the complete use case, not merely for the underlying language model.

Architecture options and trade-offs

Option	Strengths	Limitations	Suitable scenarios
Deterministic workflow	Predictable, testable, auditable, efficient	Poor at handling unknown paths and unstructured ambiguity	Payments, entitlement changes, compliance steps, repeatable operations
AI-enabled workflow	Adds language and reasoning while retaining process control	Limited flexibility outside defined branches	Document processing, case triage, policy interpretation
Single agent	Simpler state, lower coordination overhead, clearer accountability	One context may become overloaded; broad permissions are tempting	Research, investigation, low-risk task execution
Supervisor with specialist agents	Clear task decomposition and specialised contexts	More latency, cost and failure paths	Complex investigations across several domains
Peer multi-agent system	Flexible collaboration and distributed problem-solving	Difficult governance, emergent behaviour, cascading errors	Experimental or highly complex problems with constrained tools
Agent plus deterministic execution workflow	Combines flexible interpretation with controlled transactions	Requires clean hand-off contracts	High-value enterprise operations

When conventional automation is safer

Use a conventional workflow, rules engine or application service when:

the process is known and stable;
decisions can be expressed reliably as rules;
the cost of a wrong action is high;
exact repeatability is required;
transaction volumes are large;
latency must be predictable;
regulators or auditors require explainable decision logic;
the process can be integrated through structured data;
exceptions are rare and can be escalated.

An AI agent should not replace deterministic automation merely because natural language is a more entertaining programming interface.

Single-agent versus multi-agent architecture

A single agent should be the default when one bounded context and one tool set can solve the task.

Introduce multiple agents only when there is a measurable benefit from:

separate specialist knowledge;
distinct permissions;
parallel work;
independent review;
context isolation;
organisational or trust boundaries.

Agent-to-Agent, or A2A, is an emerging open protocol designed to let independent agents discover capabilities, delegate tasks and exchange information without exposing internal memory or tools. As of July 2026, version 1.0.0 is published, but enterprise adoption patterns, governance conventions and security profiles are still maturing.

Build versus buy

Consideration	Build or open framework	Managed agent platform
Control and customisation	Usually greater	Limited to platform extension points
Initial delivery speed	Slower	Faster for supported patterns
Operational responsibility	Organisation carries most responsibility	Shared with provider
Portability	Potentially higher, if abstractions remain open	Risk of runtime and service lock-in
Governance integration	Must be engineered	May include built-in inventory and policy features
Transparency	Source and internals may be inspectable	Provider internals may be opaque
Upgrade management	Organisation controls timing	Provider may change services or models
Skills required	Higher engineering depth	Higher platform-specific knowledge
Cost model	Infrastructure and engineering cost	Consumption, licence and platform cost

Procurement should assess capabilities using tests and contractual evidence rather than feature checkboxes.

Enterprise use cases

Financial services: supervised transaction investigation

Business objective: Reduce the time required to investigate potentially fraudulent business payments.

Proposed architecture: A single investigation agent queries transaction history, customer records and threat intelligence through read-only tools. It produces an evidence-backed risk assessment and recommends whether to release, hold or escalate the payment. A deterministic case-management workflow performs the final status change.

Governance considerations: Customer data must remain within approved jurisdictions. Tool permissions should be read-only. The agent must cite every material fact and must not make the final regulated decision unless legal and model-risk reviews explicitly support that use.

Expected result: Faster evidence gathering and more consistent case summaries without granting the model direct payment authority.

Government: permit-application case assistant

Business objective: Help case officers process applications across complex legislation, policy and historical records.

Proposed architecture: An AI-enabled workflow retrieves current legislation, policy guidance and applicant documents. The agent identifies missing information, drafts an assessment and explains which sources support each conclusion. The official decision remains with an authorised officer.

Governance considerations: The architecture must preserve records, source versions, accessibility, procedural fairness and reasons for decisions. Personal information should be minimised in model telemetry and memory.

Expected result: Reduced administrative work while maintaining human accountability and an auditable decision trail.

Healthcare: clinical administration agent

Business objective: Coordinate non-diagnostic follow-up tasks after a patient discharge.

Proposed architecture: The agent reads the approved discharge plan, checks appointment availability and prepares follow-up actions. It may send low-risk reminders automatically but requires staff approval before changing clinical appointments or communicating sensitive instructions.

Governance considerations: Clinical records remain authoritative. The agent must not infer new medical advice, alter prescriptions or treat model-generated memory as clinical data. All patient communications require identity verification and approved templates.

Expected result: Better coordination and fewer missed administrative tasks without granting autonomous clinical decision authority.

Telecommunications: supervised network remediation

Business objective: Reduce restoration time for recurring network incidents.

Proposed architecture: A supervisor agent coordinates telemetry analysis, topology lookup and historical-incident retrieval. It proposes a remediation plan. Low-risk diagnostic commands may run automatically, while configuration changes are handed to a deterministic change workflow with policy checks, maintenance-window controls and rollback.

Governance considerations: Limit the number of devices and commands per task. Detect correlated incidents to prevent multiple agents from applying conflicting changes. Suspend automation during major events.

Expected result: Faster diagnosis and preparation of safe remediation while retaining deterministic control of network changes.

Agent maturity model

Level	Description	Permitted behaviour	Required evidence
1. Experimentation	Isolated proof of concept using synthetic or public data	No production actions	Basic functional tests and threat model
2. Assisted operation	Agent produces recommendations for human users	Read-only access; human performs actions	Quality evaluation, source grounding and user training
3. Supervised autonomy	Agent prepares or executes bounded actions with approval	Limited write access behind policy and approval	Adversarial testing, audit, rollback and operational support
4. Constrained autonomy	Agent independently performs predefined low-risk actions	Narrow authority, limits, monitoring and automatic shutdown	Demonstrated reliability, control effectiveness and incident exercises

This model deliberately stops at constrained autonomy. “Unrestricted autonomy” is not a responsible enterprise maturity target.

Promotion between levels should be based on evidence for a specific use case. An organisation may operate mature assisted agents while correctly refusing autonomous operation in high-risk domains.

Implementation roadmap

Phase	Main activities	Key outputs	Exit criteria
Discovery	Define objective, process, risk, users, data and alternatives	Use-case assessment, process model, risk classification	Agent architecture justified over simpler automation
Design	Define components, identity, policies, tools, approval and telemetry	Reference design, threat model, NFRs, data flows	Architecture and control owners approve design
Pilot	Use representative data in restricted environment	Evaluation dataset, test results, support procedures	Quality and risk thresholds achieved
Controlled rollout	Limit users, volumes, tools and autonomy	Production runbook, dashboards, incident integration	Stable operation within defined risk tolerance
Continuous improvement	Monitor drift, incidents, costs and business outcomes	Evaluation history, policy changes, improvement backlog	Continuing evidence that benefits exceed residual risk

Discovery

Start with the business process, not the agent platform. Document the current task, decision rights, exceptions, failure consequences and existing automation options.

Warning signs that the organisation is not ready include:

no accountable business owner;
undocumented processes;
poor source-data quality;
excessive shared accounts;
no API or integration governance;
inability to identify sensitive data;
no production AI evaluation capability;
no incident-response ownership;
pressure to grant broad permissions for convenience.

Design

Suggested architecture artefacts include:

context and goal model;
capability and tool catalogue;
data-flow diagram;
trust-boundary diagram;
agent interaction and sequence diagrams;
delegated-authority model;
policy catalogue;
approval matrix;
threat model;
misuse and abuse cases;
evaluation plan;
operational support model;
rollback and shutdown design;
architecture decision records.

Pilot

The pilot should use representative tasks, including difficult and adversarial examples. Tests should cover prompt injection, unavailable tools, conflicting sources, malformed results, approval rejection, expired credentials, retries and emergency shutdown.

Do not measure only successful demonstrations. Record every failure and near miss.

Controlled rollout

Limit the first production release by user group, transaction volume, data domain, tool set, time window and autonomy level.

Use feature flags and policy controls to expand capabilities gradually. Ensure human teams can handle the expected approval and exception volume before increasing adoption.

Continuous improvement

Re-evaluate agents after changes to models, prompts, tools, policies, knowledge sources or workflow logic. Monitor whether user behaviour and operating conditions have shifted away from the evaluation baseline.

Organisational roles and responsibilities

Role	Core responsibility
Business owner	Defines outcomes, accepts residual business risk and owns accountability
Product or service owner	Manages backlog, users, lifecycle and service performance
Enterprise or solution architect	Ensures architectural fit, boundaries, integration and control placement
AI or model owner	Manages model selection, evaluation, limitations and change
Data owner	Approves data use, quality, classification and retention
Security architect	Defines threat model, identity, policy, monitoring and incident controls
Privacy and legal teams	Assess lawful use, privacy, records and regulatory obligations
Platform engineering	Operates runtimes, gateways, CI/CD, secrets and observability
Process owner	Defines decision rights, exceptions and human approval paths
Operations or SRE	Monitors reliability, cost, incidents, rollback and capacity
Internal audit or assurance	Tests evidence, control operation and governance effectiveness

Responsibility may be shared. Accountability may not be assigned to the agent.

Non-functional requirements and success measures

Agent NFRs should include:

maximum end-to-end latency;
availability and recovery objectives;
maximum tool-call and model-call count;
cost per successful task;
accuracy and grounding thresholds;
prohibited-action rate;
approval response targets;
maximum privilege duration;
audit completeness;
rollback time;
residency and retention requirements;
accessibility;
concurrency and scaling;
termination guarantees;
maximum tolerable uncontrolled impact.

Key operational metrics include:

Category	Example metrics
Business	Completion rate, cycle-time reduction, user adoption, avoided manual effort
Quality	Grounded-answer rate, task success, false escalation, human correction rate
Safety	Policy denials, unsafe-action proposals, injection success rate, data-loss events
Autonomy	Actions by approval tier, intervention rate, autonomous completion rate
Reliability	Tool failures, loops prevented, retries, timeout rate, rollback success
Cost	Cost per task, tokens per successful task, tool and infrastructure consumption
Governance	Inventory coverage, overdue reviews, unowned agents, audit-record completeness

Optimising autonomous completion alone can reward unsafe behaviour. Measures must balance value, quality, risk and cost.

Procurement and technology-selection criteria

An enterprise agent platform or framework should be evaluated for:

model and deployment portability;
durable state and checkpointing;
tool and protocol support;
identity propagation;
fine-grained delegated authorisation;
external policy integration;
approval workflow support;
prompt and agent versioning;
evaluation and test automation;
complete traces and audit export;
data residency and retention controls;
tenant isolation;
emergency shutdown;
software supply-chain transparency;
upgrade and deprecation policy;
contractual incident notification;
model-provider terms for submitted data;
cost visibility;
exit and migration arrangements.

Require vendors to demonstrate controls using the organisation’s own abuse cases. A slide containing a shield icon is not control evidence, despite the technology industry’s long-running experiment to prove otherwise.

Common mistakes and anti-patterns

Giving agents permanent broad credentials

Standing credentials make every model error and injection attempt more dangerous. Use short-lived, task-specific authority.

Allowing arbitrary tool discovery in production

Dynamic discovery can introduce malicious or inappropriate capabilities. Use approved registries, signed metadata and risk classification.

Treating tool descriptions as trustworthy instructions

Tool metadata can itself be manipulated. Validate tool definitions and keep security decisions outside model-visible descriptions.

Relying on the model to approve its own actions

A second prompt asking “is this safe?” is not independent control. Use deterministic policy and, where required, accountable human approval.

Storing every interaction as memory

This creates privacy, poisoning and relevance problems. Persist only information with a defined purpose, provenance and retention period.

Building multi-agent systems before a single agent works

Multiple agents increase coordination overhead and failure paths. Complexity should follow measured need.

Logging everything without classification

Full prompts and tool results may contain credentials, personal data and confidential records. Observability needs its own data-governance design.

Measuring only final-answer accuracy

An agent can reach the correct result through an unacceptable process. Evaluate tool use, policy compliance, evidence, resource consumption and recovery behaviour.

No explicit stopping conditions

Every execution requires limits for time, steps, cost, retries, tool calls and fan-out.

No owner after deployment

Agent quality and risk change with models, data, tools and user behaviour. Production ownership cannot end when the project team celebrates its launch.

Architecture decision checklist

Before approving an enterprise agent for production, the Architecture Review Board should ask:

Business and scope

Is the business objective measurable?
Has a deterministic workflow or simpler AI-assisted design been considered?
Is the permitted scope explicit?
Are the consequences of failure understood?
Is there a named accountable business owner?

Agent behaviour

Is the distinction between model-led and deterministic decisions documented?
Are planning, retry and stopping conditions bounded?
Is agent memory necessary, controlled and reviewable?
Are multi-agent interactions justified?
Can the complete task state be recovered after failure?

Data and knowledge

Are authoritative sources identified?
Are access controls enforced before retrieval?
Is source provenance preserved?
Are personal, confidential and regulated data handled appropriately?
Can incorrect memory or indexed content be corrected or deleted?

Identity and authority

Does the agent have a unique workload identity?
Is initiating-user or process identity preserved?
Are permissions short-lived and task-specific?
Are transaction, purpose and resource limits enforced?
Can authority be revoked immediately?

Tools and actions

Is every tool registered, owned and risk-rated?
Are inputs schema-validated and outputs treated as untrusted?
Are write tools separated from read tools?
Are actions idempotent or reversible?
Are high-impact actions independently approved?

Security and resilience

Has prompt injection been tested across all external content channels?
Are tool abuse, data leakage and supply-chain risks addressed?
Are loops, cascading failures and uncontrolled fan-out bounded?
Are circuit breakers and emergency shutdown tested?
Can the service fall back to a safe manual or deterministic process?

Evaluation and operations

Is there a representative evaluation dataset?
Are quality, safety, cost and reliability thresholds defined?
Are agent, model, prompt, tool and policy versions traceable?
Are audit records complete enough to reconstruct every material action?
Are support, incident-response and rollback responsibilities assigned?

Governance and compliance

Is the agent registered in the enterprise AI inventory?
Has the use case been assessed against applicable law and policy?
Are human-oversight arrangements meaningful?
Are retention and evidence requirements defined?
Is periodic review triggered by material changes?

Production approval should be conditional on evidence, operational limits and an agreed autonomy level.

Governance lifecycle

The governance process should allow teams to reduce autonomy or stop a use case when evidence does not support production deployment. Ending an unsuitable experiment is a valid architecture outcome, even if it produces fewer dramatic conference slides.

Conclusion

The agentic enterprise will not be defined by the number of agents it deploys. It will be defined by whether those agents operate inside clear authority, data, policy and accountability boundaries.

A production agent should be treated as a governed enterprise workload with probabilistic decision capabilities, not as an independent employee and certainly not as an unaccountable digital executive. It requires identity, scoped delegation, controlled tools, durable state, policy enforcement, meaningful approval, evaluation, observability, audit and emergency shutdown.

The safest adoption path is incremental:

use models to assist people;
allow read-only investigation;
introduce supervised, bounded actions;
automate only low-risk actions that have demonstrated acceptable performance;
retain deterministic controls around material business outcomes.

Models can decide how to pursue a goal within a constrained space. The enterprise must continue to decide which goals are legitimate, which resources may be used, which actions are acceptable and who remains accountable when the result affects the real world.

That is the difference between adding agents to an organisation and architecting an agentic enterprise.

Frequently asked questions

Is an AI agent simply an LLM connected to tools?

No. The model and tools are only part of the system. A production agent also requires orchestration, state, identity, authority, policy, evaluation, observability, audit and operational controls.

Should every enterprise AI use case become an agent?

No. Many use cases are better served by retrieval, a single model call, an AI-enabled workflow or conventional automation. Agents are appropriate when the path genuinely requires dynamic interpretation and adaptation.

Are multi-agent systems more reliable than single agents?

Not automatically. Specialisation or independent review can improve some tasks, but multi-agent systems introduce more communication, cost, state, permissions and failure paths. They should be adopted only when tests demonstrate a clear benefit.

Can human approval make any agent safe?

No. Approval can reduce risk only when the approver understands the proposed action and has enough context, authority and time to challenge it. Approval fatigue and vague confirmation screens weaken the control.

Can prompts enforce security policy?

Prompts can guide behaviour but should not enforce permissions. Authorisation, data controls, financial limits and prohibited actions must be implemented through deterministic services.

Should an agent have its own identity?

Yes. The runtime needs a workload identity, while actions should also preserve the identity and authority of the user or process on whose behalf the agent operates.

Is MCP required for enterprise agents?

No. MCP can standardise connections to tools and contextual resources, but conventional APIs, events and integration platforms remain valid. MCP is an emerging interoperability option, not a replacement for enterprise security and integration governance.

How should an organisation select the autonomy level?

Base autonomy on business impact, reversibility, data sensitivity, regulatory requirements, evaluation evidence and operational maturity. Begin with the lowest level that produces useful value.

What is the most important production control?

There is no single control, but separating model reasoning from action authority is foundational. The model should not decide both what it wants to do and whether it is permitted to do it.

When should an agent be removed from production?

Remove or reduce autonomy when failure rates exceed thresholds, data or model behaviour drifts, controls cannot be verified, ownership becomes unclear, incidents reveal unacceptable impact or the business benefit no longer justifies the residual risk.

References

Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1. National Institute of Standards and Technology, 26 July 2024. https://doi.org/10.6028/NIST.AI.600-1
Artificial Intelligence Risk Management Framework 1.0, NIST AI 100-1. National Institute of Standards and Technology, January 2023. https://doi.org/10.6028/NIST.AI.100-1
Announcing the AI Agent Standards Initiative for Interoperable and Secure Innovation. National Institute of Standards and Technology, 17 February 2026. https://www.nist.gov/news-events/news/2026/02/announcing-ai-agent-standards-initiative-interoperable-and-secure
New Concept Paper on Identity and Authority of Software Agents. National Institute of Standards and Technology, 5 February 2026. https://www.nist.gov/news-events/news/2026/02/new-concept-paper-identity-and-authority-software-agents
ISO/IEC 42001:2023, Information technology — Artificial intelligence — Management system. International Organization for Standardization and International Electrotechnical Commission, 18 December 2023. https://www.iso.org/standard/42001
ISO/IEC 23894:2023, Information technology — Artificial intelligence — Guidance on risk management. International Organization for Standardization and International Electrotechnical Commission, February 2023. https://www.iso.org/standard/77304.html
OWASP Top 10 for Large Language Model Applications, 2025 Edition. OWASP Generative AI Security Project, 2025. https://genai.owasp.org/llm-top-10/
Agentic AI — Threats and Mitigations. OWASP Agentic Security Initiative, 2025–2026 resource series. https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/
MITRE ATLAS. MITRE, continuously maintained. https://atlas.mitre.org/
Zero Trust Architecture, NIST SP 800-207. National Institute of Standards and Technology, August 2020. https://doi.org/10.6028/NIST.SP.800-207
Best Current Practice for OAuth 2.0 Security, RFC 9700. Internet Engineering Task Force, January 2025. https://www.rfc-editor.org/rfc/rfc9700
OAuth 2.0 Token Exchange, RFC 8693. Internet Engineering Task Force, January 2020. https://www.rfc-editor.org/rfc/rfc8693
Model Context Protocol Specification, version 2025-11-25. Model Context Protocol Project, 25 November 2025. https://modelcontextprotocol.io/specification/2025-11-25
Model Context Protocol Security Best Practices. Model Context Protocol Project, continuously maintained. https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices
Agent2Agent Protocol Specification, version 1.0.0. A2A Protocol Project, 2026. https://a2a-protocol.org/latest/specification/
Open Policy Agent Documentation. Open Policy Agent Project, continuously maintained. https://www.openpolicyagent.org/docs
SPIFFE Specification Overview. SPIFFE Project, continuously maintained. https://spiffe.io/docs/latest/spiffe-about/overview/
Inside the LLM Call: GenAI Observability with OpenTelemetry. OpenTelemetry Project, 14 May 2026. https://opentelemetry.io/blog/2026/genai-observability/
Business Process Model and Notation, version 2.0.2. Object Management Group, January 2014. https://www.omg.org/spec/BPMN/2.0.2/
CloudEvents Specification and Project. Cloud Native Computing Foundation, version 1.0 family; graduated January 2024. https://cloudevents.io/
Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. European Parliament and Council of the European Union, 13 June 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
Building Effective AI Agents. Anthropic, December 2024. https://www.anthropic.com/engineering/building-effective-agents
ReAct: Synergizing Reasoning and Acting in Language Models. Yao, S. et al., October 2022. https://arxiv.org/abs/2210.03629
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Lewis, P. et al., May 2020. https://arxiv.org/abs/2005.11401
AgentBench: Evaluating LLMs as Agents. Liu, X. et al., International Conference on Learning Representations, 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/e9df36b21ff4ee211a8b71ee8b7e9f57-Abstract-Conference.html