This post is part of the “Responsible AI and AI Governance for Agents” series, covering responsible AI principles and evaluation, followed by AI governance principles, a data model, a governance prototype, and a comparison of governance platforms. This post introduces AI and agentic governance, including requirements, responsibilities and policies.
The Marketplace examples illustrate a proposed governance approach.
Objective
Before approving an AI system for use, an organization needs a governance policy: what it may be used for, who is accountable, which requirements apply and what evidence is needed for approval and continued review.
For enterprises deploying AI across teams and business processes, governance helps establish accountability and provide evidence for audits, customer commitments and applicable regulatory obligations. Startups need it too, with a process proportionate to the system’s use and risk.
This post explains governance requirements across eight areas and the additional controls needed for systems that use tools and act. I use two examples throughout the eight areas. The document summarizer illustrates general AI, and the Marketplace Assistant illustrates agentic AI.
A Correct Answer Leaves Other Questions Open
Consider the Marketplace Assistant that helps support staff look up orders and explain their recorded status. It might answer a question accurately and still leave several things unknown. Was the employee authorized to see that order? Did the agent have permission only to read the order, or could it also modify the order record? Did the model receive personally identifiable information (PII), such as the customer’s email address? Where did the tracing service store the conversation?
Those questions reach beyond the quality of the answer. They concern the system’s approved purpose, the resources it can access, its deployment and the people responsible for it.
AI Governance and Agentic Governance
My working definition of AI governance is the set of policies, responsibilities, decisions and evidence used to direct and oversee AI use. It covers which uses are acceptable, who is accountable, how systems are approved and controlled, and when those decisions must be reviewed.
AI governance applies to general AI applications and agentic systems. General AI applications here means applications that use models for summarization, image generation or video generation through predefined steps. Agentic systems, such as the Marketplace Assistant, can choose approved tools, inspect their results and decide the next step.
Both need a shared governance foundation: an accountable owner, an inventory entry, approved models, spending limits, data-handling and location requirements, release approval and ongoing review. Authentication and access controls also apply to both.
Agentic governance builds on that foundation with controls for the agent’s tool use and actions. For the Marketplace Assistant, specify which tools it may call, whose identity and permissions each call uses, which orders it may access, whether it may only read or also modify order records, and when human approval or intervention is required.

General AI applications and agentic systems share a governance foundation; agents need additional controls over tool use and actions.
What Drives the Requirements
When building an AI system, where do its governance requirements come from? I group the sources into three categories:
Organizational requirements: internal policies, customer contracts and standards the organization adopts. For the marketplace, these might specify approved model providers, customer-data handling and who may authorize changes to an order.
Principles and frameworks. These help organizations identify risks and turn broad expectations into requirements for their own systems.
External obligations: requirements based on geography and industry. Geographic examples include the EU AI Act for systems within its scope. Industry examples include HIPAA rules for covered entities and business associates handling protected health information in US healthcare. In US banking, model-risk supervisory guidance is another external source to review, while distinguishing guidance from binding law. Applicability depends on the organization, intended use, data and jurisdiction.
AI governance is still evolving. There is no single checklist that settles every use case, and applicability can require interpretation. Record the requirements selected, the reasons for selecting them and who owns the decision, then revisit them as the system and external guidance change.
Eight Areas to Make the Review Concrete
I use eight areas to organize governance requirements, each with a short definition and two illustrative examples. The general AI example is a document-summary application: it supplies an authorized document and a prompt to a model, then returns a summary through predefined steps. The agent example is the Marketplace Assistant, which chooses approved lookup tools to investigate an order. The examples show shared governance requirements and the additional controls needed for an agent’s tools and actions.

1 Ownership Inventory and Shadow AI
Definition: know which AI applications, models, agents and supporting tools are in use, where they run and who owns them.
General AI example: register the document-summary application, the model and version it uses, its provider, business purpose, deployment and accountable owner. Compare registered entries with discovered usage to identify unregistered applications or model endpoints.
Agent example: register the Marketplace Assistant, its model and version, MCP servers, available tools, deployment and accountable owner. Link discovered deployments to that inventory and flag unregistered agents, models or tools, missing owners and unexpected environments for review.
2 Purpose Impact and Acceptable Risk
Definition: approve the intended work, identify who could be affected and set acceptable outcomes and restrictions.
General AI example: approve summarization of internal reports, require human review before a summary is shared externally, and prohibit using the application to make personnel decisions.
Agent example: approve authorized order and refund-status lookups, while prohibiting refunds or order changes. Identify the risks of a wrong status misleading a customer or a cross-customer lookup exposing their information.
3 Identity Access and Authorization
Definition: control which people and service identities may access the AI system, models and data.
General AI example: allow approved employees to summarize only documents they are authorized to access. The application checks access before supplying text to the model and uses approved model routes.
Agent example: user identity limits which orders an employee may see; agent identity carries read-only permissions and approved lookup-tool access.
4 Data Privacy Residency Retention and Deletion
Definition: set data-exposure, processing and storage location, retention and deletion requirements across all services handling AI data.
General AI example: remove unnecessary personal identifiers before sending documents to the model. Define approved processing and storage regions, retention periods and deletion rules for uploads and summaries, including provider copies, traces and backups.
Agent example: mask the customer’s email before a model call and return only authorized order details. Check location, retention and deletion for the database, model endpoint, gateway, evaluation, tracing, monitoring, memory and backup services that handle the data.
5 Behavioral and Operational Assurance
Definition: evaluate safety, security, values and failure behavior against the approved use.
General AI example: supply a report containing embedded instructions and incomplete information. Check that the application summarizes supported facts without following those instructions or inventing missing details.
Agent example: test that the Marketplace Assistant declines help with a false claim, resists instructions it encounters in retrieved content, reports only supported status and uses the agreed fallback when a lookup times out.
6 Release Approval and Changes
Definition: record who approved which release contents and which changes require reassessment.
General AI example: record the build deployed for the document summarizer. Link its prompt and model versions, configuration, evaluation results and approver to that release. Review changes to the model, provider or document-processing steps before adopting them.
Agent example: also record the tool versions and permissions used by the Marketplace agent. Adding a write-capable tool triggers a review of the read-only rule and the checks that enforce it.
7 Runtime Limits Cost Visibility and Intervention
Definition: set operating limits, track attributed spend and define when a person must intervene.
General AI example: limit document size and request volume, track summarization spend against its budget, and route unreadable or incomplete documents to a human reviewer rather than producing a guessed summary.
Agent example: set tool-call and retry limits, track the assistant’s daily model spend and hand an unresolved order lookup to support staff. If the budget must block further calls, require and verify a hard-cap control.
8 Periodic Evaluation and Sign Off
Definition: renew approval on a schedule using current evidence, even when no release has changed.
General AI example: review the document-summary application’s approved users, document-access checks, data handling and latest evaluations on the review schedule. Record when checks ran, who signed off and when the next review is due.
Agent example: refresh the Marketplace Assistant’s customer-access, read-only and model-access checks on the review schedule. Record unresolved findings and the decision to continue approval or require a fix.
Putting Governance into Practice
Once a governance requirement is agreed, turn it into checks, controls and ongoing observation.

Measure whether the requirement holds using evaluations, access tests and configuration inspection. Enforce it through backend authorization, tool permissions and guardrails. Monitor denied requests, tool calls and unexpected behavior during operation.
After release, use production observations to refresh evaluations, refine the controls and decide whether the system can continue operating.
Where Governance Controls Are Applied
This table shows example enforcement points and how to verify the controls. Implementations may place controls differently, provided they have the necessary context, cover the relevant access paths and reliably enforce the rule.
Requirement | Where the control is applied | How we verify it |
|---|---|---|
Approved agent callers | Agent gateway checks which users or applications can invoke the Marketplace Assistant. | Verify that the support app can invoke the agent and that an unapproved app is denied. |
Approved models | Model gateway: restrict permitted models, routes and credentials; SDK checks can add a layer. | Inspect resolved models and fallbacks; attempt a disallowed model call. |
Approved data and log regions | Deployment and service settings for the model endpoint, database, logs, traces and backups; location constraints where supported. | Inspect actual resource locations and configuration evidence. A configuration check verifies the setting. |
Agent model-spend budget | Model gateway budget and rate-limit controls tied to the agent or application identity; use a blocking limit when required. | Check spend attribution and test the configured limit. Alerts alone do not stop calls. |
Read-only order access | Runtime guardrail checks a tool action before effective execution. The order API independently checks write permission. | Inspect effective tool and database permissions; attempt a controlled write and verify no record changes. |
Access only permitted orders | Business backend authorization using trusted caller identity and order scope; a tool gateway can enforce this if it has the required context. | Test cross-user access and inspect which records the service actually returns. |
Human approval for sensitive actions | Runtime guardrail holds the action for approval. The backend can validate approval before committing the write. | Attempt the action without valid approval and verify that execution is blocked. |
An agent gateway checks who can invoke the agent. A model gateway checks which models can be used and applies configured budget limits. A runtime guardrail checks a proposed tool action before execution and can hold sensitive actions for human approval. If a request reaches the business API, the API checks whether the caller may read or change the record. Configuration checks verify settings such as where data and logs are stored.
Summary
These eight areas organize governance requirements: ownership and inventory, purpose and risk, access, data handling, assurance, release approval, runtime limits and periodic review. The general AI and marketplace-agent examples show how the requirements apply to different systems.
The applicable requirements inform the organization’s AI governance policy: what is required, who is accountable and how compliance will be checked. This post identifies the areas to consider; it does not define a complete policy for every organization.
The next post turns these governance requirements into a concrete data model, connecting each requirement to the system it applies to, its checks and the evidence used for review.
Acknowledgements
This post was prepared with AI assistance in Codex.
References
Principles Frameworks and Standards
OECD AI Principles — trustworthy-AI principles, including accountability.
NIST AI RMF and Playbook — voluntary risk-management guidance and suggested actions.
ISO/IEC 42001 — organizational AI management-system requirements.
Jurisdiction and Sector Examples
European Commission: AI Act — risk-based framework and implementation context.
Colorado Attorney General: AI rulemaking — current state-law and commencement information.
China CAC: Interim Measures — primary text, including Article 2 scope.
HHS: HIPAA applicability — covered entities and business associates.
Federal Reserve: SR 26-2 — revised model-risk guidance and applicability.