The first article in this series, Enterprise AI Adoption: Requirements for Security, Governance, and Scale, defined the problem: enterprises consume AI through coding tools, standalone hosted applications, custom agents, and enterprise knowledge systems, but need consistent controls across identity, credentials, data, cost, providers, and operations.

The second article, A Reference Architecture for Governed Enterprise AI, translated those requirements into architectural building blocks and described the available implementation choices.

This article makes the opinionated decisions. It recommends what to implement, what to defer, and which representative products to consider for three enterprise profiles.

Product recommendations reflect the market as of July 2026 and should be revalidated before implementation.

Scope and Assumptions

Company size is the primary organizing variable because adding industry, geography, regulation, data sensitivity, cloud provider, and AI maturity as independent dimensions would make a simple prescription impossible.

The recommendations assume a software-enabled company with moderate data sensitivity, one primary public cloud, an existing corporate identity provider, both employee and custom AI usage, and a preference for managed services unless scale or control justifies ownership.

Size remains a proxy. Move to the next architecture tier when regulation, data sensitivity, AI expenditure, workload volume, geographic isolation, or operational complexity exceeds what is typical for the company profile.

Engineering scale is more useful than total employee count because it better predicts the number of custom workloads and the organization’s ability to operate platform components.

Small Enterprise: Prefer Simplicity

The small-enterprise architecture deliberately accepts coarser attribution and fewer centralized controls in exchange for low operating overhead.

Prescription

Hosted AI and coding

Use ChatGPT Business as the company-managed employee workspace and Codex as the standard coding assistant. Codex is included in ChatGPT Business, although usage limits and credit options depend on the plan. This reduces unmanaged subscriptions and keeps general AI and coding under one commercial and administrative relationship. OpenAI documents current Codex plan availability here.

Use SSO where available, central billing, approved data terms, and an explicit user-removal process. ChatGPT Business supports SSO but does not provide SCIM directory synchronization, so lifecycle management remains partly administrative. OpenAI documents the Business and Enterprise identity differences here.

Model access without a gateway

Applications call the primary cloud’s managed model platform directly using distinct workload identities. AWS-centric organizations use Amazon Bedrock, Google Cloud organizations use Vertex AI, and Microsoft-centric organizations use Azure AI Foundry. A company without meaningful cloud alignment may use a direct OpenAI or Anthropic API organization.

A shared client library may standardize model aliases, metadata, timeouts, retries, and trace identifiers. It is not a gateway and cannot guarantee centralized enforcement.

Attribution is therefore based on separate cloud projects or accounts, workload identities, billing tags, provider reports, and application logs. This is adequate while the workload portfolio is small, but it may not support reliable per-user or per-feature allocation.

Routing and guardrails

Use static model aliases such as general-fast, general-quality, and sensitive-approved. A centrally maintained configuration maps each alias to an approved cloud model deployment. Cloud IAM should restrict workloads to eligible deployments where supported.

Use provider-native guardrails and application controls. The application remains responsible for user authorization, tool permissions, structured output, agent limits, and approval of consequential actions.

Agents and knowledge

Do not standardize an agent framework yet. The first application can use an appropriate SDK and run within existing application infrastructure. Every agent still needs narrow tool permissions, least-privilege credentials, step and cost limits, and human approval for irreversible actions.

Do not build an enterprise knowledge platform. Employees can use administrator-approved connectors in the hosted workspace, subject to source permissions. Build retrieval only for a product workflow with a measurable requirement.

Graduation trigger

Introduce a managed gateway when the organization has several production AI applications, more than one provider, duplicated integrations, shared credential problems, or a need for centralized budgets, guardrails, routing, and detailed attribution.

Mid-Size Enterprise: Introduce a Shared Control Plane

The mid-size architecture requires all custom production model traffic to pass through a dedicated gateway. The platform remains primarily managed because the organization has an emerging platform team rather than a large AI infrastructure organization.

Prescription

Enterprise workspace

Use standard ChatGPT Enterprise seats for employees and engineers. A standard seat includes ChatGPT and Codex and offers the greatest flexibility. Codex-only seats should be reserved for people who need coding access but should not receive the broader ChatGPT workspace. OpenAI’s current Enterprise documentation describes both seat types and their access boundaries.

Use enforced SSO, SCIM, groups, and role-based access. Separate groups should cover general AI users, Codex users, AI developers, production operators, platform administrators, and sensitive-data users.

Gateway and model portfolio

Use managed Portkey as the default gateway. It provides multi-provider access, virtual credentials, routing, budgets, rate limits, retries, circuit breaking, observability, and integrated guardrails without requiring the emerging platform team to operate the full gateway stack. Portkey documents these gateway capabilities and deployment options.

Use the primary cloud model platform as the main backend and one approved direct provider as a secondary path. All production applications authenticate to the gateway; provider credentials remain with the platform team.

Use logical model aliases, policy restrictions, availability fallbacks, and controlled model-version rollout. Do not add a dedicated intelligent router until expenditure is material and evaluation data can prove that dynamic routing preserves quality.

Guardrails, agents, and tools

Apply provider-native controls, gateway input/output guardrails, and application-specific rules. Portkey guardrails support checks and actions on gateway requests and responses. Add Check Point Lakera only for public-facing or higher-risk workloads that justify specialized prompt-injection and data-leakage protection.

Adopt the OpenAI Agents SDK as the default for straightforward tool-using agents. Allow LangGraph for durable, stateful, resumable, or human-in-the-loop workflows. Run agents on the existing container or serverless platform and provide shared standards for identity, tool registration, MCP servers, traces, execution limits, approvals, and action logs.

Knowledge, observability, and FinOps

Use company knowledge in ChatGPT Enterprise with administrator-approved apps for employee search. Build a shared managed-cloud retrieval foundation only when multiple production applications need common ingestion, permissions, citations, and evaluation.

Use Portkey for gateway-level telemetry and Langfuse Cloud for multi-step traces and evaluations where deeper analysis is required. Use gateway budgets for enforcement and the existing cloud-finance process for reporting. Add Finout or CloudZero when model and supporting cloud costs span providers and require formal allocation or forecasting.

Graduation trigger

Move to the large-enterprise pattern when the company needs regional isolation, multiple business-unit control, formal chargeback, private deployment, dedicated AI security, a shared agent runtime, or self-hosted inference.

Large Enterprise: Central Policy, Federated Delivery

The large-enterprise architecture supports multiple providers, regions, business units, and control teams. Policy and visibility are centralized, while application ownership remains federated.

Prescription

Regional gateway and inference platform

Deploy LiteLLM Enterprise in approved regions as the standard model-access plane. Each deployment should provide high availability, business-unit isolation, local credentials, virtual keys, budgets, quotas, model aliases, audit records, guardrail integration, and approved fallback policies. LiteLLM documents virtual-key, authentication, SSO, and enterprise gateway capabilities in its product overview.

Applications should not receive direct provider credentials except through documented exceptions. The model portfolio can include strategic cloud AI platforms, direct OpenAI and Anthropic access, selected inference providers, and self-hosted open-weight models for justified workloads.

Expose internal capability aliases such as enterprise-fast, enterprise-reasoning, enterprise-code, regulated-region, and open-weight-private rather than embedding provider model names throughout applications.

Routing and guardrails

Apply policy routing to every request. Region, data classification, business unit, provider approval, and budget determine the eligible model set before performance or cost optimization.

Use intelligent routing only for workloads with evaluation datasets and measurable quality thresholds. RouteLLM or another router may select among eligible models, but the gateway remains the enforcement point.

Use four guardrail layers: provider-native safety, gateway policy, Check Point Lakera for high-risk workloads, and application business controls. Measure false positives, latency, and bypass resistance rather than treating guardrails as absolute security boundaries.

Agent platform

Provide an approved framework portfolio and a shared runtime. Use the OpenAI Agents SDK for the default tool-using pattern and LangGraph for durable workflows. The runtime should provide workload and delegated identity, queues, resumability, isolated code execution, an approved tool and MCP registry, secrets injection, limits, approvals, traces, cancellation, and kill switches.

The gateway governs model inference. The agent runtime governs execution and tools. Deterministic services should verify authorization and execute consequential actions proposed by a model.

Enterprise knowledge

Use company knowledge in ChatGPT Enterprise with approved apps for employee search and synthesis. This is a ChatGPT feature, not a separate product and not a backend for custom applications.

Operate a shared permission-aware RAG platform for product applications. It should provide connectors, ingestion, permission synchronization, deletion, retrieval APIs, citations, evaluation, and regional isolation.

Consider Glean when dedicated cross-enterprise search is a strategic requirement that company knowledge in ChatGPT does not satisfy. Do not deploy both without defining their distinct user populations and responsibilities.

Observability and FinOps

Use self-hosted Langfuse Enterprise for traces, evaluations, prompt management, datasets, RBAC, retention policies, and audit logs. These self-hosted enterprise controls are described in Langfuse’s deployment and pricing documentation. Export platform telemetry through OpenTelemetry and security events to the enterprise SIEM.

Use gateway budgets for real-time enforcement and CloudZero for allocation, anomaly detection, forecasting, unit economics, and showback or chargeback across inference and supporting infrastructure.

Self-host models through vLLM only for workloads with stable demand, demonstrated economics, suitable model quality, and an operations team able to own serving and upgrades.

How the Three Architectures Progress

When Size Is Not Enough

Use a stronger architecture tier when any of the following applies:

• Regulated or highly sensitive data
• Mandatory data residency or private networking
• High AI expenditure or customer-facing AI cost of revenue
• Many autonomous agents with consequential tools
• Multiple cloud providers or geographic regions
• Formal business-unit isolation or chargeback
• Strict availability requirements
• A large portfolio of production AI applications

A small healthcare or financial company may need large-enterprise security controls. A large company with limited AI use may begin with the mid-size platform. Size determines the default, not the exception policy.

Final Recommendation

The prescribed progression is intentionally conservative:

• Small: one managed employee workspace, Codex, one model platform, no gateway, no enterprise RAG, and no dedicated FinOps platform.
• Mid-size: ChatGPT Enterprise and Codex, a managed gateway, two inference paths, shared guardrails and tracing, and application-focused RAG.
• Large: regional gateways, a governed provider portfolio, layered AI security, a shared agent runtime, enterprise retrieval, formal observability, and chargeback.

The architecture should become more sophisticated only when usage, risk, cost, or organizational scale creates a concrete reason. The objective is not to deploy every available AI platform component. It is to provide the minimum architecture that enables adoption without losing control.

In the first article in this series, Enterprise AI Adoption: Requirements for Security, Governance, and Scale, we identified four ways enterprises consume AI and seven requirements that apply across them. Those requirements cover identity, non-human credentials, cost governance, security and data controls, model flexibility, optimization, and observability.

This article translates those requirements into a reference architecture. It defines the architectural building blocks, their responsibilities, and the implementation choices available to an enterprise. The third article in the series will use these building blocks to prescribe specific architectures for small, mid-size, and large organizations.

One Architecture, Two Control Paths

Enterprise AI cannot be governed through a single product because not all AI traffic follows the same path.

Custom applications, agents, retrieval systems, and compatible coding tools can access models through an enterprise-controlled model gateway. The gateway can authenticate workloads, protect provider credentials, enforce policies, route requests, attribute cost, apply common guardrails, and record telemetry.

Standalone hosted applications such as ChatGPT, Claude, Gemini, and specialized cloud tools normally communicate directly with their providers. Their requests do not pass through the enterprise model gateway. They must instead be governed through corporate identity, provisioning, product administration, contracts, connector controls, SaaS security, data-loss prevention, and audit integrations.

This distinction is fundamental. A model gateway is an important control point, but it is not a universal control plane for every form of enterprise AI consumption.

Fig: Enterprise AI requires separate control paths for hosted applications and enterprise-controlled model traffic.

Architecture Principles

The reference architecture follows five principles.

Centralize policy without centralizing every decision

The enterprise needs common boundaries for identity, credentials, approved models, data, cost, and auditability. Application teams should remain free to select models and implementation patterns within those boundaries. The central platform should provide a governed path rather than become a manual approval queue.

Separate human, workload, and delegated identities

Employees authenticate through the corporate identity provider. Applications and agents use workload identities. When an agent acts for a person, the trace should preserve both the agent identity and the initiating user identity.

Keep provider credentials away from applications

Applications should authenticate to an enterprise-controlled gateway. The gateway protects and rotates provider credentials, maps workload identities to policies, and issues virtual credentials where needed.

Make data policy part of routing

Not every model or provider is approved for every data class. The eligible destination should be constrained by data sensitivity, contractual terms, region, retention policy, and workload risk before cost or performance optimization is considered.

Allow complexity to grow with need

The logical architecture can remain consistent while its implementation grows from cloud-native services to managed gateways, regional control planes, dedicated security tools, and self-hosted inference.

The Technology Landscape

The following table identifies representative implementation choices. These are examples, not recommendations for every organization.

1. Enterprise Identity and User Lifecycle

The corporate identity provider is the starting point for human access. Hosted AI products, coding applications, gateway consoles, observability platforms, and internal applications should use SSO wherever supported. Automated provisioning should create, update, suspend, and remove accounts as employees change roles or leave.

Groups and roles should represent approved populations such as general AI users, coding users, sensitive-data users, platform administrators, and model approvers. Identity establishes accountability and lifecycle control, but it does not determine what data a user may enter into an AI product. Data policy and product-specific administration remain necessary.

2. Governance for Standalone Hosted AI Applications

Standalone hosted applications sit outside the model-gateway path. The enterprise should maintain an approved product portfolio and evaluate each product for SSO, provisioning, administrative controls, retention, training terms, audit logs, regional processing, sharing, and connectors.

Connectors require particular care. When a hosted application connects to email, files, source code, or collaboration systems, it gains a new authorization path into enterprise information. Administrators must control which connections are available, which OAuth permissions are granted, and how access is revoked.

This is an architectural concern, but not a separate deployable product. It is implemented through identity, SaaS administration, procurement, CASB or DLP controls, connector governance, and organizational policy.

3. Workload Identity and Secrets

Every production workload should have an identity independent of the developer who created it. Production and non-production environments should not share identities or unrestricted credentials.

Where possible, use short-lived cloud workload credentials. When secrets are necessary, keep them in an enterprise secrets platform and rotate them through an established process. Applications should receive gateway credentials or virtual keys instead of underlying provider keys.

This allows a compromised application to be revoked independently, enables per-workload budgets and model permissions, and permits provider credentials to change without application redeployment.

4. Model Gateway

The model gateway is the shared access layer between enterprise-controlled workloads and model providers. It should authenticate the caller and evaluate attributes such as application, team, environment, data classification, use case, cost center, region, and requested capability.

The gateway capability set may include:

• Workload authentication and model authorization
• Provider credential protection and virtual keys
• Standard request and response interfaces
• Model aliases and provider abstraction
• Usage metering, budgets, quotas, and rate limits
• Routing, fallback, retries, and circuit breaking
• Audit events and trace propagation
• Caching where authorization permits it
• Input and output guardrails
• Sensitive-data detection and redaction
• Model and use-case allowlists
• Request-size and context limits
• Schema and structured-output validation

The gateway should not contain all business logic. End-user authorization, prompt construction, tool selection, and domain rules normally belong in the application or agent runtime.

5. Model Providers and Inference Platforms

Behind the gateway, the enterprise may use direct model-provider APIs, public-cloud AI platforms, specialized inference providers, or self-hosted open-weight models.

Cloud-native platforms are a practical starting point for companies already standardized on a cloud because they integrate with cloud identity, networking, billing, and regional controls. Direct providers may expose capabilities earlier but create an additional contract, credential, billing path, and data boundary. Self-hosted inference offers deployment control but transfers responsibility for scaling, security, availability, upgrades, and model serving to the enterprise.

The reference architecture permits these options to coexist behind a controlled access path.

6. Policy, Data Controls, and Guardrails

Policy connects enterprise requirements to runtime decisions. It may consider workload, user, environment, data class, model, provider, region, and budget before allowing or routing a request.

Guardrails should be layered:

Guardrail actions may block, redact, warn, log, retry, or reroute. No guardrail product can determine domain-specific correctness or eliminate the need for least-privilege application design.

7. Routing, Resilience, and Portability

Routing is a logical architecture block even when it is implemented inside the gateway.

Three patterns are available:

1. Gateway-integrated routing: The gateway selects a deployment using aliases, policy, cost, latency, availability, or region.
2. Dedicated intelligent routing: A router such as RouteLLM selects between eligible models based on predicted quality, complexity, or cost. The gateway remains the enforcement point.
3. Application-directed routing: The application selects a capability because it has the best task context, while the gateway verifies that the selection is allowed.

Fallbacks must satisfy the same data and compliance policy as the original destination. A technically available model is not automatically an approved fallback.

8. Agent Framework, Runtime, and Tool Governance

An agent framework defines agents, tools, handoffs, state, and execution flow. The runtime operates those workflows, including long-running execution, retries, queues, isolation, and resumability. The gateway controls model access but does not govern every action an agent can take.

The agent layer should provide:

• An approved framework or supported framework portfolio
• Workload and delegated user identity
• Tool and MCP-server registry
• Per-agent tool permissions
• Step, time, cost, and concurrency limits
• Isolated code execution
• Action-level audit logs
• Human approval for consequential actions
• Retry, cancellation, and kill-switch behavior

For high-impact actions, the model should propose the action while deterministic code verifies authorization and performs the operation.

9. Cost Governance and FinOps

The gateway should record application, team, environment, provider, model, usage, and estimated cost. This enables real-time budgets, alerts, quotas, and rate limits.

Enterprise FinOps has a broader responsibility. It must reconcile provider invoices and include supporting costs such as retrieval, databases, queues, agent compute, and self-hosted inference. As maturity increases, the progression is:

Metering → budgets and alerts → attribution → forecasting → optimization → showback or chargeback.

Optimization may include smaller-model routing, caching, prompt reduction, agent-loop limits, batching, or self-hosted inference. Quality and cost must be measured together.

10. Enterprise Knowledge and Retrieval

Employee knowledge access and application RAG are separate architectural needs.

Company knowledge in ChatGPT can search approved connected apps for employee questions and return cited answers while respecting source permissions. It is a feature inside ChatGPT Business and Enterprise, not a separate product or a reusable backend for custom applications.

Product applications require a retrieval architecture with ingestion, change detection, classification, permission synchronization, authorized retrieval, citations, and evaluation. Permission checks should occur before unauthorized content is provided to the model or written to traces.

Organizations may use a dedicated enterprise-search product, managed cloud services, or a custom shared platform depending on scale and requirements.

11. Monitoring, Tracing, Audit, and Evaluation

The architecture produces four kinds of operational evidence:

• Infrastructure monitoring for availability, latency, failures, and resource health
• AI traces across retrieval, model calls, and tool actions
• Security and compliance audit records
• Quality evaluations using test datasets, production feedback, automated scoring, and human review

Telemetry requires its own data policy. Prompts, responses, retrieved documents, and tool output may contain sensitive information. The enterprise must define redaction, retention, encryption, regional storage, and operator access.

Cloud-Native Access or a Dedicated Gateway?

A cloud-native model platform may be sufficient when most workloads use one cloud, the model portfolio is limited, cloud identity and billing provide enough control, and cross-provider routing is unnecessary.

A dedicated gateway becomes valuable when applications use multiple providers, credentials must be isolated centrally, budgets and policies must apply across teams, or common routing, guardrails, and audit records are required.

The two approaches are complementary: a dedicated gateway can use cloud-native model platforms as backends.

Managed, Self-Hosted, or Built Internally?

Managed products reduce operating effort. Self-hosted products increase deployment and data control but require upgrades, security maintenance, scaling, and availability ownership. Internal development should be reserved for requirements that available products cannot satisfy.

A model proxy or RAG demonstration may be simple. A production platform with identity, policy, streaming, retries, accounting, permissions, deletion, tracing, and high availability is not.

From Reference Architecture to Prescription

The architecture is intentionally broader than any one organization should deploy immediately. A small company may need only an approved employee assistant, one cloud model platform, and basic budgets. A mid-size organization may need a managed gateway, shared tracing, and application-level RAG. A large enterprise may require regional gateways, multi-provider routing, dedicated guardrails, chargeback, a shared agent runtime, and self-hosted components.

The next article, Prescribed Enterprise AI Architectures for Small, Mid-Size, and Large Organizations, selects a coherent implementation and representative vendors for each profile.

AI adoption inside enterprises rarely begins with a coordinated platform strategy. It usually starts with individuals and teams selecting tools that solve immediate problems.

A developer subscribes to an AI coding assistant. Another team builds an application using a foundation-model API. Employees begin using standalone cloud applications such as ChatGPT, Claude, Gemini, or Codex. A business unit connects an AI application to internal documents. Eventually, the organization considers an enterprise-wide knowledge assistant or retrieval-augmented generation system.

Each decision may be reasonable on its own. Together, however, they create a fragmented operating environment.

Users authenticate differently across tools. Applications depend on provider API keys. Usage is billed through unrelated subscriptions and cloud accounts. Sensitive data may pass through systems with different retention and training policies. Security teams have limited visibility into what is being used, while finance teams struggle to determine which teams, applications, or business outcomes are responsible for the expenditure.

The enterprise AI problem is therefore not simply about selecting the best model or application. It is about enabling several forms of AI consumption while maintaining consistent control over identity, credentials, data, cost, providers, and operations.

Fig: Different AI consumption paths require one consistent set of enterprise controls

Four Ways Enterprises Consume AI

Enterprise AI adoption can be organized into four broad consumption patterns. These patterns overlap, but their ownership, identity, billing, and data-flow characteristics differ enough to require separate consideration.

1. AI-assisted software development

The first pattern is the use of AI by software-development teams. It includes IDE assistants, command-line tools, coding agents, code-review assistants, and autonomous development applications.

Some products are purchased as per-user subscriptions. Others allow users to connect a model-provider account or consume models through API credentials. More autonomous tools may execute commands, inspect repositories, modify files, or interact with development infrastructure.

The enterprise must determine whether users authenticate through corporate identity, which repositories and environments the tool may access, how source code is handled, how expenditure is attributed, and whether agent actions are logged and auditable.

The risk is not limited to source-code disclosure. Coding agents can access credentials, execute commands, change infrastructure, or introduce dependencies. Their permissions and actions require greater scrutiny than conventional autocomplete.

2. Custom AI applications, agents, and workflows

The second pattern consists of applications built or operated by the enterprise. Examples include customer-service assistants, document-processing workflows, incident-response agents, internal copilots, and automated business processes.

These systems usually consume foundation models through APIs. They may retrieve enterprise data, invoke tools, call internal services, maintain state, and perform actions on behalf of users.

Unlike a standalone application, the enterprise owns much of the operating responsibility. It must manage application identities, API credentials, authorization, model selection, usage limits, monitoring, and failure handling.

Agents make this especially important. A traditional application normally follows a path defined in code. An agent can dynamically choose a model, call tools, retrieve information, and take several steps to complete a task. The architecture must govern both model access and access to data and tools.

3. Standalone hosted AI applications

The third pattern is the direct use of cloud-hosted AI applications by employees. This includes general-purpose assistants and specialized applications for coding, research, writing, design, analytics, and productivity.

The provider operates the application and normally controls the interface, model selection, data handling, and underlying infrastructure. The enterprise consumes the product through user accounts, team subscriptions, or an enterprise agreement.

These products may be introduced through centralized procurement, but they can also enter through individual subscriptions and expense claims. This creates shadow AI: employees using services that the organization has not reviewed or cannot administer.

The enterprise must evaluate SSO, automated provisioning, administrative controls, retention and training terms, audit logs, sharing, and connections to internal systems. A model gateway cannot normally intercept these interactions because the application communicates directly with its provider.

4. Enterprise knowledge search and RAG

The fourth pattern is enterprise knowledge search, often implemented using retrieval-augmented generation, or RAG.

These systems search enterprise data sources and provide relevant content as context for a model response. Sources may include document repositories, collaboration systems, source-code platforms, ticketing systems, knowledge bases, or business applications.

An organization can purchase enterprise search, use knowledge features within a hosted assistant, build a custom RAG platform, or combine managed services with internal components.

The difficult part is rarely retrieving a document and sending it to a model. The harder requirements are preserving source permissions, keeping indexes current, removing content when permissions change, recording citations, preventing unauthorized retrieval, and measuring retrieval and answer quality.

Why Fragmentation Becomes a Governance Problem

The four patterns do not share a common operating model.

Coding applications may be licensed per user. Custom applications consume APIs based on tokens or requests. Hosted applications may use individual, team, or enterprise subscriptions. Knowledge platforms may combine licenses, indexed-content charges, infrastructure costs, and model consumption.

Identity is similarly fragmented. Employees authenticate interactively, while applications and agents use non-human identities. Some products support enterprise federation and automated provisioning; others rely on independently managed accounts. One employee may also connect several applications to internal data using separate authorization grants.

Data may be entered into a hosted application, sent to an API by an internal workload, retrieved from enterprise repositories, recorded in traces, or stored in provider logs.

Managing every product independently leads to predictable problems:

• Former employees retain access to tools that were not centrally provisioned.
• Provider API keys are shared between developers or embedded in applications.
• Teams cannot identify which application caused an increase in expenditure.
• Sensitive information is sent to models that are not approved for that data class.
• Security teams cannot reconstruct which prompt, context, model, or tool produced an action.
• Applications become tightly coupled to provider-specific interfaces.
• Different teams build duplicate integrations and governance mechanisms.

A scalable AI program requires one set of enterprise requirements, even when those requirements are implemented differently across consumption patterns.

Seven Requirements for Enterprise AI Adoption

1. Identity and access governance

Human access should connect to the corporate identity system wherever the product permits it. This includes SSO, multi-factor authentication, role-based access, centralized provisioning, and timely deprovisioning.

Access decisions should reflect the sensitivity of the tool and the data it can reach. A general assistant with no internal connections does not present the same risk as an agent that can access source code, customer records, or production systems.

The organization also needs clear ownership. Someone must approve the product, determine who may use it, review access, and remove access when it is no longer required.

2. Non-human identity and credential governance

Applications and agents should not depend on personal API keys or shared provider credentials.

They require non-human identities with narrowly scoped permissions. Credentials should be issued through an approved process, stored in a secrets-management system, rotated, and revoked without disrupting unrelated applications.

Where possible, applications should receive virtual or intermediary credentials rather than direct provider keys. This allows application-specific limits and provider changes without redistributing credentials to every workload.

3. Cost attribution and financial governance

AI cost is variable and can grow quickly as applications gain users, agents perform more steps, or teams select more capable models.

Enterprises need visibility beyond the provider invoice. Usage should be attributable to a user, application, team, environment, or business unit. Budgets, quotas, alerts, and rate limits should operate at the same levels.

Small organizations may need only visibility and spending alerts. Larger organizations may require forecasting, showback or chargeback, unit-cost measurements, and formal FinOps ownership.

Where possible, expenditure should be connected to outcomes such as completed tasks, resolved cases, generated documents, or developer workflows. Token counts alone do not indicate whether an application is successful.

4. Security and data governance

The enterprise must define what data can be processed by which products, models, providers, and regions.

This requires consistent review of retention, provider-training terms, encryption, residency, subprocessors, auditability, and incident-response commitments. It must also cover information stored outside the primary request, including traces, prompts, retrieved context, cached responses, evaluation datasets, and application memory.

Controls should be proportional to data sensitivity. Public information, internal documents, source code, personal data, and regulated records should not automatically follow the same approval path.

5. Model and vendor flexibility

Model capabilities, prices, performance, and availability change rapidly. An application unnecessarily coupled to one provider becomes difficult to optimize or migrate.

Enterprises should use stable internal interfaces where the benefits justify the effort. Provider-specific features can still be used, but the dependency should be intentional and documented.

Vendor flexibility does not mean every request must switch dynamically between providers. It means the architecture preserves a practical path to compare, route, replace, or add models without redesigning every application.

6. Cost and performance optimization

Not every task requires the most capable or expensive model.

Applications should select models according to task complexity, latency, quality, privacy, and cost. Repeated requests may benefit from caching. High-volume and predictable workloads may eventually justify self-hosted open-weight models.

Optimization should follow measurement. A cheaper model is not cheaper if poor outputs cause repeated calls, human correction, or failed workflows. Cost, latency, and quality must be evaluated together.

7. Monitoring, tracing, and evaluation

Traditional infrastructure metrics are necessary but insufficient for AI systems.

Teams need to understand which model was used, how much was consumed, how long each step took, which context was retrieved, which tools were called, and where failures occurred. Agent traces may span multiple model calls and actions within one task.

Operational monitoring must be complemented by evaluation. An application can remain technically available while quality deteriorates because of a model change, prompt update, retrieval problem, or altered data source.

A Shared Control Point for Model Access

For custom applications, agents, and compatible coding workflows, many requirements can be enforced through a shared model-access layer.

This layer sits between enterprise workloads and model providers. Depending on the organization’s needs, it can authenticate applications, protect provider credentials, enforce model policies, route requests, apply quotas, attribute cost, record audit events, collect traces, and provide a consistent interface across providers.

It is often described as a model gateway or AI gateway.

The gateway is an important architectural control point, but it is not the entire governance solution. Standalone hosted applications generally operate outside it. Knowledge systems require source authorization and retrieval controls. Agents require tool and action governance. Coding tools require safeguards for source code, credentials, local execution, and development infrastructure.

Requirements Change with Enterprise Context

The requirements are broadly consistent, but implementation should reflect scale, risk, and operating maturity.

A small company may prioritize a limited set of approved tools, centralized billing, basic SSO, and managed model access. A mid-size organization may need a dedicated gateway, per-application credentials, team budgets, audit logs, and shared observability. A large enterprise may require multiple providers, regional deployments, formal data classifications, policy enforcement, business-unit isolation, chargeback, advanced evaluation, and self-hosted components.

Employee count alone is not enough. A smaller company handling regulated data may require stronger controls than a larger organization working mainly with public information. AI expenditure, data sensitivity, number of applications, regulation, and platform-team capability are often better indicators of architectural need.

Questions the Reference Architecture Must Answer

These requirements lead to concrete architectural questions:

• Where should model access be centralized?
• How should human users, applications, and agents authenticate?
• How should provider credentials be isolated?
• Which controls belong in the identity provider, gateway, application, or provider?
• How should standalone hosted applications be governed?
• Where should prompts, traces, retrieved context, and evaluation data be stored?
• How should costs be attributed and controlled?
• When is a cloud provider’s model platform sufficient?
• When does an organization need a dedicated gateway?
• When should enterprise knowledge search be purchased, built, or deferred?
• How should the architecture evolve as usage, risk, and maturity increase?

There is no single product that answers every question. The solution is an architecture that combines identity, model access, security, cost management, observability, and application-level controls.

The next article, A Reference Architecture for Governed Enterprise AI, develops that architecture and explains the available implementation choices.