In the first article in this series, Enterprise AI Adoption: Requirements for Security, Governance, and Scale, we identified four ways enterprises consume AI and seven requirements that apply across them. Those requirements cover identity, non-human credentials, cost governance, security and data controls, model flexibility, optimization, and observability.

This article translates those requirements into a reference architecture. It defines the architectural building blocks, their responsibilities, and the implementation choices available to an enterprise. The third article in the series will use these building blocks to prescribe specific architectures for small, mid-size, and large organizations.

One Architecture, Two Control Paths

Enterprise AI cannot be governed through a single product because not all AI traffic follows the same path.

Custom applications, agents, retrieval systems, and compatible coding tools can access models through an enterprise-controlled model gateway. The gateway can authenticate workloads, protect provider credentials, enforce policies, route requests, attribute cost, apply common guardrails, and record telemetry.

Standalone hosted applications such as ChatGPT, Claude, Gemini, and specialized cloud tools normally communicate directly with their providers. Their requests do not pass through the enterprise model gateway. They must instead be governed through corporate identity, provisioning, product administration, contracts, connector controls, SaaS security, data-loss prevention, and audit integrations.

This distinction is fundamental. A model gateway is an important control point, but it is not a universal control plane for every form of enterprise AI consumption.

Fig: Enterprise AI requires separate control paths for hosted applications and enterprise-controlled model traffic.

Architecture Principles

The reference architecture follows five principles.

Centralize policy without centralizing every decision

The enterprise needs common boundaries for identity, credentials, approved models, data, cost, and auditability. Application teams should remain free to select models and implementation patterns within those boundaries. The central platform should provide a governed path rather than become a manual approval queue.

Separate human, workload, and delegated identities

Employees authenticate through the corporate identity provider. Applications and agents use workload identities. When an agent acts for a person, the trace should preserve both the agent identity and the initiating user identity.

Keep provider credentials away from applications

Applications should authenticate to an enterprise-controlled gateway. The gateway protects and rotates provider credentials, maps workload identities to policies, and issues virtual credentials where needed.

Make data policy part of routing

Not every model or provider is approved for every data class. The eligible destination should be constrained by data sensitivity, contractual terms, region, retention policy, and workload risk before cost or performance optimization is considered.

Allow complexity to grow with need

The logical architecture can remain consistent while its implementation grows from cloud-native services to managed gateways, regional control planes, dedicated security tools, and self-hosted inference.

The Technology Landscape

The following table identifies representative implementation choices. These are examples, not recommendations for every organization.

Building block	Implementation choices	Representative examples
Human identity	Enterprise identity provider	Microsoft Entra ID, Okta, Google Workspace
Workload identity and secrets	Cloud IAM, workload federation, secrets platform	AWS IAM, Azure Managed Identities, Google Cloud IAM, HashiCorp Vault
Model gateway	Cloud-native, managed, self-hosted	Amazon Bedrock, Vertex AI, Azure AI Foundry, Portkey, LiteLLM, Kong, Cloudflare AI Gateway
Routing	Gateway rules, intelligent router, application-directed routing	Gateway policies, RouteLLM, Not Diamond, Martian, OpenRouter
Guardrails and AI security	Provider-native, gateway-integrated, dedicated platform, open-source framework	Amazon Bedrock Guardrails, Azure AI Content Safety, Google Model Armor, Portkey, Kong, Cloudflare, Check Point Lakera, NVIDIA NeMo Guardrails
First-party inference	Direct model-provider API	OpenAI, Anthropic, Google, Cohere, Mistral
Cloud inference	Managed cloud model platform	Amazon Bedrock, Google Vertex AI, Azure AI Foundry
Hosted open-model inference	Managed inference provider	Together AI, Fireworks AI, GroqCloud, Cerebras
Self-hosted inference	Enterprise-operated model serving	vLLM, NVIDIA NIM, Hugging Face TGI
Agent framework	SDK, graph framework, managed agent service	OpenAI Agents SDK, LangGraph, cloud-native agent services
AI observability	Gateway telemetry, specialized tracing, open telemetry	Langfuse, LangSmith, Helicone, Arize Phoenix, OpenTelemetry
AI cost governance and FinOps	Gateway budgets, attribution, forecasting, showback and chargeback	LiteLLM, Portkey, CloudZero, Finout, Vantage
Enterprise knowledge	Hosted knowledge feature, enterprise search, managed or custom RAG	Company knowledge in ChatGPT, Glean, cloud search services, vector and search platforms

1. Enterprise Identity and User Lifecycle

The corporate identity provider is the starting point for human access. Hosted AI products, coding applications, gateway consoles, observability platforms, and internal applications should use SSO wherever supported. Automated provisioning should create, update, suspend, and remove accounts as employees change roles or leave.

Groups and roles should represent approved populations such as general AI users, coding users, sensitive-data users, platform administrators, and model approvers. Identity establishes accountability and lifecycle control, but it does not determine what data a user may enter into an AI product. Data policy and product-specific administration remain necessary.

2. Governance for Standalone Hosted AI Applications

Standalone hosted applications sit outside the model-gateway path. The enterprise should maintain an approved product portfolio and evaluate each product for SSO, provisioning, administrative controls, retention, training terms, audit logs, regional processing, sharing, and connectors.

Connectors require particular care. When a hosted application connects to email, files, source code, or collaboration systems, it gains a new authorization path into enterprise information. Administrators must control which connections are available, which OAuth permissions are granted, and how access is revoked.

This is an architectural concern, but not a separate deployable product. It is implemented through identity, SaaS administration, procurement, CASB or DLP controls, connector governance, and organizational policy.

3. Workload Identity and Secrets

Every production workload should have an identity independent of the developer who created it. Production and non-production environments should not share identities or unrestricted credentials.

Where possible, use short-lived cloud workload credentials. When secrets are necessary, keep them in an enterprise secrets platform and rotate them through an established process. Applications should receive gateway credentials or virtual keys instead of underlying provider keys.

This allows a compromised application to be revoked independently, enables per-workload budgets and model permissions, and permits provider credentials to change without application redeployment.

4. Model Gateway

The model gateway is the shared access layer between enterprise-controlled workloads and model providers. It should authenticate the caller and evaluate attributes such as application, team, environment, data classification, use case, cost center, region, and requested capability.

The gateway capability set may include:

• Workload authentication and model authorization
• Provider credential protection and virtual keys
• Standard request and response interfaces
• Model aliases and provider abstraction
• Usage metering, budgets, quotas, and rate limits
• Routing, fallback, retries, and circuit breaking
• Audit events and trace propagation
• Caching where authorization permits it
• Input and output guardrails
• Sensitive-data detection and redaction
• Model and use-case allowlists
• Request-size and context limits
• Schema and structured-output validation

The gateway should not contain all business logic. End-user authorization, prompt construction, tool selection, and domain rules normally belong in the application or agent runtime.

5. Model Providers and Inference Platforms

Behind the gateway, the enterprise may use direct model-provider APIs, public-cloud AI platforms, specialized inference providers, or self-hosted open-weight models.

Cloud-native platforms are a practical starting point for companies already standardized on a cloud because they integrate with cloud identity, networking, billing, and regional controls. Direct providers may expose capabilities earlier but create an additional contract, credential, billing path, and data boundary. Self-hosted inference offers deployment control but transfers responsibility for scaling, security, availability, upgrades, and model serving to the enterprise.

The reference architecture permits these options to coexist behind a controlled access path.

6. Policy, Data Controls, and Guardrails

Policy connects enterprise requirements to runtime decisions. It may consider workload, user, environment, data class, model, provider, region, and budget before allowing or routing a request.

Guardrails should be layered:

Layer	Responsibility
Application or agent	User authorization, business rules, tool permissions, transaction limits, output validation
Gateway	Common input/output policies, data detection, model restrictions, schema checks, centralized enforcement
Model platform	Provider-native content and model safety controls
Dedicated AI security	Prompt-injection defense, data-leakage controls, adversarial detection for higher-risk workloads

Guardrail actions may block, redact, warn, log, retry, or reroute. No guardrail product can determine domain-specific correctness or eliminate the need for least-privilege application design.

7. Routing, Resilience, and Portability

Routing is a logical architecture block even when it is implemented inside the gateway.

Three patterns are available:

1. Gateway-integrated routing: The gateway selects a deployment using aliases, policy, cost, latency, availability, or region.
2. Dedicated intelligent routing: A router such as RouteLLM selects between eligible models based on predicted quality, complexity, or cost. The gateway remains the enforcement point.
3. Application-directed routing: The application selects a capability because it has the best task context, while the gateway verifies that the selection is allowed.

Fallbacks must satisfy the same data and compliance policy as the original destination. A technically available model is not automatically an approved fallback.

8. Agent Framework, Runtime, and Tool Governance

An agent framework defines agents, tools, handoffs, state, and execution flow. The runtime operates those workflows, including long-running execution, retries, queues, isolation, and resumability. The gateway controls model access but does not govern every action an agent can take.

The agent layer should provide:

• An approved framework or supported framework portfolio
• Workload and delegated user identity
• Tool and MCP-server registry
• Per-agent tool permissions
• Step, time, cost, and concurrency limits
• Isolated code execution
• Action-level audit logs
• Human approval for consequential actions
• Retry, cancellation, and kill-switch behavior

For high-impact actions, the model should propose the action while deterministic code verifies authorization and performs the operation.

9. Cost Governance and FinOps

The gateway should record application, team, environment, provider, model, usage, and estimated cost. This enables real-time budgets, alerts, quotas, and rate limits.

Enterprise FinOps has a broader responsibility. It must reconcile provider invoices and include supporting costs such as retrieval, databases, queues, agent compute, and self-hosted inference. As maturity increases, the progression is:

Metering → budgets and alerts → attribution → forecasting → optimization → showback or chargeback.

Optimization may include smaller-model routing, caching, prompt reduction, agent-loop limits, batching, or self-hosted inference. Quality and cost must be measured together.

10. Enterprise Knowledge and Retrieval

Employee knowledge access and application RAG are separate architectural needs.

Company knowledge in ChatGPT can search approved connected apps for employee questions and return cited answers while respecting source permissions. It is a feature inside ChatGPT Business and Enterprise, not a separate product or a reusable backend for custom applications.

Product applications require a retrieval architecture with ingestion, change detection, classification, permission synchronization, authorized retrieval, citations, and evaluation. Permission checks should occur before unauthorized content is provided to the model or written to traces.

Organizations may use a dedicated enterprise-search product, managed cloud services, or a custom shared platform depending on scale and requirements.

11. Monitoring, Tracing, Audit, and Evaluation

The architecture produces four kinds of operational evidence:

• Infrastructure monitoring for availability, latency, failures, and resource health
• AI traces across retrieval, model calls, and tool actions
• Security and compliance audit records
• Quality evaluations using test datasets, production feedback, automated scoring, and human review

Telemetry requires its own data policy. Prompts, responses, retrieved documents, and tool output may contain sensitive information. The enterprise must define redaction, retention, encryption, regional storage, and operator access.

Cloud-Native Access or a Dedicated Gateway?

A cloud-native model platform may be sufficient when most workloads use one cloud, the model portfolio is limited, cloud identity and billing provide enough control, and cross-provider routing is unnecessary.

A dedicated gateway becomes valuable when applications use multiple providers, credentials must be isolated centrally, budgets and policies must apply across teams, or common routing, guardrails, and audit records are required.

The two approaches are complementary: a dedicated gateway can use cloud-native model platforms as backends.

Managed, Self-Hosted, or Built Internally?

Managed products reduce operating effort. Self-hosted products increase deployment and data control but require upgrades, security maintenance, scaling, and availability ownership. Internal development should be reserved for requirements that available products cannot satisfy.

A model proxy or RAG demonstration may be simple. A production platform with identity, policy, streaming, retries, accounting, permissions, deletion, tracing, and high availability is not.

From Reference Architecture to Prescription

The architecture is intentionally broader than any one organization should deploy immediately. A small company may need only an approved employee assistant, one cloud model platform, and basic budgets. A mid-size organization may need a managed gateway, shared tracing, and application-level RAG. A large enterprise may require regional gateways, multi-provider routing, dedicated guardrails, chargeback, a shared agent runtime, and self-hosted components.

The next article, Prescribed Enterprise AI Architectures for Small, Mid-Size, and Large Organizations, selects a coherent implementation and representative vendors for each profile.