The first article in this series, Enterprise AI Adoption: Requirements for Security, Governance, and Scale, defined the problem: enterprises consume AI through coding tools, standalone hosted applications, custom agents, and enterprise knowledge systems, but need consistent controls across identity, credentials, data, cost, providers, and operations.
The second article, A Reference Architecture for Governed Enterprise AI, translated those requirements into architectural building blocks and described the available implementation choices.
This article makes the opinionated decisions. It recommends what to implement, what to defer, and which representative products to consider for three enterprise profiles.
Product recommendations reflect the market as of July 2026 and should be revalidated before implementation.
Scope and Assumptions
Company size is the primary organizing variable because adding industry, geography, regulation, data sensitivity, cloud provider, and AI maturity as independent dimensions would make a simple prescription impossible.
The recommendations assume a software-enabled company with moderate data sensitivity, one primary public cloud, an existing corporate identity provider, both employee and custom AI usage, and a preference for managed services unless scale or control justifies ownership.
Size remains a proxy. Move to the next architecture tier when regulation, data sensitivity, AI expenditure, workload volume, geographic isolation, or operational complexity exceeds what is typical for the company profile.
| Profile | Working definition | Operating reality |
|---|---|---|
| Small | 5–50 engineers | No dedicated AI platform team; CTO or engineering lead owns AI |
| Mid-size | 50–500 engineers | Emerging platform team; shared security and FinOps responsibilities |
| Large | 500+ engineers | Dedicated platform, security, data, and FinOps teams; multiple business units |
Engineering scale is more useful than total employee count because it better predicts the number of custom workloads and the organization’s ability to operate platform components.
Small Enterprise: Prefer Simplicity
The small-enterprise architecture deliberately accepts coarser attribution and fewer centralized controls in exchange for low operating overhead.
Prescription
| Capability | Recommendation |
|---|---|
| Employee AI | ChatGPT Business |
| Coding assistant | Codex under the managed company workspace |
| Human identity | Existing Google Workspace or Microsoft Entra ID |
| Workload identity | Primary-cloud IAM and secrets services |
| Model inference | Existing cloud’s managed AI platform |
| Model gateway | None initially |
| Model selection | Static aliases in shared application configuration |
| Routing | No dedicated intelligent router |
| Guardrails | Provider-native controls plus application validation |
| Agent framework | Application-selected SDK; no enterprise standard |
| Agent runtime | Existing application infrastructure |
| Knowledge | Approved user connectors; no enterprise RAG platform |
| Observability | Cloud telemetry and application traces |
| FinOps | Provider budgets, cloud billing, and monthly review |
| Self-hosted inference | Do not implement |
Hosted AI and coding
Use ChatGPT Business as the company-managed employee workspace and Codex as the standard coding assistant. Codex is included in ChatGPT Business, although usage limits and credit options depend on the plan. This reduces unmanaged subscriptions and keeps general AI and coding under one commercial and administrative relationship. OpenAI documents current Codex plan availability here.
Use SSO where available, central billing, approved data terms, and an explicit user-removal process. ChatGPT Business supports SSO but does not provide SCIM directory synchronization, so lifecycle management remains partly administrative. OpenAI documents the Business and Enterprise identity differences here.
Model access without a gateway
Applications call the primary cloud’s managed model platform directly using distinct workload identities. AWS-centric organizations use Amazon Bedrock, Google Cloud organizations use Vertex AI, and Microsoft-centric organizations use Azure AI Foundry. A company without meaningful cloud alignment may use a direct OpenAI or Anthropic API organization.
A shared client library may standardize model aliases, metadata, timeouts, retries, and trace identifiers. It is not a gateway and cannot guarantee centralized enforcement.
Attribution is therefore based on separate cloud projects or accounts, workload identities, billing tags, provider reports, and application logs. This is adequate while the workload portfolio is small, but it may not support reliable per-user or per-feature allocation.
Routing and guardrails
Use static model aliases such as general-fast, general-quality, and sensitive-approved. A centrally maintained configuration maps each alias to an approved cloud model deployment. Cloud IAM should restrict workloads to eligible deployments where supported.
Use provider-native guardrails and application controls. The application remains responsible for user authorization, tool permissions, structured output, agent limits, and approval of consequential actions.
Agents and knowledge
Do not standardize an agent framework yet. The first application can use an appropriate SDK and run within existing application infrastructure. Every agent still needs narrow tool permissions, least-privilege credentials, step and cost limits, and human approval for irreversible actions.
Do not build an enterprise knowledge platform. Employees can use administrator-approved connectors in the hosted workspace, subject to source permissions. Build retrieval only for a product workflow with a measurable requirement.
Graduation trigger
Introduce a managed gateway when the organization has several production AI applications, more than one provider, duplicated integrations, shared credential problems, or a need for centralized budgets, guardrails, routing, and detailed attribution.
Mid-Size Enterprise: Introduce a Shared Control Plane
The mid-size architecture requires all custom production model traffic to pass through a dedicated gateway. The platform remains primarily managed because the organization has an emerging platform team rather than a large AI infrastructure organization.
Prescription
| Capability | Default recommendation | Alternative |
|---|---|---|
| Employee AI and coding | ChatGPT Enterprise standard seats, including Codex | Codex-only seats for restricted populations |
| Human identity | Entra ID or Okta with SSO and SCIM | Google Workspace |
| Workload identity | Cloud IAM plus gateway virtual credentials | Vault where already established |
| Model gateway | Managed Portkey | Self-hosted LiteLLM |
| Inference | Primary cloud model platform plus one direct provider | Two approved cloud platforms |
| Routing | Gateway aliases, conditional rules, and fallback | LiteLLM routing |
| Guardrails | Provider-native, Portkey, and application controls | Check Point Lakera for higher-risk workloads |
| Agent framework | OpenAI Agents SDK | LangGraph for durable workflows |
| Agent runtime | Existing container or serverless platform | Dedicated runtime when volume justifies it |
| Knowledge | Company knowledge in ChatGPT plus application RAG | Enterprise search when justified |
| Observability | Portkey plus Langfuse Cloud | LiteLLM plus self-hosted Langfuse |
| FinOps | Gateway budgets plus cloud billing | Finout or CloudZero |
| Self-hosted inference | Generally defer | Workload-specific exception |
Enterprise workspace
Use standard ChatGPT Enterprise seats for employees and engineers. A standard seat includes ChatGPT and Codex and offers the greatest flexibility. Codex-only seats should be reserved for people who need coding access but should not receive the broader ChatGPT workspace. OpenAI’s current Enterprise documentation describes both seat types and their access boundaries.
Use enforced SSO, SCIM, groups, and role-based access. Separate groups should cover general AI users, Codex users, AI developers, production operators, platform administrators, and sensitive-data users.
Gateway and model portfolio
Use managed Portkey as the default gateway. It provides multi-provider access, virtual credentials, routing, budgets, rate limits, retries, circuit breaking, observability, and integrated guardrails without requiring the emerging platform team to operate the full gateway stack. Portkey documents these gateway capabilities and deployment options.
Use the primary cloud model platform as the main backend and one approved direct provider as a secondary path. All production applications authenticate to the gateway; provider credentials remain with the platform team.
Use logical model aliases, policy restrictions, availability fallbacks, and controlled model-version rollout. Do not add a dedicated intelligent router until expenditure is material and evaluation data can prove that dynamic routing preserves quality.
Guardrails, agents, and tools
Apply provider-native controls, gateway input/output guardrails, and application-specific rules. Portkey guardrails support checks and actions on gateway requests and responses. Add Check Point Lakera only for public-facing or higher-risk workloads that justify specialized prompt-injection and data-leakage protection.
Adopt the OpenAI Agents SDK as the default for straightforward tool-using agents. Allow LangGraph for durable, stateful, resumable, or human-in-the-loop workflows. Run agents on the existing container or serverless platform and provide shared standards for identity, tool registration, MCP servers, traces, execution limits, approvals, and action logs.
Knowledge, observability, and FinOps
Use company knowledge in ChatGPT Enterprise with administrator-approved apps for employee search. Build a shared managed-cloud retrieval foundation only when multiple production applications need common ingestion, permissions, citations, and evaluation.
Use Portkey for gateway-level telemetry and Langfuse Cloud for multi-step traces and evaluations where deeper analysis is required. Use gateway budgets for enforcement and the existing cloud-finance process for reporting. Add Finout or CloudZero when model and supporting cloud costs span providers and require formal allocation or forecasting.
Graduation trigger
Move to the large-enterprise pattern when the company needs regional isolation, multiple business-unit control, formal chargeback, private deployment, dedicated AI security, a shared agent runtime, or self-hosted inference.
Large Enterprise: Central Policy, Federated Delivery
The large-enterprise architecture supports multiple providers, regions, business units, and control teams. Policy and visibility are centralized, while application ownership remains federated.
Prescription
| Capability | Default recommendation | Alternative |
|---|---|---|
| Employee AI and coding | ChatGPT Enterprise standard seats, including Codex | Codex-only seats for restricted populations |
| Identity | Entra ID or Okta with SSO, SCIM, groups, and RBAC | Existing enterprise IdP |
| Workload identity | Cloud workload federation plus gateway virtual keys | Vault where standardized |
| Model gateway | Regional self-hosted LiteLLM Enterprise | Managed or self-hosted Portkey Enterprise |
| Inference | Cloud AI platforms plus direct OpenAI and Anthropic | Additional approved providers |
| Routing | Gateway policy routing plus evaluated intelligent routing | Application-directed routing |
| Guardrails | Provider controls, gateway policy, and Check Point Lakera | NVIDIA NeMo Guardrails |
| Agent framework | OpenAI Agents SDK plus LangGraph where justified | Approved framework by exception |
| Agent runtime | Shared Kubernetes-based runtime | Managed cloud agent runtime |
| Knowledge | Company knowledge in ChatGPT plus shared product RAG | Glean for dedicated enterprise search |
| Observability | Self-hosted Langfuse Enterprise plus OpenTelemetry | Managed enterprise platform |
| FinOps | CloudZero | Finout |
| Self-hosted inference | vLLM for selected stable workloads | NVIDIA NIM |
Regional gateway and inference platform
Deploy LiteLLM Enterprise in approved regions as the standard model-access plane. Each deployment should provide high availability, business-unit isolation, local credentials, virtual keys, budgets, quotas, model aliases, audit records, guardrail integration, and approved fallback policies. LiteLLM documents virtual-key, authentication, SSO, and enterprise gateway capabilities in its product overview.
Applications should not receive direct provider credentials except through documented exceptions. The model portfolio can include strategic cloud AI platforms, direct OpenAI and Anthropic access, selected inference providers, and self-hosted open-weight models for justified workloads.
Expose internal capability aliases such as enterprise-fast, enterprise-reasoning, enterprise-code, regulated-region, and open-weight-private rather than embedding provider model names throughout applications.
Routing and guardrails
Apply policy routing to every request. Region, data classification, business unit, provider approval, and budget determine the eligible model set before performance or cost optimization.
Use intelligent routing only for workloads with evaluation datasets and measurable quality thresholds. RouteLLM or another router may select among eligible models, but the gateway remains the enforcement point.
Use four guardrail layers: provider-native safety, gateway policy, Check Point Lakera for high-risk workloads, and application business controls. Measure false positives, latency, and bypass resistance rather than treating guardrails as absolute security boundaries.
Agent platform
Provide an approved framework portfolio and a shared runtime. Use the OpenAI Agents SDK for the default tool-using pattern and LangGraph for durable workflows. The runtime should provide workload and delegated identity, queues, resumability, isolated code execution, an approved tool and MCP registry, secrets injection, limits, approvals, traces, cancellation, and kill switches.
The gateway governs model inference. The agent runtime governs execution and tools. Deterministic services should verify authorization and execute consequential actions proposed by a model.
Enterprise knowledge
Use company knowledge in ChatGPT Enterprise with approved apps for employee search and synthesis. This is a ChatGPT feature, not a separate product and not a backend for custom applications.
Operate a shared permission-aware RAG platform for product applications. It should provide connectors, ingestion, permission synchronization, deletion, retrieval APIs, citations, evaluation, and regional isolation.
Consider Glean when dedicated cross-enterprise search is a strategic requirement that company knowledge in ChatGPT does not satisfy. Do not deploy both without defining their distinct user populations and responsibilities.
Observability and FinOps
Use self-hosted Langfuse Enterprise for traces, evaluations, prompt management, datasets, RBAC, retention policies, and audit logs. These self-hosted enterprise controls are described in Langfuse’s deployment and pricing documentation. Export platform telemetry through OpenTelemetry and security events to the enterprise SIEM.
Use gateway budgets for real-time enforcement and CloudZero for allocation, anomaly detection, forecasting, unit economics, and showback or chargeback across inference and supporting infrastructure.
Self-host models through vLLM only for workloads with stable demand, demonstrated economics, suitable model quality, and an operations team able to own serving and upgrades.
How the Three Architectures Progress
| Capability | Small | Mid-size | Large |
|---|---|---|---|
| Employee workspace | ChatGPT Business | ChatGPT Enterprise | ChatGPT Enterprise with enterprise policy |
| Coding | Codex | Codex in standard seats | Codex with role and spend controls |
| Gateway | None | Managed Portkey | Regional LiteLLM Enterprise |
| Providers | One primary platform | Primary plus one secondary | Governed multi-provider portfolio |
| Routing | Static configuration | Gateway rules and fallback | Policy plus evaluated intelligent routing |
| Guardrails | Provider plus application | Provider, gateway, application | Provider, gateway, dedicated security, application |
| Agents | Application-selected SDK | Standard SDK and shared patterns | Framework portfolio and shared runtime |
| Knowledge | User connectors | Company knowledge plus application RAG | Company knowledge plus shared product RAG |
| Observability | Cloud and application logs | Portkey plus Langfuse Cloud | Self-hosted Langfuse Enterprise plus OTel |
| FinOps | Budgets and monthly review | Gateway budgets and optional FinOps platform | CloudZero with showback or chargeback |
| Self-hosted inference | No | Usually no | Selected workloads only |
When Size Is Not Enough
Use a stronger architecture tier when any of the following applies:
- Regulated or highly sensitive data
- Mandatory data residency or private networking
- High AI expenditure or customer-facing AI cost of revenue
- Many autonomous agents with consequential tools
- Multiple cloud providers or geographic regions
- Formal business-unit isolation or chargeback
- Strict availability requirements
- A large portfolio of production AI applications
A small healthcare or financial company may need large-enterprise security controls. A large company with limited AI use may begin with the mid-size platform. Size determines the default, not the exception policy.
Final Recommendation
The prescribed progression is intentionally conservative:
- Small: one managed employee workspace, Codex, one model platform, no gateway, no enterprise RAG, and no dedicated FinOps platform.
- Mid-size: ChatGPT Enterprise and Codex, a managed gateway, two inference paths, shared guardrails and tracing, and application-focused RAG.
- Large: regional gateways, a governed provider portfolio, layered AI security, a shared agent runtime, enterprise retrieval, formal observability, and chargeback.
The architecture should become more sophisticated only when usage, risk, cost, or organizational scale creates a concrete reason. The objective is not to deploy every available AI platform component. It is to provide the minimum architecture that enables adoption without losing control.
Sreenivas Makam's Blog 