Category Archives: AI

7 Common Misconceptions When Choosing an AI Model

AI, , , , , , , , , ,

Choosing an AI model often appears straightforward: compare the leaderboard, check the token price, look at the context window, and select the winner.

In production, it is rarely that simple.

One resource I regularly use is Artificial Analysis, which compares models across intelligence, speed, and cost per task. For an actual product, I translate this into three broad dimensions:

  • Intelligence for the capabilities my system needs
  • End-to-end latency
  • Cost per successful task

Even these dimensions cannot be evaluated in isolation. A model only becomes useful when it is placed inside a system with prompts, context, tools, validation, recovery mechanisms, and real users.

Here are seven common misconceptions I have seen people make when choosing an AI model.

The benchmark figures in this article reflect published results available in August 2026 and may change as evaluations and models are updated.

1. At similar intelligence, a lower token price means a cheaper model

Token price and task cost are not the same thing.

Consider Kimi K3 and GPT-5.6 Sol. Kimi K3’s output-token price was $15 per million tokens, compared with $30 for GPT-5.6 Sol.

Based on the headline price, Kimi K3 appeared to be 50% cheaper.

However, Artificial Analysis estimated their cost per Intelligence Index task at:

  • Kimi K3: $0.94
  • GPT-5.6 Sol: $1.04

That is only about a 10% difference in task cost.

The reason is token efficiency. A model with cheaper tokens may consume more reasoning and output tokens to complete the same work. Input tokens, cache reads and writes, and answer tokens also contribute to Artificial Analysis’s task-cost calculation.

In a production system, the calculation can become even broader. Retries, failed outputs, tool calls, validation steps, and human corrections can all affect the final cost.

The lesson is:

Among models that provide the required intelligence, compare cost per successful task—not merely cost per token.

Read the Kimi K3 analysis and the GPT-5.6 Sol analysis.

2. The benchmark leader is the best model

Intelligence is not one universal capability.

A model may excel at coding but perform less well in automation, knowledge work, tool use, visual reasoning, instruction following, or presentation.

Kimi K3 illustrates this difference. It scored 57 on the overall Artificial Analysis Intelligence Index, behind GPT-5.6 Sol at 59. But Kimi K3 took the leading position on AutomationBench-AA at launch, scoring 53%.

Which model was better?

It depended on the task.

There is another reason to treat public leaderboards carefully: benchmark contamination. Evaluation questions or closely related material may appear in training data, potentially inflating the reported performance. Models may also be repeatedly optimized against familiar public evaluations without those improvements generalizing to new workloads.

This does not make benchmarks useless. They remain valuable for discovery and shortlisting. But the final decision should come from private evaluations based on:

  • Your actual prompts and data
  • The capabilities your workflow depends on
  • Your tools and system instructions
  • Known failure conditions
  • Required output formats
  • Your definition of a successful task

Research on benchmark-data contamination explains why public scores may not always represent performance on unseen tasks.

The lesson is:

Use public benchmarks to create the shortlist. Use workload-specific evaluations to select the model.

3. Open-weight models are cheaper

Open-weight models can be cheaper, particularly when there is enough sustained demand to use the underlying infrastructure efficiently.

But self-hosting also introduces costs:

  • GPUs and infrastructure
  • Capacity planning and utilization
  • Deployment engineering
  • Monitoring and reliability
  • Security
  • Model upgrades
  • Operational support

Cost may not even be the strongest reason to choose an open-weight model.

Control, privacy, deployment flexibility, data residency, offline operation, and independence from a hosted provider may be more important.

A recent Hugging Face security incident provides a powerful example. During its investigation, Hugging Face needed to analyze real attack commands, exploit payloads, and command-and-control artifacts. The commercial models it initially tried blocked those requests through their safety guardrails.

Hugging Face instead ran the open-weight GLM 5.2 on its own infrastructure. This also ensured that attacker data and referenced credentials did not leave its environment.

Importantly, Hugging Face did not identify which model powered the attacker’s system. Its conclusion was about defensive readiness: organizations may need a capable model they can operate within their own environment when hosted services cannot support the workflow.

Read the Hugging Face security-incident disclosure.

The lesson is:

Choose open-weight models for control and operational independence—not because they are automatically cheaper.

4. Domain-specific knowledge requires fine-tuning

Fine-tuning is one way to build a domain-specific system. It is not the only way.

Depending on the problem, domain knowledge can be introduced through:

  • Detailed prompting
  • In-context examples
  • Retrieval-augmented generation
  • SQL or structured-data retrieval
  • APIs and tool calling
  • Knowledge graphs
  • Deterministic business rules
  • User or session memory
  • Fine-tuning
  • A combination of these approaches

A useful distinction is to separate knowledge from behavior.

If the information already exists in documents, databases, or business systems—and changes regularly—it may be better to retrieve it at runtime.

If the model must consistently learn a new task pattern, classification boundary, response structure, tool-selection behavior, or specialized style, fine-tuning may be appropriate.

AWS explored this using Amazon Nova models and AWS-specific questions. In its experiment, both RAG and fine-tuning improved the average evaluated response score by approximately 30% over the base Nova Lite model. Combining fine-tuning with RAG produced the strongest improvement.

This was a limited experiment involving ten domain-specific questions and LLM-based judges, so the percentages should not be treated as a universal rule. The valuable result is that the appropriate architecture depends on the problem—and sometimes the best answer is a combination.

Read the AWS comparison of RAG, fine-tuning, and a combined approach.

The lesson is:

Domain-specific knowledge does not automatically require fine-tuning. First identify whether you are solving a knowledge problem, a behavior problem, or both.

5. The biggest model is always the best model

The most capable frontier model may produce the strongest answer, but that does not mean it creates the best user experience.

For bounded tasks such as classification, extraction, autocomplete, short summarization, or on-device assistance, a smaller model may provide:

  • Lower latency
  • Better privacy
  • Offline availability
  • Predictable cost
  • Reduced network dependence
  • Sufficient intelligence for the task

Apple’s Foundation Models guidance provides a practical example. Apple positions its on-device model for lightweight, latency-sensitive, privacy-sensitive, and offline tasks. When an application needs deeper reasoning or a larger context window, it can use a more capable server model through Private Cloud Compute.

The point is not that smaller models are always faster or better. Their performance still depends on the device, optimization, task, and model architecture.

The principle is:

Use the smallest model that reliably meets the intelligence, latency, and operational requirements of the task.

Read Apple’s documentation on on-device Foundation Models and server-side intelligence through Private Cloud Compute.

6. The same context-window size means the same context understanding

A context window describes how much information a model can accept. It does not guarantee how effectively the model can use that information.

NVIDIA’s RULER benchmark makes this distinction concrete.

Qwen3-235B-A22B and Mistral-Large-2411 both advertise 128K-token context windows. At 128K on RULER, their reported scores were:

  • Qwen3-235B-A22B: 90.6
  • Mistral-Large-2411: 48.1

RULER classified Qwen’s effective context length as greater than 128K, while Mistral-Large-2411’s effective length was approximately 64K.

The input could fit inside both models. Their ability to use it was substantially different.

Long-context performance can depend on whether a model can:

  • Find relevant details
  • Preserve instructions from earlier in the prompt
  • Connect information across distant sections
  • Ignore irrelevant material
  • Perform aggregation and multi-hop reasoning

The exact RULER results are one benchmark rather than a universal ranking. They nevertheless demonstrate why advertised context capacity should not be treated as effective context intelligence.

Review the NVIDIA RULER benchmark and results.

The lesson is:

Context capacity tells us what a model can accept. Effective context tells us how much it can use reliably.

7. Using the best model produces the best product

Users do not interact with a model in isolation. They interact with a complete system.

The harness around a model may manage:

  • Prompt and context construction
  • File and repository access
  • Tools and permissions
  • State and memory
  • Validation
  • Retries and recovery
  • Stopping conditions
  • Output formatting
  • Observability and human escalation

A coding-agent benchmark called Claw-SWE-Bench demonstrates how much this can matter.

The researchers evaluated OpenClaw with the same GLM 5.1 model using two different adapters.

With a minimal coding adapter, it scored 19.1% Pass@1. With a full repository-editing adapter, it scored 73.4%.

The full adapter placed the agent in the correct repository workspace, allowed it to edit files directly, extracted the resulting patch from the repository state, removed unrelated artifacts, and produced the format required by the evaluator.

The model did not become more intelligent. The surrounding system became better at converting its intelligence into a valid result.

The dramatic difference partly reflects correct integration with the benchmark’s patch-submission contract, so it should not be interpreted as proof that every harness improvement will produce a fourfold gain. However, the study’s broader controlled comparisons also found meaningful performance differences when the model was held fixed and the harness changed.

Read the Claw-SWE-Bench paper.

The lesson is:

Model intelligence is potential. The harness determines how effectively that potential becomes completed work.

A more practical model-selection process

Instead of asking, “Which model is best?”, start with a more specific set of questions:

  1. What intelligence does this workload require?
  2. Which failures are unacceptable?
  3. What end-to-end latency can users tolerate?
  4. What is the cost per successful task?
  5. How effectively does the model use the context we provide?
  6. Does the model work reliably with our tools and harness?
  7. What privacy, deployment, and operational controls do we need?
  8. How does it perform on our own evaluation set?

The best model is rarely the one with the most impressive number in a single column.

It is the model—and the surrounding system—that delivers the required intelligence, latency, reliability, control, and task economics for the application being built.

What misconceptions or unexpected quirks have you encountered while selecting models for production?

Why We Built Model Academy: Understanding How AI Models Produce Answers

AI, , , , , , , ,

Over the past few years, most of my work and learning in AI has been closer to the application layer—understanding models, building AI applications, and exploring how businesses can use them.

I knew the foundational ideas behind models, weights, prompts, and tokens, but I had not explored inference infrastructure and GPU execution in sufficient depth. I wanted to understand what really happens after an application sends a prompt to a model.

How is the model loaded and prepared for inference? What happens during prefill and decode? How are multiple GPUs used? What is a GPU kernel? How do warps, schedulers, compute units, and memory work together to produce the next token?

These concepts are often explained separately. You might find one resource about model weights, another about inference serving, and a highly technical tutorial about GPU internals. It is much harder to find a single learning journey that connects all these layers—from a trained model to the answer displayed in an application.

That gap became the motivation for creating Model Academy.

Two complementary perspectives

My background is primarily in networking and, more recently, AI applications. Systems concepts are familiar to me, but I had not previously gone deeply into modern AI infrastructure or the internal execution model of GPUs.

Ritesh Dhoot has spent considerably more time working with AI infrastructure and understands this part of the stack in much greater depth. I brought the perspective of an application builder trying to understand what happens beneath the interfaces we use every day.

That combination worked particularly well.

As we discussed each concept, we repeatedly asked two questions:

  1. Is this technically accurate?
  2. Can someone without a deep GPU or infrastructure background understand it?

The second question was just as important as the first. If I found a concept difficult to connect to the larger story, it was a useful signal that the explanation needed to be simplified, visualized, or placed in better context.

One connected journey

Model Academy follows the complete path from models to answers.

It begins with how models are built, what their weights contain, and what a model release actually provides. It then follows a request through an inference system: request preparation, model loading, batching, prefill, decode, KV cache, and multi-GPU execution.

From there, the journey moves inside the GPU to explain kernels, thread blocks, warps, scheduling, compute, and memory movement. It concludes by replaying one prompt from the initial request to the final streamed answer.

The academy is intended for engineers, architects, technology leaders, and curious learners who want a practical mental model of the complete system. The goal is not to turn every reader into a GPU programmer. It is to help people understand how the layers fit together and why different architectural decisions matter.

Learning through interaction

We strongly believe that complex systems are easier to understand when you can see them operating.

For that reason, Model Academy is not only a collection of written chapters. We have tried to use animations and interactive exercises throughout the learning journey.

Instead of merely reading that temperature changes a token distribution, you can change it and observe the result. Instead of reading a list of inference stages, you can follow a prompt through request preparation, model execution, and token generation. You can also zoom inside a GPU and explore how kernels, blocks, warps, schedulers, compute units, and memory work together.

The academy provides a Beginner mode that explains the complete journey in approachable language. Expert mode progressively exposes the mechanisms underneath without requiring a separate curriculum.

The first part of a larger learning series

This academy covers the journey from models and weights through inference systems and into GPU execution. But we see it as the first part of a broader learning series.

Over time, we would like to explore more of the systems operating “under the hood” of AI: model architectures, training infrastructure, inference optimization, distributed systems, networking, accelerators, and other topics that connect AI applications to the platforms beneath them.

For now, we hope Model Academy makes an intimidating subject more approachable, visual, and connected.

Explore the academy at ModelAcademy.tech. The project is also available as an open-source repository on GitHub.

We welcome your feedback and hope the academy helps you understand—and confidently explain—how AI models produce answers.

Model Academy is created and maintained by Sreenivas Makam and Ritesh Dhoot.

AI in CI Should Not Replace Your Tools. It Should Read the Evidence.

AI, Programming, , , , , , , , , , ,

In Part 1, I compared three approaches to AI-assisted code review: hosted Codex review, a repository-owned GitHub Action using Codex, and CodeRabbit. That experiment intentionally focused on one stage of the pull request lifecycle: review comments on code.

This article moves one step wider.

Once a pull request is opened, code review is only one part of the continuous integration story. A real CI pipeline also has static analysis, security checks, unit/API tests, maybe dynamic analysis, and some final judgement about whether the PR is ready for a human reviewer.

The question I wanted to answer was not:

Can AI replace CI tools?

That is the wrong framing.

The better framing is:

Can deterministic tools produce evidence, and can AI turn that evidence into a useful reviewer-facing judgement?

That is the architecture I tested.

The design principle

The core design principle is simple:

Deterministic tools produce evidence.
AI interprets the evidence.
The human reviewer makes the final decision.

AI is not the linter. AI is not the security scanner. AI is not the test runner. AI should not be trusted as the source of truth for whether code actually passed.

Instead:

  • The static-analysis tool should decide whether linting and formatting passed.
  • The security scanner should report known security patterns.
  • The test framework should execute tests.
  • AI can generate additional targeted tests based on the PR diff.
  • pytest should still execute those AI-generated tests.
  • AI can then read the outputs and produce a final review summary.

In my experiment, those tools were Ruff, Bandit, and pytest. In another company, they could be different. A Java service might use Checkstyle, SpotBugs, PMD, Semgrep, Maven, Gradle, JUnit, or Snyk. A JavaScript service might use ESLint, TypeScript, npm audit, Playwright, Jest, or Vitest. A mobile team might use SwiftLint, Detekt, XCTest, Espresso, or Appium.

The specific tool is not the point. The point is that each tool should do the thing it is built to do. AI should assist around those tools: fill gaps, generate targeted probes, correlate outputs, and produce a reviewer-facing summary.

That gives us a more scalable pattern than asking one model to “review everything.”

The experiment

I used the same repository from the code-review experiment:

The base workflow is implemented in .github/workflows/ci-evidence-report.yml. During the blocked-path experiment, I added a deterministic evidence summary and a separate readiness gate; that refined seven-stage version is preserved in the PR #8 experiment commit.

The AI prompts live in:

The base workflow has five core jobs. The refined experiment adds two orchestration jobs, producing seven distinct stages:

  1. Static analysis
  2. Security analysis
  3. Backend unit/API tests
  4. AI-generated targeted tests
  5. Deterministic CI evidence summary
  6. AI final reviewer report
  7. CI readiness gate

The workflow also uploads artifacts between jobs. That matters because the AI final-report job should not rely on vague impressions. It should read concrete outputs: Ruff JSON, Ruff format output, Bandit JSON, pytest XML/output, and generated-test output. GitHub Actions artifacts are designed for exactly this kind of cross-job evidence sharing.

What the AI was instructed to do

There were two separate AI responsibilities, with two separate prompts.

The first prompt, targeted-test-generation.md, asks the AI to inspect the PR diff and generate temporary pytest tests only when the changed behavior justifies it. The important constraints are:

  • focus on risky changed backend behavior;
  • generate tests under the allowed temporary path only;
  • do not modify production code;
  • do not treat the model’s judgement as the test result;
  • let pytest execute the generated tests.

The second prompt, ci-final-report.md, asks the AI to read the evidence artifacts and produce a PR comment for the manual reviewer. It is not supposed to invent results. It reads files like:

  • static-analysis-results/ruff-check.json
  • static-analysis-results/ruff-format.txt
  • security-analysis-results/bandit.json
  • backend pytest output
  • AI-targeted pytest output

That final report should answer a different question from the test-generation prompt:

Given all the evidence, is this PR ready for manual review, and where should the reviewer focus?

What the pipeline looked like

At a high level, the pipeline looked like this:

Deterministic checks control readiness. AI turns their evidence into a reviewer-facing explanation.

The diagram shows the refined architecture. The separate evidence-summary and readiness-gate jobs were added while iterating on PR #8; the earlier clean PR #11 run used the five-job base workflow.

The “AI code review” box is the bridge to the first article. In the actual Blog 2 workflow, I focused on CI evidence. But in a complete implementation, code-review findings should also become one more input to the final reviewer report.

The most important detail is the separation between generation and execution.

For AI-generated tests, Codex is allowed to write a temporary pytest file under a controlled path:

ci-generated-tests/backend/test_pr_targeted.py

But Codex does not decide whether the tests pass. pytest does.

That distinction is important. The model can suggest a hypothesis. The test runner proves or disproves it.

Static analysis: let the deterministic tool do its job

For static analysis, I used Ruff:

ruff check backend --output-format=json
ruff format --check backend

Ruff is fast, deterministic, and well suited to CI. This is exactly the kind of work AI should not replace.

In another stack, replace Ruff with the tool the team already trusts. The workflow remains the same: run the tool, store the output, and let AI interpret that output alongside other CI signals.

One practical lesson showed up immediately: the existing repo baseline was not fully Ruff-clean. That created noise. So I first created a baseline cleanup PR:

That PR is not the core blog experiment. It was housekeeping. But it was necessary because PR-level CI evidence becomes much clearer when the baseline is already clean.

This is a general lesson: before you use CI evidence to judge a PR, decide whether your tools are checking the whole repo, changed files, or changed lines. If the baseline is noisy, the AI report will also become noisy.

Security analysis: deterministic scanner first, AI second

For security analysis, I used Bandit:

bandit -r backend/app -f json -o ci-results/security/bandit.json

Bandit is a static security scanner for Python. In this experiment, it was not meant to prove the application is secure. It was meant to provide a deterministic signal for known classes of Python security issues.

That distinction matters.

If Bandit finds a high-signal issue, the AI report can explain the impact and connect it to the PR. If Bandit finds nothing, the AI report should not overstate that as “the code is secure.” It should say something closer to:

Bandit did not report findings in the scanned backend application code. This does not prove runtime authorization, data access, or business-logic security.

That is the kind of nuance AI is useful for.

Testing: existing tests plus AI-generated targeted tests

There are three different testing layers that are easy to confuse:

  1. Tests the developer writes before opening the PR.
  2. Existing CI tests that protect known behavior.
  3. Temporary AI-generated targeted tests created from the PR diff.

This experiment focused on the second and third.

The normal backend test job ran the existing pytest suite. Separately, the AI-targeted-test job asked Codex to inspect the PR diff and generate a temporary pytest file for risky changed behavior.

The generated tests were not committed back to the repository. They were CI evidence.

That is a useful pattern. AI-generated tests do not have to become permanent tests every time. They can still be valuable as temporary review probes:

  • Does this PR change an authorization path?
  • Does this PR change field mapping?
  • Does this PR change input normalization?
  • Does this PR change error behavior?

If the generated test exposes a defect, the PR should be blocked. If the generated test is genuinely useful long term, the developer can convert it into a committed regression test.

That last point matters. AI-generated tests should not remain temporary forever if they expose a real gap. Once a targeted generated test catches an important issue, the baseline test suite should be updated with a durable version of that test. Otherwise the pipeline has to rediscover the same risk again in a future PR.

The bad-path PR

To test whether the pipeline could catch real problems, I created an intentionally bad PR:

This PR intentionally included multiple defects:

  • an access-control regression;
  • a registration-correction mapping bug;
  • a security-pattern issue for Bandit/static security evidence.

The result was what we wanted: deterministic jobs produced evidence, the AI-generated targeted tests failed, and the CI readiness gate blocked the PR.

The blocked path: evidence was produced successfully, but the readiness gate failed.

The static-analysis and security-analysis jobs appear green because they completed their work and uploaded evidence instead of stopping the workflow immediately. Their tool-status files still recorded failures. The readiness gate read those internal statuses and blocked the PR. This allowed the AI report to be generated even when scanners found problems.

The important part is not just that something failed. The important part is that the workflow made the failure explainable.

The final report could say:

  • Ruff/Bandit/backend tests produced their respective signals.
  • The AI-generated targeted tests found blocking evidence.
  • The PR is not ready for merge.
  • The manual reviewer should focus on the authorization and mapping behavior.

That is much more useful than a generic failed check.

The AI report explains why the PR is blocked and where the reviewer should focus.

There is an important status distinction here. The AI final reviewer report job can be green even when the PR is blocked. Green means the report was generated and published successfully. It does not mean the report recommends merging. The separate CI readiness gate represents whether the underlying evidence passed.

The good-path PR

I also created a clean PR:

This PR added a small, legitimate backend behavior change:

  • normalize medical registration corrections;
  • add a focused unit test for that normalization.

The result was the good path:

  • Static analysis passed.
  • Security analysis passed.
  • Existing backend tests passed.
  • AI-generated targeted tests passed.
  • AI final reviewer report passed.

The clean path: the five-job base workflow completed successfully.

This clean run predates the separate readiness-gate job added on PR #8, so I am not claiming that PR #11 executed that later check. It demonstrates the clean evidence path and a “ready for manual review” AI report; PR #8 demonstrates the explicit gate behavior.

The final AI report did not merely say “all green.” It summarized the evidence and still told the human reviewer where to focus:

Even on the clean path, the report gives the human reviewer a concrete focus area.

This is the workflow I want from AI in CI: not blind approval, not replacement of the reviewer, but a compact evidence report.

Constrain the AI job

There is one security point that should not be treated as an implementation detail.

If an AI agent runs inside CI, it should run with limited permissions.

In this experiment:

  • deterministic jobs used read-only repository access;
  • the AI test-generation job could write only inside the runner workspace;
  • generated tests were restricted to one expected path;
  • the final report job had permission to write a PR comment, but not to merge;
  • expensive AI jobs were label-gated;
  • the workflow used sandboxing and privilege reduction options exposed by openai/codex-action.

The enforcement happens in the workflow file itself: .github/workflows/ci-evidence-report.yml.

The practical controls are:

  • GitHub Actions permissions blocks, such as contents: read, so jobs do not receive unnecessary repository permissions.
  • pull-requests: write only on the final-report job, because that job needs to post a PR comment.
  • sandbox: workspace-write for Codex so it can write generated tests only inside the checked-out workspace.
  • safety-strategy: drop-sudo so Codex does not keep elevated runner privileges.
  • codex-args: '["--ephemeral"]' so the Codex run does not persist unnecessary state.
  • a scope-verification step that checks git status and fails if Codex modifies anything outside the expected generated-test file.
  • label gating with ci-ai-report, so expensive AI jobs run only when intentionally requested.

That is the right default posture.

An AI job in CI should not inherit broad write permissions just because it is convenient. Treat it like any other automation: give it the minimum access required for the specific job.

What does this architecture cost?

There are four different costs in this design:

  1. The deterministic tools.
  2. GitHub Actions runner time and artifact storage.
  3. AI model usage.
  4. Engineering time to build and maintain the orchestration.

Ruff, Bandit, and pytest are open source, so this experiment did not add license fees for those tools. An enterprise could substitute commercial scanners or test platforms, which would add their own licensing costs without changing the architecture.

The workflow has two AI calls when the ci-ai-report label is present:

  • one to generate targeted tests;
  • one to create the final reviewer report.

For a concrete planning example, assume those two calls together use 150,000 input tokens and 10,000 output tokens for one labeled PR run. That is an illustrative workload, not a measurement of the experiment’s actual bill. The workflow did not pin a model, and real usage depends on diff size, repository context, generated tests, evidence volume, caching, and retries.

Using current public API prices, that workload would cost approximately:

Reference modelInput price per 1M tokensOutput price per 1M tokensIllustrative cost per labeled PR run
GPT-5.4 mini$0.75$4.50About $0.16
GPT-5.4$2.50$15.00About $0.53

The corresponding uncached monthly model-cost estimates are:

Labeled AI-assisted CI runs per monthGPT-5.4 mini referenceGPT-5.4 reference
100About $16About $53
250About $39About $131
500About $79About $263

GitHub runner time is additional. GitHub currently lists a standard two-core Linux runner at $0.006 per minute beyond any included allowance. If the complete workflow consumes an illustrative 20 billable runner-minutes across its jobs, that is about $0.12 per PR run, or $12 for 100 runs. Public-repository rules, included minutes, job-level rounding, artifact storage, and the actual duration of each job can change that number.

If the Part 1 AI code review is also executed as another model-backed job inside the same orchestration, its token and runner usage must be added. Dynamic analysis would similarly add application-startup time, scanner time, and possibly commercial tool licensing.

The direct usage cost can therefore be modest. The larger cost in the repository-owned approach is engineering ownership: prompts, schemas, permissions, artifacts, inline publication, deduplication, retries, gates, and ongoing maintenance.

Could CodeRabbit replace this GitHub Actions workflow?

CodeRabbit can cover parts of this workflow. Its Pro plan includes AI pull-request review plus linter and SAST-tool support. Pro+ adds capabilities around the review process, including unit-test generation and other finishing actions.

But I would not describe CodeRabbit and this GitHub Actions pipeline as exact substitutes.

The Blog 2 experiment used GitHub Actions for the complete evidence flow. I did not run an equivalent end-to-end CodeRabbit experiment combining the same Ruff, Bandit, pytest, temporary-test, artifact, readiness-gate, and final-report stages. The comparison here is therefore architectural and economic, not a claim that both implementations were tested identically.

AreaRepository-owned GitHub ActionsCodeRabbit
OrchestrationFully controlled in the repositoryProduct-managed with configuration controls
Tool selectionAny tool the runner can executeSupported product integrations and features
Model choiceTeam chooses the model; changing provider means replacing the AI step or adapterCodeRabbit controls model routing
Evidence formatTeam owns artifacts, schemas, gates, and report formatProduct-defined workflow and reporting
Custom test executionFully programmableProduct capability, with unit-test generation on Pro+
Dynamic analysisCan add any start-and-scan jobDepends on supported integrations and product workflow
Operational effortHighestLower because the product owns the review machinery
Pricing shapeRunner usage + model usage + engineering timePrimarily per-developer subscription, with plan limits and add-ons

Current annual-billing prices list CodeRabbit Pro at $24 per developer per month and Pro+ at $48 per developer per month. For a five-developer team, that is $120 or $240 per month before any applicable usage add-ons.

For comparison, 100 labeled GitHub Actions runs under the assumptions above would be roughly $16 to $53 in model usage, plus an illustrative $12 in billable Linux runner time. That makes the repository-owned path look cheaper in direct usage—but it excludes the engineering time required to build and operate everything CodeRabbit packages as a product.

My preference for this broader CI architecture is GitHub Actions because it provides the maximum flexibility. The team can select the deterministic tools, choose the AI model, define permissions, decide what blocks the PR, add dynamic analysis later, and control exactly what the final reviewer sees.

That does not mean CodeRabbit has no place. A practical hybrid is to use GitHub Actions as the CI evidence backbone and feed CodeRabbit—or another dedicated reviewer’s findings—into the final evidence report as the code-review signal from Part 1.

A useful edge case: no generated tests

One of the most useful lessons came from a failure in the workflow itself.

On a formatting-only PR, Codex correctly decided that no targeted tests were needed. It created a placeholder pytest file explaining that no targeted backend test was generated.

That was logically correct.

But pytest collected zero tests and exited with code 5. According to pytest’s documented exit codes, exit code 5 means no tests were collected. GitHub Actions treated that as a failed job.

So the AI decision was right, but the CI wrapper was wrong.

The workflow now treats this specific case as non-blocking:

if [ "$pytest_status" -eq 5 ] && \
   grep -q "collected 0 items" ci-results/ai-targeted-tests/pytest-output.txt; then
  pytest_status=0
fi

That keeps the semantics clean:

  • generated tests fail: block the PR;
  • no generated tests needed: pass;
  • scope violations: block the PR;
  • real pytest failures: block the PR.

This is a small implementation detail, but it is an important design lesson: AI-generated CI steps need explicit no-op semantics. Otherwise a correct AI decision can become a false CI failure.

What about dynamic analysis?

Dynamic analysis is still important. Tools like OWASP ZAP can scan a running application and catch issues that static analysis cannot see.

But I did not include dynamic analysis in this experiment.

That does not change the architecture.

Dynamic analysis would simply become another deterministic evidence producer:

Start app in CI
Run dynamic scan
Upload scan output
AI final report reads scan output
Human reviewer gets the combined summary

The reason I skipped it here is practical scope. Dynamic analysis introduces extra operational concerns:

  • starting the application in CI;
  • configuring test-safe environment variables;
  • database or dependency setup;
  • authentication/session handling;
  • seed data;
  • scan tuning to avoid noisy findings;
  • deciding whether the scan is advisory or merge-blocking.

Those are real topics, but they do not change the core AI-assisted CI workflow. They are a next extension, not a prerequisite for proving the pattern.

For this article, Ruff + Bandit + pytest + AI-generated targeted tests were enough to validate the architecture.

What this means for AI-assisted CI

The experiment made the boundary clearer for me.

AI is useful in CI when it does things deterministic tools cannot do well:

  • read multiple tool outputs together;
  • connect failures back to PR intent;
  • explain whether a finding is likely introduced by the PR;
  • generate targeted tests for changed behavior;
  • tell the manual reviewer where to focus;
  • summarize uncertainty instead of hiding it.

AI is weaker when it tries to replace purpose-built tools:

  • linting;
  • formatting;
  • dependency scanning;
  • security pattern matching;
  • test execution;
  • coverage calculation;
  • merge gating.

The scalable approach is not “AI does CI.”

The scalable approach is:

CI tools produce structured evidence.
AI reads the evidence.
The reviewer gets a decision-ready summary.

Practical recommendations

If I were applying this pattern across projects, I would start with these rules.

1. Keep deterministic checks deterministic

Use existing tools for static analysis, security analysis, and test execution. Do not ask the model to decide whether formatting passed or whether a test passed.

2. Store outputs as artifacts

The AI report should read artifacts, not screenshots or vague logs. GitHub Actions artifacts make this straightforward.

3. Separate AI generation from execution

Let AI generate targeted tests, but run them with the normal test framework.

4. Label-gate expensive AI jobs

In the experiment, AI-generated tests and the final AI report were gated behind a label:

ci-ai-report

That gives control over cost and avoids running expensive AI jobs on every trivial PR.

5. Create a clear no-op path

If no targeted tests are needed, that should be a successful outcome, not a failed job.

6. Keep the final human reviewer in control

The AI report should recommend readiness. It should not silently merge code.

7. Pin and measure the production model

Choose the model deliberately, record actual token and runner usage, and set a budget per AI stage. Planning estimates are useful, but production cost controls should be based on measured runs from the team’s real PR distribution.

Final takeaway

The best role for AI in CI is not replacing the pipeline.

It is making the pipeline readable.

Static analysis, security scanners, and tests already know how to produce signals. The problem is that those signals are often scattered across logs, artifacts, comments, and check statuses.

AI can turn those signals into a coherent review brief:

  • what passed;
  • what failed;
  • what matters;
  • what is probably introduced by the PR;
  • what the human reviewer should inspect next.

That is a practical use of AI in CI.

Not magic. Not autonomous merging. Just better evidence flow.

Experiment references

Implementation files

Tool references

AI Code Review in CI: Codex Cloud vs GitHub Actions vs CodeRabbit

AI, Programming, , , , , , , ,

I have been thinking about how AI should fit into CI/CD. Not the part where a developer uses an AI coding agent before opening a pull request. That is important, but it is still developer workflow.

The part I wanted to test was what happens after the pull request is opened.

At that point, a normal CI pipeline already has several responsibilities:

  • Run tests.
  • Run static analysis.
  • Run security checks.
  • Possibly run dynamic or integration checks.
  • Give reviewers useful evidence before a human approves the change.

AI can fit into this, but it should not replace every deterministic tool. Linters, type checkers, test frameworks, dependency scanners, and security scanners are still valuable because they are repeatable. The interesting question is where an AI reviewer adds judgment: finding logical defects, interpreting tool output, spotting missing tests, and giving a reviewer a concise explanation of risk.

This first experiment focuses only on AI code review for pull requests. I compared three approaches:

  1. Hosted Codex Cloud code review
  2. A repository-owned GitHub Action using Codex
  3. CodeRabbit

The broader CI question — static analysis, dynamic analysis, security tooling, and testing with AI interpreting the results — is covered in Part 2: AI in CI Should Not Replace Your Tools. It Should Read the Evidence.

The Question I Wanted to Answer

The starting point was simple: if an AI tool can write code, can a different AI tool review it?

My instinct was that the reviewer should be independent from the coding agent. If the same model, prompt, and context wrote the change and then reviewed it, the review is less convincing. It may repeat the same assumptions. Independence matters.

But independence does not automatically require a dedicated code review product. A coding agent can also act as a reviewer if it is run with a different prompt, different constraints, and a review-only role.

So the actual comparison is not just “model vs product.” It is:

  • How much review quality comes from the model?
  • How much comes from the prompt and repository instructions?
  • How much comes from operational product features such as GitHub integration, inline comments, deduplication, review status, dashboards, and configuration?
  • How much control does the repository owner retain?
  • How much effort is required to make the workflow reliable?

The Test Repository

I used a real repository for the experiment:

Repository: smakam/doctor-parser

The repository setup was introduced in PR #1. The repository-owned inline-review publisher was added in PR #3. I then used controlled experiment PRs for the actual comparison rather than reviewing the setup changes themselves.

The application has a backend and frontend, including code paths for uploaded doctor nameboard extraction and review/correction flows. That gave us enough realistic surface area to test authorization and data-mapping bugs.

To keep the comparison fair, I used two controlled defects.

Defect 1: Authorization Check Inversion

In backend/app/routers/nameboard.py, the access check was changed from denying access when neither ownership identifier matched to denying access when the session identifier did match.

-    if uploader != requesting_user_id and record.session_id != requesting_user_id:
+    if uploader != requesting_user_id and record.session_id == requesting_user_id:

The expected finding was a high-priority authorization issue: a requester whose ID differs from both the uploader and session ID could access or mutate another record, while a matching guest session could be incorrectly denied.

Defect 2: Incorrect Correction Mapping

In backend/app/services/review_service.py, the correction mapping for medical_registration_no was changed to write into qualifications.

-        "medical_registration_no": "medical_registration_no",
+        "medical_registration_no": "qualifications",

The expected finding was a data integrity issue: submitting a registration-number correction would overwrite qualifications and leave the actual registration number unchanged.

These are useful defects for this kind of experiment because they are not formatting issues. They require the reviewer to understand intent, data flow, and security impact.

Shared Review Guidance

One important design choice was to make the review guidance repository-owned.

The shared policy lives in .github/review/code-review.md.

The policy tells the reviewer to prioritize:

  • Authorization and access-control regressions
  • PII exposure
  • Security-sensitive behavior
  • Logic defects introduced by the pull request
  • Incorrect persistence or field mapping
  • Missing validation that changes behavior

It also sets a threshold: only report actionable defects introduced by the PR. Do not produce a generic style review.

This is an important lesson from the experiment. The model matters, but the review role matters just as much. If you want an AI reviewer, you need to define what “good review” means for your repository.

How the Three Review Paths Fit Together

All three approaches start with the same pull request and the same repository-owned review policy. What changes is who runs the review and who owns the operational layer around it.

The review policy is shared. The execution and publication layers differ.

Option 1: Hosted Codex Cloud Code Review

The first approach used hosted Codex Cloud code review.

In this mode, the repository uses AGENTS.md as the durable instruction entry point. For PR reviews, AGENTS.md routes the reviewer to the shared review policy.

There is no GitHub Actions workflow involved in this path. The hosted Codex integration reviews the pull request and posts inline GitHub review comments.

In our experiment, hosted Codex posted inline comments as chatgpt-codex-connector on PR #4.

Hosted Codex posted native inline findings on the changed code.

What Worked Well

The setup cost was low. Once the repository guidance was in place, the hosted reviewer could use that context and post inline comments directly on the PR.

It found the important seeded defects, including the authorization issue and the incorrect field mapping.

This is the simplest path if you want an AI reviewer without building a review pipeline.

What You Give Up

The tradeoff is control.

You do not own the workflow in the same way you own a GitHub Action. You have less control over:

  • Trigger rules
  • Output schema
  • Deduplication behavior
  • Review gating
  • Model/provider substitution
  • Combining the review with other CI signals
  • Custom publishing logic

For many teams, that may be fine. But if the long-term goal is a larger AI-assisted CI system, this approach may become only one review source rather than the central orchestration layer.

Option 2: Repository-Owned GitHub Action Using Codex

The second approach used a GitHub Action.

The GitHub Action does not use a separate definition of what good review means. It uses the same AGENTS.md and shared .github/review/code-review.md policy as hosted Codex. The Action prompt explicitly tells Codex to read AGENTS.md, which routes it to that common policy.

On top of that shared layer, the repository owns four Action-specific files:

This separation is deliberate. The shared code-review.md file defines review priorities, repository invariants, and the finding threshold. The Action-specific pr-review.md file defines how this particular automated run should execute and how it must structure its output. It does not duplicate the review policy.

The workflow is label-gated. A pull request is reviewed only when the codex-action-review label is present. It also runs again when a new commit is pushed to the PR branch, as long as the label remains present.

The action asks Codex to produce structured JSON with sections such as:

  • summary
  • findings
  • validation
  • coverage
  • uncertainty

That structure is not mainly for humans. It is an execution contract between the reviewer and the publishing script. The script needs predictable fields so it can decide which findings can become inline comments and which ones should fall back to a summary.

The GitHub Action then publishes review comments as github-actions[bot] on the same PR #4.

The repository-owned Action produced native inline comments after we built the publishing layer.

What We Had to Build

This approach required the most custom work.

To make the review useful, we had to build:

  • A review prompt with strict output expectations
  • A JSON schema for review results
  • Patch parsing to understand changed lines
  • Inline comment publishing through the GitHub API
  • Fallback summary behavior for findings that cannot be anchored to a changed line
  • Fingerprinting to avoid duplicate comments on repeated runs
  • A safer permission model where the trusted publisher code comes from the base branch

This is the biggest practical difference between a coding agent and a code review product. The model can identify findings, but the product workflow around those findings still has to exist somewhere.

The Fingerprint Problem

Fingerprinting is specific to the repository-owned Action path.

If a workflow runs every time a commit is pushed, the reviewer may find the same issue more than once. Without deduplication, the PR gets noisy. The same comment can be posted repeatedly.

The fingerprint solves that by creating a stable identity for a finding, usually based on fields such as:

  • File path
  • Line number
  • Rule or issue title
  • Finding body or normalized explanation

Before posting a new comment, the publisher checks whether an equivalent finding has already been posted. If yes, it avoids posting a duplicate.

Dedicated review products usually handle this operational detail internally. In the GitHub Action approach, the repository owner has to implement it.

What Worked Well

The GitHub Action found both controlled defects and posted inline comments.

It also gave the most control:

  • The model can be swapped.
  • The prompt can be changed.
  • The trigger policy can be changed.
  • The output schema is owned by the repository.
  • The review can later be combined with static-analysis, security, or test output.

This is the strongest architecture if the goal is a scalable, model-agnostic AI review pipeline.

What You Give Up

You own the operational burden.

Inline comments, deduplication, schema validation, retries, permissions, security hardening, and future maintenance are all your responsibility.

If you only want PR review, this may be too much. If you want an AI-assisted CI orchestration layer, this is probably the most extensible foundation.

Option 3: CodeRabbit

The third approach used CodeRabbit, a dedicated AI code review product.

For this experiment, CodeRabbit was configured with .coderabbit.yaml.

The configuration used:

  • The assertive review profile
  • Code guidelines pointing to .github/review/code-review.md
  • Path instructions for backend routers and services

The CodeRabbit configuration was first merged independently in PR #5. The clean CodeRabbit test was PR #6. It was created as a non-draft PR, with the configuration already present on main, and without the competing Codex Action label.

CodeRabbit posted inline comments as coderabbitai[bot].

CodeRabbit produced native inline comments in the clean, non-draft experiment.

What Worked Well

Under clean conditions, CodeRabbit found both seeded defects:

  • The authorization bypass in backend/app/routers/nameboard.py
  • The incorrect registration-number mapping in backend/app/services/review_service.py

The product experience is clearly stronger than a bare GitHub Action. CodeRabbit provides more than model output:

  • Inline comments
  • PR walkthroughs and summaries
  • Review status
  • Pre-merge checks
  • Configuration through YAML
  • Code guidelines
  • Path instructions
  • Knowledge-base behavior
  • Learnings
  • Autofix and docstring generation depending on plan
  • Unit test generation on higher plans
  • Analytics and reports
  • IDE and CLI review options
  • Integrations such as Jira and Linear
  • Linter and SAST tool support

That is the real value of a dedicated code review product. It is not just that it has a model. It packages the operational workflow around the model.

The Inconsistent Run

Before the clean PR #6 test, we had the noisier PR #4.

That PR was draft at one point, had hosted Codex comments, GitHub Action comments, manual CodeRabbit review commands, and CodeRabbit configuration changes inside the same PR.

In that noisy run, CodeRabbit selected the relevant files but did not publish the two expected actionable inline comments. Later, a CodeRabbit chat response acknowledged that the two issues should have been flagged.

I would not treat that as proof that draft PRs are unsupported or that CodeRabbit cannot find these defects. The cleaner conclusion is more limited:

CodeRabbit’s behavior was inconsistent in a noisy mixed-bot/draft/config-changing experiment, but it worked correctly in a clean non-draft PR with configuration already present on main.

That distinction matters. The blog should not overclaim from a messy experiment.

Feature Comparison

AreaHosted Codex CloudGitHub Action + CodexCodeRabbit
Setup effortLowHighMedium
Inline commentsYesYes, after custom publisherYes
Comment authorchatgpt-codex-connectorgithub-actions[bot]coderabbitai[bot]
Uses shared repo guidanceYes, via AGENTS.md routingYes, via prompt and shared policyYes, via code guidelines/path instructions
Workflow ownershipMostly product-ownedRepository-ownedProduct-owned with YAML configuration
Model flexibilityLimited from repo owner perspectiveStrongestProduct-controlled
DeduplicationProduct-managedMust buildProduct-managed
Trigger controlProduct settings / manual invocationFull GitHub Actions controlProduct settings / comments / YAML
Best use caseLow-friction AI PR reviewScalable AI-assisted CI orchestrationProductized AI code review
Main weaknessLess pipeline controlOperational burdenLess transparency into internal model decisions
Controlled-defect resultFound the seeded defectsFound the seeded defectsFound the seeded defects in the clean PR; inconsistent in the noisy mixed-bot PR

This was a controlled comparison, not a complete market survey. GitHub Copilot Code Review, Cursor Bugbot, Greptile, Qodo, and other review products may be worth evaluating, but I did not include tools that I had not tested on the same defects.

Cost Comparison

The pricing models differ, so the right comparison is not just “which one is cheaper per review.” The better comparison is how predictable the cost is and what operational work is included.

Hosted Codex Cloud

Hosted Codex review cost depends on the user’s ChatGPT/Codex plan and Codex credit model. It is not exposed as a simple repository-level per-PR API bill. OpenAI’s current Codex pricing documentation includes cloud-based automatic code review in supported ChatGPT plans, while API-key usage is separately token-billed and does not include the hosted GitHub code-review feature.

For an individual or small team already using Codex, this can be the lowest-friction option because it fits into the existing product experience. For a formal engineering cost model, it is harder to forecast per PR than an API-backed GitHub Action.

GitHub Action + Codex

The GitHub Action path has two costs:

  1. GitHub Actions runner minutes
  2. Model API usage

Runner cost is usually not the dominant factor for a small review job. The model cost is the meaningful part.

For a concrete API-cost reference, the table below uses the public GPT-5-Codex rate of $1.25 per 1M input tokens and $10.00 per 1M output tokens, with cached input at $0.125 per 1M tokens. The model page now marks GPT-5-Codex as deprecated, so these numbers should be read as a reproducible reference calculation, not as a recommendation to start a new production workflow on that model.

The experiment workflow did not pin a model, and the estimates below are not a measurement of its actual bill. A production workflow should explicitly choose an available model and recalculate with that model’s current prices.

The cost per review depends on:

  • Size of the diff
  • Amount of repository context included
  • Prompt length
  • Whether cached input applies
  • Number and length of findings
  • Whether the workflow reruns on every push

For planning, here is a rough uncached model-cost estimate:

Review shapeInput tokensOutput tokensApprox model cost
Small PR50,0005,000About $0.11
Medium PR150,00010,000About $0.29
Large PR400,00025,000About $0.75

These calculations assume uncached input and use the GPT-5-Codex API token prices above. Actual cost can be lower with prompt caching and higher if the workflow loads excessive repository context or reruns aggressively.

At a monthly level, the model-cost side looks like this:

Monthly PR review runsSmall PR estimateMedium PR estimateLarge PR estimate
100 reviews/monthAbout $11About $29About $75
250 reviews/monthAbout $28About $72About $188
500 reviews/monthAbout $56About $144About $375

GitHub Actions runner cost is additional, but for a lightweight Linux review workflow it is usually smaller than the model cost unless the workflow is slow or run at very high volume.

This approach can be very cost-efficient if reviews are scoped carefully. But if the workflow loads too much context or reruns aggressively on every commit, cost and PR noise can grow quickly.

CodeRabbit

CodeRabbit is primarily seat-based, not line-of-code based.

Public pricing lists:

  • Pro: $24/mo/user billed annually
  • Pro Plus: $48/mo/user billed annually

The docs describe per-developer rolling review limits:

  • Pro: 5 PR reviews per developer per hour
  • Pro+: 10 PR reviews per developer per hour

The current docs list a limit of 300 files per review for both Pro and Pro+.

Each review run consumes review allowance, including automatic incremental reviews after pushes and manual review commands such as @coderabbitai review or @coderabbitai full review.

This means CodeRabbit is not priced like “X cents per line reviewed.” You are paying for a productized review workflow, plus plan limits and optional usage-based expansion.

For a five-developer team, the annual-billing subscription comparison is:

OptionMonthly platform/model cost for 5 developersWhat this does not include
CodeRabbit Pro$120/monthUsage add-ons, if needed
CodeRabbit Pro Plus$240/monthUsage add-ons, if needed
GitHub Action + Codex, 100 medium reviews/monthAbout $29/month model costBuild/maintenance time, GitHub runner cost
GitHub Action + Codex, 250 medium reviews/monthAbout $72/month model costBuild/maintenance time, GitHub runner cost
GitHub Action + Codex, 500 medium reviews/monthAbout $144/month model costBuild/maintenance time, GitHub runner cost

So the GitHub Action path can be materially cheaper in direct model spend. But that is not the full cost. The engineering time to build and maintain inline publishing, deduplication, schema validation, security hardening, retries, and future integrations is real.

CodeRabbit costs more than the raw model calls, but it includes the product workflow. That is the main economic tradeoff.

Configuration Lesson: Keep One Review Policy

The best scalable pattern is not to duplicate long instructions across every tool.

I would organize the repository around four linked layers:

The responsibilities should be different:

  • AGENTS.md: tells coding agents and hosted Codex where the project guidance and review policy live.
  • The shared review policy contains the durable, provider-neutral priorities and finding threshold.
  • The Action prompt contains the GitHub Action execution contract, including JSON output requirements.
  • The CodeRabbit YAML contains CodeRabbit-specific configuration, review profile, code-guideline references, and short path instructions.

The important point is that the CodeRabbit YAML should not become a second full review policy. But I also would not remove path instructions entirely. In the successful CodeRabbit run, its inline comments cited path instructions as the source. So the better approach is to keep path instructions short and targeted while keeping the broader policy in the shared review-policy file.

My Recommendation

If I were adopting this in a real engineering workflow, I would not choose only one answer for all teams.

For a small team that wants quick value, I would start with hosted Codex Cloud review or CodeRabbit.

For a team that wants a polished productized PR review experience, I would evaluate CodeRabbit seriously. Its value is not only the model. Its value is the review workflow around the model.

For a platform team building a broader AI-assisted CI system, I would invest in the GitHub Action approach. It is more work, but it becomes the orchestration point. The same pipeline can later combine:

  • AI code review
  • Static-analysis output
  • Security scan output
  • Test failures
  • Coverage changes
  • Dependency risk
  • Runtime or dynamic-analysis signals

Then AI can do what it is better at: interpret multiple signals, identify risk patterns, explain impact, and help the human reviewer make a better decision.

What I Would Not Do

I would not ask a generic coding agent to “review this PR” with no policy, no structured output, and no integration plan.

That may produce useful comments sometimes, but it is not a CI/CD architecture.

For AI review to work inside CI, it needs:

  • A clear review role
  • Repository-specific guidance
  • A defect threshold
  • Inline comments or structured output
  • Deduplication
  • A trigger model
  • A human review handoff
  • Cost controls
  • A way to evolve the policy over time

The model is only one part of the system.

Conclusion

This experiment changed how I think about AI code review.

A dedicated code review product is not necessarily better because it has a different model. A repository-owned coding agent workflow is not automatically better because it is more flexible.

The real tradeoff is operational.

Hosted Codex Cloud gives a low-friction review path. A GitHub Action gives maximum control and model flexibility, but requires custom engineering. CodeRabbit gives a productized review workflow, but the internal decision-making is less transparent and the buyer does not choose the exact model stack.

For now, my preferred direction for a scalable CI architecture is:

  1. Keep review policy in the repository.
  2. Use productized review tools where they reduce operational burden.
  3. Use a repository-owned action when orchestration, model flexibility, and CI integration matter.
  4. Treat AI as a reviewer and interpreter, not as a replacement for deterministic CI tools.

That fourth point is the bridge to the next article: how deterministic static analysis, security checks, and tests can produce evidence, and how AI can turn that evidence into a concise report for the final human reviewer. In that larger design, AI code review becomes one input to the decision rather than the decision itself.

References

Constrain, Adapt, Evaluate: Rethinking Coding Assessments for the AI Era

AI, , , , ,

AI is becoming part of everyday software engineering. Developers use it to understand unfamiliar code, explore alternatives, generate tests, debug failures, and accelerate implementation. A technical assessment that asks candidates to pretend these tools do not exist is becoming less representative of the work we actually expect engineers to do.

But the opposite approach creates a different problem. If a candidate receives an unrestricted coding agent that can inspect the entire task, identify every defect, rewrite complete files, and prepare the final explanation, what exactly is the assessment measuring?

This tension led us to build SignalLoop, an AI-native candidate evaluator for software engineering hiring. SignalLoop is a collaboration between me, Sreenivas Makam, and my project partner, Ritesh Dhoot. The ideas, product direction, and implementation grew through our work together.

SignalLoop started as a proof of concept. It has grown into a runnable reference implementation with a hosted pilot, but it is not a production-grade hiring system. We are publishing it to demonstrate and test the assessment model, make the design discussable, and invite others to help improve it—not to suggest that it is ready to be adopted unchanged for consequential hiring decisions.

SignalLoop includes the things any usable assessment platform needs: employer and candidate experiences, administration, invitations, test execution, dashboards, scoring, and evidence reports. Those components matter, but they are not the central idea.

The differentiated part of SignalLoop rests on three design choices:

  1. Constrain AI collaboration without pretending AI does not exist.
  2. Adapt the assessment to the company and role while preserving comparability across candidates.
  3. Evaluate how the candidate worked, not only the final output.

We think of these as Constrain, Adapt, and Evaluate.

SignalLoop is also a work in progress. Some parts of this model are implemented and running today. Others are foundations or future directions that we are actively exploring. This article describes both, because the open questions are as important as the software already built.

The architecture behind the three ideas

The portals and dashboards are delivery surfaces around a deeper assessment loop:

Company context + role + JD
              ↓
Comparable role-level assessment blueprint
              ↓
Candidate works with a constrained AI collaborator
              ↓
Code, tests, prompts, snapshots and decisions are captured
              ↓
Technical evaluation + FAVO interpretation
              ↓
Evidence-based employer report

Each layer is designed to preserve a different kind of signal: relevance, candidate ownership, and evidence about the engineering process.

1. Constrain: allow AI collaboration without allowing AI delegation

There are two easy positions to take on AI in coding assessments.

The first is to prohibit it. That preserves the familiar assessment format, but it increasingly creates an artificial environment. A candidate may be evaluated under conditions that no longer resemble how the employer expects engineers to work.

The second is to allow an unrestricted assistant. That is realistic in one sense, but it can erase the distinction between a candidate who directs and verifies the work and one who delegates the entire task.

SignalLoop takes a position between these extremes. The candidate receives an AI collaborator inside the assessment workspace, but that collaborator is designed to be a coach, not an autopilot.

The assistant can:

  • explain candidate-visible code and public test output,
  • clarify concepts,
  • suggest a general debugging approach,
  • help reason through one candidate-identified issue,
  • compare tradeoffs the candidate has already framed,
  • provide bounded implementation help as the candidate demonstrates understanding.

It cannot:

  • enumerate every defect in the assessment,
  • produce the complete solution,
  • rewrite whole files,
  • reconstruct hidden tests,
  • reveal scoring internals or evaluator material,
  • make the candidate’s design decisions,
  • write the final explanation on the candidate’s behalf.

This boundary is enforced in multiple layers. A deterministic pre-gate catches obvious protected requests. A policy-classification component evaluates the intent of the interaction. A separate response-generation component produces help within the allowed boundary. Progressive disclosure lets the assistant become more specific when the candidate has identified the problem and articulated an approach.

SignalLoop also applies an anti-decomposition rule. A candidate should not be able to turn one disallowed request into a full solution by splitting it into a sequence of smaller prompts such as “list every issue,” followed by “give me the code for each issue,” followed by “write every missing test.” The system considers the combined effect of the interaction, not only each message in isolation.

There is another boundary that matters just as much: the AI receives only candidate-visible information. Hidden tests, seeded issue lists, reference solutions, evaluator notes, and scoring internals are never part of the assistant’s context.

This constrained collaborator is implemented today. It is not perfect, and policy evaluation will require continued testing and calibration, but it lets us ask a more useful question than “Did the candidate use AI?”

The better question is:

How did the candidate divide responsibility between themselves and the AI?

A prompt such as “Find every bug and fix the project” tells us something different from “The duplicate-email test is failing; I think normalization is missing. What should I inspect before I change the behavior?”

The interaction itself becomes evidence.

2. Adapt: tailor the assessment to the role, not arbitrarily to each candidate

Most engineering assessments face a relevance problem. A single generic coding exercise is easy to administer and compare, but it may be only loosely related to the role. A fully personalized assessment may be more relevant, but it can destroy comparability.

SignalLoop’s design principle is:

Adapt across companies and roles. Preserve consistency within a hiring cohort.

The intended role-level blueprint is derived from inputs that belong to the hiring decision:

Company context
+ hiring area
+ role and seniority
+ job description
+ cognitive areas to emphasize
+ target duration
→ role-level assessment blueprint

Company context matters because the same job title can imply very different engineering work.

A backend engineer at a fintech company may need stronger evidence around authorization, auditability, data boundaries, and failure handling. A backend engineer at an AI-infrastructure company may need more emphasis on control-plane APIs, observability, reliability, deployment tradeoffs, and safe AI-assisted debugging.

The JD may be similar, but the assessment emphasis should not necessarily be identical.

Once the employer reviews and approves a blueprint, however, every candidate for that role should receive the same scored assessment. This creates a stable evidence surface for comparison. An employer can create a new version of the role assessment, but SignalLoop should not silently generate an easier or harder scored task for each applicant.

Candidate resumes can still add value. They can inform:

  • skill gaps to probe during an interview,
  • claims that require validation,
  • candidate-specific follow-up questions,
  • report caveats and interviewer notes.

They should not initially determine:

  • which scored questions the candidate receives,
  • the scoring rubric,
  • the time allocation,
  • the required coverage for the role.

This distinction is important. Personalization is not automatically fairness. A candidate-specific test may look intelligent while making employer comparisons less defensible.

What exists today

SignalLoop currently implements guided role matching. An employer can provide the role, JD, seniority, team or domain context, expected AI usage, and an optional resume. The system maps the inputs to a versioned skill taxonomy, recommends the closest supported assessment pack, and explains:

  • directly tested skills,
  • partially tested skills,
  • unsupported skills,
  • the rationale for the match,
  • suggested interview follow-ups.

The current executable coverage is deliberately narrow: SignalLoop can recommend a Standard or Advanced FastAPI assessment. If a frontend or data-engineering JD is outside current executable coverage, the product says so instead of pretending that a backend score evaluated those skills.

The project therefore demonstrates the role-matching model today; it does not yet claim to be a complete dynamic question composer. The consolidated roadmap later in this article explains that next layer.

3. Evaluate: measure the engineering process, not only the output

Traditional coding assessments typically emphasize the final artifact:

  • Did the tests pass?
  • Is the code readable?
  • Was the task completed on time?
  • What was the final score?

Those signals remain important. AI-assisted engineering, however, introduces additional questions.

  • Did the candidate understand the problem before changing code?
  • Did they recognize ambiguity and risk?
  • Did they ask AI focused questions or delegate wholesale?
  • Did they verify AI-assisted changes?
  • Did they add meaningful tests?
  • Can they explain and defend the final result?

SignalLoop organizes this process evidence through FAVO: Frame, Ask, Verify, Own.

Frame

Did the candidate understand the problem, constraints, risks, and ambiguous requirements? Did their implementation reflect deliberate prioritization and product judgment?

Ask

How did the candidate use AI? Were the questions focused and grounded in an identified behavior, or did the candidate attempt to outsource the complete solution?

Verify

Did the candidate run tests, add tests, inspect edge cases, revisit assumptions, and validate changes after receiving AI assistance?

Own

Could the candidate explain what changed, why they made particular tradeoffs, what remains uncertain, and what they would improve next?

Candidates do not manually write a FAVO score. SignalLoop derives the interpretation from captured evidence such as:

  • code snapshots,
  • public and hidden test runs,
  • candidate-written tests,
  • AI conversations and policy redirects,
  • large code-paste signals,
  • final submission-review answers,
  • the consistency between the explanation and submitted code.

The Engineering Evidence Report combines this with the technical evaluation. FAVO is not intended to replace correctness, and it is not a personality or psychometric score.

The technical score tells the employer what worked. FAVO helps explain how the candidate got there and whether the process is trustworthy.

An initial deterministic version of this interpretation is implemented today. Some signals are necessarily simple proxies: prompt counts, test runs, candidate test files, feature behavior, hidden-test status, and submission-review completeness. Improving and validating those mappings is part of the consolidated roadmap below.

What exists today and what comes next

Rather than scatter the project status across the three ideas, here is the boundary in one place.

Constrain

Implemented today: A constrained assistant, candidate-visible context boundary, policy classification, progressive disclosure, anti-decomposition behavior, interaction logging, and evidence capture.

Next: Stronger adversarial testing, policy-evaluation datasets, progressive-disclosure calibration, and better measures for distinguishing productive coaching from disguised delegation.

Adapt

Implemented today: Guided role matching from role, JD, seniority, team or domain context, and optional resume information to the closest registered assessment pack. The result includes tested, partially tested, and unsupported skill coverage. The question-bank governance foundation also supports provenance, draft review, and approval.

Next: Compose role-level assessments from approved and calibrated questions:

Role/company/JD/cognitive requirements
              ↓
Required assessment slots
              ↓
Approved question bank
              ↓
Reviewable role-level blueprint
              ↓
Employer approval or same-slot swaps
              ↓
Reusable assessment for the candidate cohort

This requires connecting the question bank to employer blueprint creation, candidate delivery, and evidence-report scoring while preserving the same scored assessment for candidates in the same cohort.

Evaluate

Implemented today: Deterministic technical scoring, public and hidden test evidence, AI-interaction evidence, submission review, timeline capture, and an initial FAVO interpretation.

Next: Better evidence mappings, evaluator calibration, longitudinal validation, clearer evidence limits, and research into which signals are reliable, useful, fair, and resistant to superficial gaming.

Seeing the complete workflow

The short demo below shows the current end-to-end SignalLoop experience: super-admin visibility, employer assessment setup, the candidate workspace with constrained AI, and the resulting evidence report.

What SignalLoop deliberately does not claim

Being explicit about the boundaries is important.

SignalLoop does not treat AI use itself as misconduct. It evaluates how the candidate uses it.

It does not allow the embedded assistant to complete the assessment or access evaluator-only artifacts.

It does not currently give every candidate a different scored test. The role-level assessment is intended to remain comparable across the cohort.

It does not convert unsupported skills into implied evidence. If the current assessment cannot test an area, the report should identify the gap.

It does not reduce a hiring decision to one number. The report is evidence for employer review, not an autonomous hiring decision.

And it does not yet implement the entire adaptive question-composition vision described here.

From proof of concept to production-grade system

The distinction between a working reference implementation and a production system matters, especially in hiring.

SignalLoop began as a POC to explore whether constrained AI collaboration and process evidence could produce a better assessment signal. The current system demonstrates the complete workflow: employers can create invites, candidates can work in a browser with a constrained assistant, tests and interactions are captured, and evidence reports can be generated.

That proves the product loop. It does not complete the production journey.

A production-grade version would still need work in areas such as:

  • isolated and hardened execution infrastructure for untrusted candidate code,
  • private, calibrated assessment content rather than public demo packs,
  • production authentication, secrets management, monitoring, backups, and incident response,
  • explicit data-retention, consent, privacy, and compliance policies,
  • accessibility, abuse testing, rate-limit hardening, and operational support,
  • evaluator calibration and studies of reliability, validity, bias, and adverse impact,
  • question effectiveness analytics and ongoing assessment-version governance,
  • human review processes appropriate for consequential hiring decisions.

Some technical scaffolding and product foundations for these areas exist in the repository. Others remain design and validation work. We would not recommend using the current project as an autonomous hiring system, and we do not believe any evidence report should replace accountable human judgment.

How you can help

The project is a working reference implementation and a foundation for further exploration. The roadmap above is intentionally open. We are particularly interested in privacy, fairness, question effectiveness, assessment versioning, policy evaluation, and methods for evaluating AI-assisted engineering without rewarding superficial activity.

SignalLoop is available on GitHub. It is joint work by Sreenivas Makam and Ritesh Dhoot, and the next phase of the project is intentionally open to a wider community of contributors and critics.

If you are working on technical hiring, assessment design, AI safety, developer tools, or evidence-based evaluation, we would value your feedback. Contributions, critique, experiments, question-bank ideas, evaluator studies, and help with any of these future directions would be genuinely appreciated.

The central idea is simple:

Do not ask whether candidates used AI. Design the assessment so that their use of AI produces meaningful evidence.

Constrain the collaboration. Adapt the assessment. Evaluate the process.

Try It Yourself

Explore the hosted SignalLoop application to experience the assessment flow firsthand. SignalLoop is fully open source—you can inspect the implementation, run it locally, contribute, or adapt it to your own hiring process in the GitHub repository.

Prescribed Enterprise AI Architectures for Small, Mid-Size, and Large Organizations

AI, , , ,

The first article in this series, Enterprise AI Adoption: Requirements for Security, Governance, and Scale, defined the problem: enterprises consume AI through coding tools, standalone hosted applications, custom agents, and enterprise knowledge systems, but need consistent controls across identity, credentials, data, cost, providers, and operations.

The second article, A Reference Architecture for Governed Enterprise AI, translated those requirements into architectural building blocks and described the available implementation choices.

This article makes the opinionated decisions. It recommends what to implement, what to defer, and which representative products to consider for three enterprise profiles.

Product recommendations reflect the market as of July 2026 and should be revalidated before implementation.

Scope and Assumptions

Company size is the primary organizing variable because adding industry, geography, regulation, data sensitivity, cloud provider, and AI maturity as independent dimensions would make a simple prescription impossible.

The recommendations assume a software-enabled company with moderate data sensitivity, one primary public cloud, an existing corporate identity provider, both employee and custom AI usage, and a preference for managed services unless scale or control justifies ownership.

Size remains a proxy. Move to the next architecture tier when regulation, data sensitivity, AI expenditure, workload volume, geographic isolation, or operational complexity exceeds what is typical for the company profile.

ProfileWorking definitionOperating reality
Small5–50 engineersNo dedicated AI platform team; CTO or engineering lead owns AI
Mid-size50–500 engineersEmerging platform team; shared security and FinOps responsibilities
Large500+ engineersDedicated platform, security, data, and FinOps teams; multiple business units

Engineering scale is more useful than total employee count because it better predicts the number of custom workloads and the organization’s ability to operate platform components.

Small Enterprise: Prefer Simplicity

The small-enterprise architecture deliberately accepts coarser attribution and fewer centralized controls in exchange for low operating overhead.

Prescription

CapabilityRecommendation
Employee AIChatGPT Business
Coding assistantCodex under the managed company workspace
Human identityExisting Google Workspace or Microsoft Entra ID
Workload identityPrimary-cloud IAM and secrets services
Model inferenceExisting cloud’s managed AI platform
Model gatewayNone initially
Model selectionStatic aliases in shared application configuration
RoutingNo dedicated intelligent router
GuardrailsProvider-native controls plus application validation
Agent frameworkApplication-selected SDK; no enterprise standard
Agent runtimeExisting application infrastructure
KnowledgeApproved user connectors; no enterprise RAG platform
ObservabilityCloud telemetry and application traces
FinOpsProvider budgets, cloud billing, and monthly review
Self-hosted inferenceDo not implement

Hosted AI and coding

Use ChatGPT Business as the company-managed employee workspace and Codex as the standard coding assistant. Codex is included in ChatGPT Business, although usage limits and credit options depend on the plan. This reduces unmanaged subscriptions and keeps general AI and coding under one commercial and administrative relationship. OpenAI documents current Codex plan availability here.

Use SSO where available, central billing, approved data terms, and an explicit user-removal process. ChatGPT Business supports SSO but does not provide SCIM directory synchronization, so lifecycle management remains partly administrative. OpenAI documents the Business and Enterprise identity differences here.

Model access without a gateway

Applications call the primary cloud’s managed model platform directly using distinct workload identities. AWS-centric organizations use Amazon Bedrock, Google Cloud organizations use Vertex AI, and Microsoft-centric organizations use Azure AI Foundry. A company without meaningful cloud alignment may use a direct OpenAI or Anthropic API organization.

A shared client library may standardize model aliases, metadata, timeouts, retries, and trace identifiers. It is not a gateway and cannot guarantee centralized enforcement.

Attribution is therefore based on separate cloud projects or accounts, workload identities, billing tags, provider reports, and application logs. This is adequate while the workload portfolio is small, but it may not support reliable per-user or per-feature allocation.

Routing and guardrails

Use static model aliases such as general-fast, general-quality, and sensitive-approved. A centrally maintained configuration maps each alias to an approved cloud model deployment. Cloud IAM should restrict workloads to eligible deployments where supported.

Use provider-native guardrails and application controls. The application remains responsible for user authorization, tool permissions, structured output, agent limits, and approval of consequential actions.

Agents and knowledge

Do not standardize an agent framework yet. The first application can use an appropriate SDK and run within existing application infrastructure. Every agent still needs narrow tool permissions, least-privilege credentials, step and cost limits, and human approval for irreversible actions.

Do not build an enterprise knowledge platform. Employees can use administrator-approved connectors in the hosted workspace, subject to source permissions. Build retrieval only for a product workflow with a measurable requirement.

Graduation trigger

Introduce a managed gateway when the organization has several production AI applications, more than one provider, duplicated integrations, shared credential problems, or a need for centralized budgets, guardrails, routing, and detailed attribution.

Mid-Size Enterprise: Introduce a Shared Control Plane

The mid-size architecture requires all custom production model traffic to pass through a dedicated gateway. The platform remains primarily managed because the organization has an emerging platform team rather than a large AI infrastructure organization.

Prescription

CapabilityDefault recommendationAlternative
Employee AI and codingChatGPT Enterprise standard seats, including CodexCodex-only seats for restricted populations
Human identityEntra ID or Okta with SSO and SCIMGoogle Workspace
Workload identityCloud IAM plus gateway virtual credentialsVault where already established
Model gatewayManaged PortkeySelf-hosted LiteLLM
InferencePrimary cloud model platform plus one direct providerTwo approved cloud platforms
RoutingGateway aliases, conditional rules, and fallbackLiteLLM routing
GuardrailsProvider-native, Portkey, and application controlsCheck Point Lakera for higher-risk workloads
Agent frameworkOpenAI Agents SDKLangGraph for durable workflows
Agent runtimeExisting container or serverless platformDedicated runtime when volume justifies it
KnowledgeCompany knowledge in ChatGPT plus application RAGEnterprise search when justified
ObservabilityPortkey plus Langfuse CloudLiteLLM plus self-hosted Langfuse
FinOpsGateway budgets plus cloud billingFinout or CloudZero
Self-hosted inferenceGenerally deferWorkload-specific exception

Enterprise workspace

Use standard ChatGPT Enterprise seats for employees and engineers. A standard seat includes ChatGPT and Codex and offers the greatest flexibility. Codex-only seats should be reserved for people who need coding access but should not receive the broader ChatGPT workspace. OpenAI’s current Enterprise documentation describes both seat types and their access boundaries.

Use enforced SSO, SCIM, groups, and role-based access. Separate groups should cover general AI users, Codex users, AI developers, production operators, platform administrators, and sensitive-data users.

Gateway and model portfolio

Use managed Portkey as the default gateway. It provides multi-provider access, virtual credentials, routing, budgets, rate limits, retries, circuit breaking, observability, and integrated guardrails without requiring the emerging platform team to operate the full gateway stack. Portkey documents these gateway capabilities and deployment options.

Use the primary cloud model platform as the main backend and one approved direct provider as a secondary path. All production applications authenticate to the gateway; provider credentials remain with the platform team.

Use logical model aliases, policy restrictions, availability fallbacks, and controlled model-version rollout. Do not add a dedicated intelligent router until expenditure is material and evaluation data can prove that dynamic routing preserves quality.

Guardrails, agents, and tools

Apply provider-native controls, gateway input/output guardrails, and application-specific rules. Portkey guardrails support checks and actions on gateway requests and responses. Add Check Point Lakera only for public-facing or higher-risk workloads that justify specialized prompt-injection and data-leakage protection.

Adopt the OpenAI Agents SDK as the default for straightforward tool-using agents. Allow LangGraph for durable, stateful, resumable, or human-in-the-loop workflows. Run agents on the existing container or serverless platform and provide shared standards for identity, tool registration, MCP servers, traces, execution limits, approvals, and action logs.

Knowledge, observability, and FinOps

Use company knowledge in ChatGPT Enterprise with administrator-approved apps for employee search. Build a shared managed-cloud retrieval foundation only when multiple production applications need common ingestion, permissions, citations, and evaluation.

Use Portkey for gateway-level telemetry and Langfuse Cloud for multi-step traces and evaluations where deeper analysis is required. Use gateway budgets for enforcement and the existing cloud-finance process for reporting. Add Finout or CloudZero when model and supporting cloud costs span providers and require formal allocation or forecasting.

Graduation trigger

Move to the large-enterprise pattern when the company needs regional isolation, multiple business-unit control, formal chargeback, private deployment, dedicated AI security, a shared agent runtime, or self-hosted inference.

Large Enterprise: Central Policy, Federated Delivery

The large-enterprise architecture supports multiple providers, regions, business units, and control teams. Policy and visibility are centralized, while application ownership remains federated.

Prescription

CapabilityDefault recommendationAlternative
Employee AI and codingChatGPT Enterprise standard seats, including CodexCodex-only seats for restricted populations
IdentityEntra ID or Okta with SSO, SCIM, groups, and RBACExisting enterprise IdP
Workload identityCloud workload federation plus gateway virtual keysVault where standardized
Model gatewayRegional self-hosted LiteLLM EnterpriseManaged or self-hosted Portkey Enterprise
InferenceCloud AI platforms plus direct OpenAI and AnthropicAdditional approved providers
RoutingGateway policy routing plus evaluated intelligent routingApplication-directed routing
GuardrailsProvider controls, gateway policy, and Check Point LakeraNVIDIA NeMo Guardrails
Agent frameworkOpenAI Agents SDK plus LangGraph where justifiedApproved framework by exception
Agent runtimeShared Kubernetes-based runtimeManaged cloud agent runtime
KnowledgeCompany knowledge in ChatGPT plus shared product RAGGlean for dedicated enterprise search
ObservabilitySelf-hosted Langfuse Enterprise plus OpenTelemetryManaged enterprise platform
FinOpsCloudZeroFinout
Self-hosted inferencevLLM for selected stable workloadsNVIDIA NIM

Regional gateway and inference platform

Deploy LiteLLM Enterprise in approved regions as the standard model-access plane. Each deployment should provide high availability, business-unit isolation, local credentials, virtual keys, budgets, quotas, model aliases, audit records, guardrail integration, and approved fallback policies. LiteLLM documents virtual-key, authentication, SSO, and enterprise gateway capabilities in its product overview.

Applications should not receive direct provider credentials except through documented exceptions. The model portfolio can include strategic cloud AI platforms, direct OpenAI and Anthropic access, selected inference providers, and self-hosted open-weight models for justified workloads.

Expose internal capability aliases such as enterprise-fast, enterprise-reasoning, enterprise-code, regulated-region, and open-weight-private rather than embedding provider model names throughout applications.

Routing and guardrails

Apply policy routing to every request. Region, data classification, business unit, provider approval, and budget determine the eligible model set before performance or cost optimization.

Use intelligent routing only for workloads with evaluation datasets and measurable quality thresholds. RouteLLM or another router may select among eligible models, but the gateway remains the enforcement point.

Use four guardrail layers: provider-native safety, gateway policy, Check Point Lakera for high-risk workloads, and application business controls. Measure false positives, latency, and bypass resistance rather than treating guardrails as absolute security boundaries.

Agent platform

Provide an approved framework portfolio and a shared runtime. Use the OpenAI Agents SDK for the default tool-using pattern and LangGraph for durable workflows. The runtime should provide workload and delegated identity, queues, resumability, isolated code execution, an approved tool and MCP registry, secrets injection, limits, approvals, traces, cancellation, and kill switches.

The gateway governs model inference. The agent runtime governs execution and tools. Deterministic services should verify authorization and execute consequential actions proposed by a model.

Enterprise knowledge

Use company knowledge in ChatGPT Enterprise with approved apps for employee search and synthesis. This is a ChatGPT feature, not a separate product and not a backend for custom applications.

Operate a shared permission-aware RAG platform for product applications. It should provide connectors, ingestion, permission synchronization, deletion, retrieval APIs, citations, evaluation, and regional isolation.

Consider Glean when dedicated cross-enterprise search is a strategic requirement that company knowledge in ChatGPT does not satisfy. Do not deploy both without defining their distinct user populations and responsibilities.

Observability and FinOps

Use self-hosted Langfuse Enterprise for traces, evaluations, prompt management, datasets, RBAC, retention policies, and audit logs. These self-hosted enterprise controls are described in Langfuse’s deployment and pricing documentation. Export platform telemetry through OpenTelemetry and security events to the enterprise SIEM.

Use gateway budgets for real-time enforcement and CloudZero for allocation, anomaly detection, forecasting, unit economics, and showback or chargeback across inference and supporting infrastructure.

Self-host models through vLLM only for workloads with stable demand, demonstrated economics, suitable model quality, and an operations team able to own serving and upgrades.

How the Three Architectures Progress

CapabilitySmallMid-sizeLarge
Employee workspaceChatGPT BusinessChatGPT EnterpriseChatGPT Enterprise with enterprise policy
CodingCodexCodex in standard seatsCodex with role and spend controls
GatewayNoneManaged PortkeyRegional LiteLLM Enterprise
ProvidersOne primary platformPrimary plus one secondaryGoverned multi-provider portfolio
RoutingStatic configurationGateway rules and fallbackPolicy plus evaluated intelligent routing
GuardrailsProvider plus applicationProvider, gateway, applicationProvider, gateway, dedicated security, application
AgentsApplication-selected SDKStandard SDK and shared patternsFramework portfolio and shared runtime
KnowledgeUser connectorsCompany knowledge plus application RAGCompany knowledge plus shared product RAG
ObservabilityCloud and application logsPortkey plus Langfuse CloudSelf-hosted Langfuse Enterprise plus OTel
FinOpsBudgets and monthly reviewGateway budgets and optional FinOps platformCloudZero with showback or chargeback
Self-hosted inferenceNoUsually noSelected workloads only

When Size Is Not Enough

Use a stronger architecture tier when any of the following applies:

  • Regulated or highly sensitive data
  • Mandatory data residency or private networking
  • High AI expenditure or customer-facing AI cost of revenue
  • Many autonomous agents with consequential tools
  • Multiple cloud providers or geographic regions
  • Formal business-unit isolation or chargeback
  • Strict availability requirements
  • A large portfolio of production AI applications

A small healthcare or financial company may need large-enterprise security controls. A large company with limited AI use may begin with the mid-size platform. Size determines the default, not the exception policy.

Final Recommendation

The prescribed progression is intentionally conservative:

  • Small: one managed employee workspace, Codex, one model platform, no gateway, no enterprise RAG, and no dedicated FinOps platform.
  • Mid-size: ChatGPT Enterprise and Codex, a managed gateway, two inference paths, shared guardrails and tracing, and application-focused RAG.
  • Large: regional gateways, a governed provider portfolio, layered AI security, a shared agent runtime, enterprise retrieval, formal observability, and chargeback.

The architecture should become more sophisticated only when usage, risk, cost, or organizational scale creates a concrete reason. The objective is not to deploy every available AI platform component. It is to provide the minimum architecture that enables adoption without losing control.

A Reference Architecture for Governed Enterprise AI

AI, , , ,

In the first article in this series, Enterprise AI Adoption: Requirements for Security, Governance, and Scale, we identified four ways enterprises consume AI and seven requirements that apply across them. Those requirements cover identity, non-human credentials, cost governance, security and data controls, model flexibility, optimization, and observability.

This article translates those requirements into a reference architecture. It defines the architectural building blocks, their responsibilities, and the implementation choices available to an enterprise. The third article in the series will use these building blocks to prescribe specific architectures for small, mid-size, and large organizations.

One Architecture, Two Control Paths

Enterprise AI cannot be governed through a single product because not all AI traffic follows the same path.

Custom applications, agents, retrieval systems, and compatible coding tools can access models through an enterprise-controlled model gateway. The gateway can authenticate workloads, protect provider credentials, enforce policies, route requests, attribute cost, apply common guardrails, and record telemetry.

Standalone hosted applications such as ChatGPT, Claude, Gemini, and specialized cloud tools normally communicate directly with their providers. Their requests do not pass through the enterprise model gateway. They must instead be governed through corporate identity, provisioning, product administration, contracts, connector controls, SaaS security, data-loss prevention, and audit integrations.

This distinction is fundamental. A model gateway is an important control point, but it is not a universal control plane for every form of enterprise AI consumption.

Fig: Enterprise AI requires separate control paths for hosted applications and enterprise-controlled model traffic.

Architecture Principles

The reference architecture follows five principles.

Centralize policy without centralizing every decision

The enterprise needs common boundaries for identity, credentials, approved models, data, cost, and auditability. Application teams should remain free to select models and implementation patterns within those boundaries. The central platform should provide a governed path rather than become a manual approval queue.

Separate human, workload, and delegated identities

Employees authenticate through the corporate identity provider. Applications and agents use workload identities. When an agent acts for a person, the trace should preserve both the agent identity and the initiating user identity.

Keep provider credentials away from applications

Applications should authenticate to an enterprise-controlled gateway. The gateway protects and rotates provider credentials, maps workload identities to policies, and issues virtual credentials where needed.

Make data policy part of routing

Not every model or provider is approved for every data class. The eligible destination should be constrained by data sensitivity, contractual terms, region, retention policy, and workload risk before cost or performance optimization is considered.

Allow complexity to grow with need

The logical architecture can remain consistent while its implementation grows from cloud-native services to managed gateways, regional control planes, dedicated security tools, and self-hosted inference.

The Technology Landscape

The following table identifies representative implementation choices. These are examples, not recommendations for every organization.

Building blockImplementation choicesRepresentative examples
Human identityEnterprise identity providerMicrosoft Entra ID, Okta, Google Workspace
Workload identity and secretsCloud IAM, workload federation, secrets platformAWS IAM, Azure Managed Identities, Google Cloud IAM, HashiCorp Vault
Model gatewayCloud-native, managed, self-hostedAmazon Bedrock, Vertex AI, Azure AI Foundry, Portkey, LiteLLM, Kong, Cloudflare AI Gateway
RoutingGateway rules, intelligent router, application-directed routingGateway policies, RouteLLM, Not Diamond, Martian, OpenRouter
Guardrails and AI securityProvider-native, gateway-integrated, dedicated platform, open-source frameworkAmazon Bedrock Guardrails, Azure AI Content Safety, Google Model Armor, Portkey, Kong, Cloudflare, Check Point Lakera, NVIDIA NeMo Guardrails
First-party inferenceDirect model-provider APIOpenAI, Anthropic, Google, Cohere, Mistral
Cloud inferenceManaged cloud model platformAmazon Bedrock, Google Vertex AI, Azure AI Foundry
Hosted open-model inferenceManaged inference providerTogether AI, Fireworks AI, GroqCloud, Cerebras
Self-hosted inferenceEnterprise-operated model servingvLLM, NVIDIA NIM, Hugging Face TGI
Agent frameworkSDK, graph framework, managed agent serviceOpenAI Agents SDK, LangGraph, cloud-native agent services
AI observabilityGateway telemetry, specialized tracing, open telemetryLangfuse, LangSmith, Helicone, Arize Phoenix, OpenTelemetry
AI cost governance and FinOpsGateway budgets, attribution, forecasting, showback and chargebackLiteLLM, Portkey, CloudZero, Finout, Vantage
Enterprise knowledgeHosted knowledge feature, enterprise search, managed or custom RAGCompany knowledge in ChatGPT, Glean, cloud search services, vector and search platforms

1. Enterprise Identity and User Lifecycle

The corporate identity provider is the starting point for human access. Hosted AI products, coding applications, gateway consoles, observability platforms, and internal applications should use SSO wherever supported. Automated provisioning should create, update, suspend, and remove accounts as employees change roles or leave.

Groups and roles should represent approved populations such as general AI users, coding users, sensitive-data users, platform administrators, and model approvers. Identity establishes accountability and lifecycle control, but it does not determine what data a user may enter into an AI product. Data policy and product-specific administration remain necessary.

2. Governance for Standalone Hosted AI Applications

Standalone hosted applications sit outside the model-gateway path. The enterprise should maintain an approved product portfolio and evaluate each product for SSO, provisioning, administrative controls, retention, training terms, audit logs, regional processing, sharing, and connectors.

Connectors require particular care. When a hosted application connects to email, files, source code, or collaboration systems, it gains a new authorization path into enterprise information. Administrators must control which connections are available, which OAuth permissions are granted, and how access is revoked.

This is an architectural concern, but not a separate deployable product. It is implemented through identity, SaaS administration, procurement, CASB or DLP controls, connector governance, and organizational policy.

3. Workload Identity and Secrets

Every production workload should have an identity independent of the developer who created it. Production and non-production environments should not share identities or unrestricted credentials.

Where possible, use short-lived cloud workload credentials. When secrets are necessary, keep them in an enterprise secrets platform and rotate them through an established process. Applications should receive gateway credentials or virtual keys instead of underlying provider keys.

This allows a compromised application to be revoked independently, enables per-workload budgets and model permissions, and permits provider credentials to change without application redeployment.

4. Model Gateway

The model gateway is the shared access layer between enterprise-controlled workloads and model providers. It should authenticate the caller and evaluate attributes such as application, team, environment, data classification, use case, cost center, region, and requested capability.

The gateway capability set may include:

  • Workload authentication and model authorization
  • Provider credential protection and virtual keys
  • Standard request and response interfaces
  • Model aliases and provider abstraction
  • Usage metering, budgets, quotas, and rate limits
  • Routing, fallback, retries, and circuit breaking
  • Audit events and trace propagation
  • Caching where authorization permits it
  • Input and output guardrails
  • Sensitive-data detection and redaction
  • Model and use-case allowlists
  • Request-size and context limits
  • Schema and structured-output validation

The gateway should not contain all business logic. End-user authorization, prompt construction, tool selection, and domain rules normally belong in the application or agent runtime.

5. Model Providers and Inference Platforms

Behind the gateway, the enterprise may use direct model-provider APIs, public-cloud AI platforms, specialized inference providers, or self-hosted open-weight models.

Cloud-native platforms are a practical starting point for companies already standardized on a cloud because they integrate with cloud identity, networking, billing, and regional controls. Direct providers may expose capabilities earlier but create an additional contract, credential, billing path, and data boundary. Self-hosted inference offers deployment control but transfers responsibility for scaling, security, availability, upgrades, and model serving to the enterprise.

The reference architecture permits these options to coexist behind a controlled access path.

6. Policy, Data Controls, and Guardrails

Policy connects enterprise requirements to runtime decisions. It may consider workload, user, environment, data class, model, provider, region, and budget before allowing or routing a request.

Guardrails should be layered:

LayerResponsibility
Application or agentUser authorization, business rules, tool permissions, transaction limits, output validation
GatewayCommon input/output policies, data detection, model restrictions, schema checks, centralized enforcement
Model platformProvider-native content and model safety controls
Dedicated AI securityPrompt-injection defense, data-leakage controls, adversarial detection for higher-risk workloads

Guardrail actions may block, redact, warn, log, retry, or reroute. No guardrail product can determine domain-specific correctness or eliminate the need for least-privilege application design.

7. Routing, Resilience, and Portability

Routing is a logical architecture block even when it is implemented inside the gateway.

Three patterns are available:

  1. Gateway-integrated routing: The gateway selects a deployment using aliases, policy, cost, latency, availability, or region.
  2. Dedicated intelligent routing: A router such as RouteLLM selects between eligible models based on predicted quality, complexity, or cost. The gateway remains the enforcement point.
  3. Application-directed routing: The application selects a capability because it has the best task context, while the gateway verifies that the selection is allowed.

Fallbacks must satisfy the same data and compliance policy as the original destination. A technically available model is not automatically an approved fallback.

8. Agent Framework, Runtime, and Tool Governance

An agent framework defines agents, tools, handoffs, state, and execution flow. The runtime operates those workflows, including long-running execution, retries, queues, isolation, and resumability. The gateway controls model access but does not govern every action an agent can take.

The agent layer should provide:

  • An approved framework or supported framework portfolio
  • Workload and delegated user identity
  • Tool and MCP-server registry
  • Per-agent tool permissions
  • Step, time, cost, and concurrency limits
  • Isolated code execution
  • Action-level audit logs
  • Human approval for consequential actions
  • Retry, cancellation, and kill-switch behavior

For high-impact actions, the model should propose the action while deterministic code verifies authorization and performs the operation.

9. Cost Governance and FinOps

The gateway should record application, team, environment, provider, model, usage, and estimated cost. This enables real-time budgets, alerts, quotas, and rate limits.

Enterprise FinOps has a broader responsibility. It must reconcile provider invoices and include supporting costs such as retrieval, databases, queues, agent compute, and self-hosted inference. As maturity increases, the progression is:

Metering → budgets and alerts → attribution → forecasting → optimization → showback or chargeback.

Optimization may include smaller-model routing, caching, prompt reduction, agent-loop limits, batching, or self-hosted inference. Quality and cost must be measured together.

10. Enterprise Knowledge and Retrieval

Employee knowledge access and application RAG are separate architectural needs.

Company knowledge in ChatGPT can search approved connected apps for employee questions and return cited answers while respecting source permissions. It is a feature inside ChatGPT Business and Enterprise, not a separate product or a reusable backend for custom applications.

Product applications require a retrieval architecture with ingestion, change detection, classification, permission synchronization, authorized retrieval, citations, and evaluation. Permission checks should occur before unauthorized content is provided to the model or written to traces.

Organizations may use a dedicated enterprise-search product, managed cloud services, or a custom shared platform depending on scale and requirements.

11. Monitoring, Tracing, Audit, and Evaluation

The architecture produces four kinds of operational evidence:

  • Infrastructure monitoring for availability, latency, failures, and resource health
  • AI traces across retrieval, model calls, and tool actions
  • Security and compliance audit records
  • Quality evaluations using test datasets, production feedback, automated scoring, and human review

Telemetry requires its own data policy. Prompts, responses, retrieved documents, and tool output may contain sensitive information. The enterprise must define redaction, retention, encryption, regional storage, and operator access.

Cloud-Native Access or a Dedicated Gateway?

A cloud-native model platform may be sufficient when most workloads use one cloud, the model portfolio is limited, cloud identity and billing provide enough control, and cross-provider routing is unnecessary.

A dedicated gateway becomes valuable when applications use multiple providers, credentials must be isolated centrally, budgets and policies must apply across teams, or common routing, guardrails, and audit records are required.

The two approaches are complementary: a dedicated gateway can use cloud-native model platforms as backends.

Managed, Self-Hosted, or Built Internally?

Managed products reduce operating effort. Self-hosted products increase deployment and data control but require upgrades, security maintenance, scaling, and availability ownership. Internal development should be reserved for requirements that available products cannot satisfy.

A model proxy or RAG demonstration may be simple. A production platform with identity, policy, streaming, retries, accounting, permissions, deletion, tracing, and high availability is not.

From Reference Architecture to Prescription

The architecture is intentionally broader than any one organization should deploy immediately. A small company may need only an approved employee assistant, one cloud model platform, and basic budgets. A mid-size organization may need a managed gateway, shared tracing, and application-level RAG. A large enterprise may require regional gateways, multi-provider routing, dedicated guardrails, chargeback, a shared agent runtime, and self-hosted components.

The next article, Prescribed Enterprise AI Architectures for Small, Mid-Size, and Large Organizations, selects a coherent implementation and representative vendors for each profile.

Enterprise AI Adoption: Requirements for Security, Governance, and Scale

AI, , , ,

AI adoption inside enterprises rarely begins with a coordinated platform strategy. It usually starts with individuals and teams selecting tools that solve immediate problems.

A developer subscribes to an AI coding assistant. Another team builds an application using a foundation-model API. Employees begin using standalone cloud applications such as ChatGPT, Claude, Gemini, or Codex. A business unit connects an AI application to internal documents. Eventually, the organization considers an enterprise-wide knowledge assistant or retrieval-augmented generation system.

Each decision may be reasonable on its own. Together, however, they create a fragmented operating environment.

Users authenticate differently across tools. Applications depend on provider API keys. Usage is billed through unrelated subscriptions and cloud accounts. Sensitive data may pass through systems with different retention and training policies. Security teams have limited visibility into what is being used, while finance teams struggle to determine which teams, applications, or business outcomes are responsible for the expenditure.

The enterprise AI problem is therefore not simply about selecting the best model or application. It is about enabling several forms of AI consumption while maintaining consistent control over identity, credentials, data, cost, providers, and operations.

Fig: Different AI consumption paths require one consistent set of enterprise controls

Four Ways Enterprises Consume AI

Enterprise AI adoption can be organized into four broad consumption patterns. These patterns overlap, but their ownership, identity, billing, and data-flow characteristics differ enough to require separate consideration.

1. AI-assisted software development

The first pattern is the use of AI by software-development teams. It includes IDE assistants, command-line tools, coding agents, code-review assistants, and autonomous development applications.

Some products are purchased as per-user subscriptions. Others allow users to connect a model-provider account or consume models through API credentials. More autonomous tools may execute commands, inspect repositories, modify files, or interact with development infrastructure.

The enterprise must determine whether users authenticate through corporate identity, which repositories and environments the tool may access, how source code is handled, how expenditure is attributed, and whether agent actions are logged and auditable.

The risk is not limited to source-code disclosure. Coding agents can access credentials, execute commands, change infrastructure, or introduce dependencies. Their permissions and actions require greater scrutiny than conventional autocomplete.

2. Custom AI applications, agents, and workflows

The second pattern consists of applications built or operated by the enterprise. Examples include customer-service assistants, document-processing workflows, incident-response agents, internal copilots, and automated business processes.

These systems usually consume foundation models through APIs. They may retrieve enterprise data, invoke tools, call internal services, maintain state, and perform actions on behalf of users.

Unlike a standalone application, the enterprise owns much of the operating responsibility. It must manage application identities, API credentials, authorization, model selection, usage limits, monitoring, and failure handling.

Agents make this especially important. A traditional application normally follows a path defined in code. An agent can dynamically choose a model, call tools, retrieve information, and take several steps to complete a task. The architecture must govern both model access and access to data and tools.

3. Standalone hosted AI applications

The third pattern is the direct use of cloud-hosted AI applications by employees. This includes general-purpose assistants and specialized applications for coding, research, writing, design, analytics, and productivity.

The provider operates the application and normally controls the interface, model selection, data handling, and underlying infrastructure. The enterprise consumes the product through user accounts, team subscriptions, or an enterprise agreement.

These products may be introduced through centralized procurement, but they can also enter through individual subscriptions and expense claims. This creates shadow AI: employees using services that the organization has not reviewed or cannot administer.

The enterprise must evaluate SSO, automated provisioning, administrative controls, retention and training terms, audit logs, sharing, and connections to internal systems. A model gateway cannot normally intercept these interactions because the application communicates directly with its provider.

4. Enterprise knowledge search and RAG

The fourth pattern is enterprise knowledge search, often implemented using retrieval-augmented generation, or RAG.

These systems search enterprise data sources and provide relevant content as context for a model response. Sources may include document repositories, collaboration systems, source-code platforms, ticketing systems, knowledge bases, or business applications.

An organization can purchase enterprise search, use knowledge features within a hosted assistant, build a custom RAG platform, or combine managed services with internal components.

The difficult part is rarely retrieving a document and sending it to a model. The harder requirements are preserving source permissions, keeping indexes current, removing content when permissions change, recording citations, preventing unauthorized retrieval, and measuring retrieval and answer quality.

Why Fragmentation Becomes a Governance Problem

The four patterns do not share a common operating model.

Coding applications may be licensed per user. Custom applications consume APIs based on tokens or requests. Hosted applications may use individual, team, or enterprise subscriptions. Knowledge platforms may combine licenses, indexed-content charges, infrastructure costs, and model consumption.

Identity is similarly fragmented. Employees authenticate interactively, while applications and agents use non-human identities. Some products support enterprise federation and automated provisioning; others rely on independently managed accounts. One employee may also connect several applications to internal data using separate authorization grants.

Data may be entered into a hosted application, sent to an API by an internal workload, retrieved from enterprise repositories, recorded in traces, or stored in provider logs.

Managing every product independently leads to predictable problems:

  • Former employees retain access to tools that were not centrally provisioned.
  • Provider API keys are shared between developers or embedded in applications.
  • Teams cannot identify which application caused an increase in expenditure.
  • Sensitive information is sent to models that are not approved for that data class.
  • Security teams cannot reconstruct which prompt, context, model, or tool produced an action.
  • Applications become tightly coupled to provider-specific interfaces.
  • Different teams build duplicate integrations and governance mechanisms.

A scalable AI program requires one set of enterprise requirements, even when those requirements are implemented differently across consumption patterns.

Seven Requirements for Enterprise AI Adoption

1. Identity and access governance

Human access should connect to the corporate identity system wherever the product permits it. This includes SSO, multi-factor authentication, role-based access, centralized provisioning, and timely deprovisioning.

Access decisions should reflect the sensitivity of the tool and the data it can reach. A general assistant with no internal connections does not present the same risk as an agent that can access source code, customer records, or production systems.

The organization also needs clear ownership. Someone must approve the product, determine who may use it, review access, and remove access when it is no longer required.

2. Non-human identity and credential governance

Applications and agents should not depend on personal API keys or shared provider credentials.

They require non-human identities with narrowly scoped permissions. Credentials should be issued through an approved process, stored in a secrets-management system, rotated, and revoked without disrupting unrelated applications.

Where possible, applications should receive virtual or intermediary credentials rather than direct provider keys. This allows application-specific limits and provider changes without redistributing credentials to every workload.

3. Cost attribution and financial governance

AI cost is variable and can grow quickly as applications gain users, agents perform more steps, or teams select more capable models.

Enterprises need visibility beyond the provider invoice. Usage should be attributable to a user, application, team, environment, or business unit. Budgets, quotas, alerts, and rate limits should operate at the same levels.

Small organizations may need only visibility and spending alerts. Larger organizations may require forecasting, showback or chargeback, unit-cost measurements, and formal FinOps ownership.

Where possible, expenditure should be connected to outcomes such as completed tasks, resolved cases, generated documents, or developer workflows. Token counts alone do not indicate whether an application is successful.

4. Security and data governance

The enterprise must define what data can be processed by which products, models, providers, and regions.

This requires consistent review of retention, provider-training terms, encryption, residency, subprocessors, auditability, and incident-response commitments. It must also cover information stored outside the primary request, including traces, prompts, retrieved context, cached responses, evaluation datasets, and application memory.

Controls should be proportional to data sensitivity. Public information, internal documents, source code, personal data, and regulated records should not automatically follow the same approval path.

5. Model and vendor flexibility

Model capabilities, prices, performance, and availability change rapidly. An application unnecessarily coupled to one provider becomes difficult to optimize or migrate.

Enterprises should use stable internal interfaces where the benefits justify the effort. Provider-specific features can still be used, but the dependency should be intentional and documented.

Vendor flexibility does not mean every request must switch dynamically between providers. It means the architecture preserves a practical path to compare, route, replace, or add models without redesigning every application.

6. Cost and performance optimization

Not every task requires the most capable or expensive model.

Applications should select models according to task complexity, latency, quality, privacy, and cost. Repeated requests may benefit from caching. High-volume and predictable workloads may eventually justify self-hosted open-weight models.

Optimization should follow measurement. A cheaper model is not cheaper if poor outputs cause repeated calls, human correction, or failed workflows. Cost, latency, and quality must be evaluated together.

7. Monitoring, tracing, and evaluation

Traditional infrastructure metrics are necessary but insufficient for AI systems.

Teams need to understand which model was used, how much was consumed, how long each step took, which context was retrieved, which tools were called, and where failures occurred. Agent traces may span multiple model calls and actions within one task.

Operational monitoring must be complemented by evaluation. An application can remain technically available while quality deteriorates because of a model change, prompt update, retrieval problem, or altered data source.

A Shared Control Point for Model Access

For custom applications, agents, and compatible coding workflows, many requirements can be enforced through a shared model-access layer.

This layer sits between enterprise workloads and model providers. Depending on the organization’s needs, it can authenticate applications, protect provider credentials, enforce model policies, route requests, apply quotas, attribute cost, record audit events, collect traces, and provide a consistent interface across providers.

It is often described as a model gateway or AI gateway.

The gateway is an important architectural control point, but it is not the entire governance solution. Standalone hosted applications generally operate outside it. Knowledge systems require source authorization and retrieval controls. Agents require tool and action governance. Coding tools require safeguards for source code, credentials, local execution, and development infrastructure.

Requirements Change with Enterprise Context

The requirements are broadly consistent, but implementation should reflect scale, risk, and operating maturity.

A small company may prioritize a limited set of approved tools, centralized billing, basic SSO, and managed model access. A mid-size organization may need a dedicated gateway, per-application credentials, team budgets, audit logs, and shared observability. A large enterprise may require multiple providers, regional deployments, formal data classifications, policy enforcement, business-unit isolation, chargeback, advanced evaluation, and self-hosted components.

Employee count alone is not enough. A smaller company handling regulated data may require stronger controls than a larger organization working mainly with public information. AI expenditure, data sensitivity, number of applications, regulation, and platform-team capability are often better indicators of architectural need.

Questions the Reference Architecture Must Answer

These requirements lead to concrete architectural questions:

  • Where should model access be centralized?
  • How should human users, applications, and agents authenticate?
  • How should provider credentials be isolated?
  • Which controls belong in the identity provider, gateway, application, or provider?
  • How should standalone hosted applications be governed?
  • Where should prompts, traces, retrieved context, and evaluation data be stored?
  • How should costs be attributed and controlled?
  • When is a cloud provider’s model platform sufficient?
  • When does an organization need a dedicated gateway?
  • When should enterprise knowledge search be purchased, built, or deferred?
  • How should the architecture evolve as usage, risk, and maturity increase?

There is no single product that answers every question. The solution is an architecture that combines identity, model access, security, cost management, observability, and application-level controls.

The next article, A Reference Architecture for Governed Enterprise AI, develops that architecture and explains the available implementation choices.

VLAs Are Winning. World Models Will Win.

AI, , , ,

A perspective from the data layer of humanoid robot training

I recently spent time working on egocentric video data — first-person footage of humans doing everyday things with their hands — and how it could be used to train humanoid robots. It’s a narrow problem, but it forced me to answer a bigger question: what do these models actually need to learn, and why is it so hard to give it to them?

That work pulled me into the two architectures competing to become the brain of the next generation of robots: Vision-Language-Action models and world models. I’m not a researcher in either. But sitting at the data layer gives you a particular vantage point — you see what these models consume, where they’re data-hungry, and where more data stops helping. This post is my read on where the field is heading, from that vantage point.

What Comes After LLMs?

LLMs learned the statistical structure of text. The next wave of AI has to learn the statistical structure of the physical world — how objects move, how forces act, what happens when you push, pour, or grasp. The vehicle for that transition is the humanoid robot.

For sixty years, robotics meant programming a machine to do one thing precisely, in a cage, because it couldn’t adapt to anything outside its parameters. What changed is the same thing that changed everything else: transformers that generalize across modalities, vision-language models that reason jointly over what they see and what they’re told, and compute cheap enough to train at scale. Teaching a robot a task used to be a years-long research project. Companies are now doing it in months.

From LLM to VLM to VLA

The path to today’s robot brains is a straight line through three architectures:

LLM → VLM → VLA

An LLM takes text in and produces text out. A VLM (vision-language model) adds a vision encoder — now the model takes images and text in, and produces text out. It can describe what it sees, answer questions about a scene, reason about spatial relationships.

A VLA takes the final step: images and language instructions in, robot actions out. The key insight — first demonstrated at scale by Google’s RT-2 in 2023 — is that robot actions can be treated as just another token type. Discretize continuous motor movements into tokens, and the same transformer that predicts words can predict actions. Everything the model learned from web-scale image data transfers into control: a robot that never saw a banana in its training demonstrations can still pick it up, because it knows what bananas look like from the internet.

The lineage since then: RT-1 proved robotics transformers work at scale (2022), RT-2 brought web-scale generalization (2023), OpenVLA open-sourced the recipe (2024), and Physical Intelligence’s π0 pushed it furthest with diffusion-based action generation (2024).

And the results are genuinely impressive. Physical Intelligence’s demos are the ones that made me stop and pay attention: a robot folding laundry — deformable objects, one of the classically hard problems in manipulation — and clearing a table into a dishwasher. Most striking, doing this in homes it had never seen. That last part matters more than the tasks themselves: the promise of a VLA is precisely that it generalizes to environments outside its training data, the way an LLM answers questions it was never explicitly trained on.

Where VLAs Hit a Wall

But working at the data layer, you start to see the shape of the ceiling. Three failure modes show up consistently:

Long-horizon tasks. Compound errors are brutal: 95% per-step reliability gives you 60% success on a ten-step chain. Folding a shirt is one skill; cooking a meal is a sequence of dozens, and today’s VLAs degrade sharply as the chain grows.

Instruction drift. As a task progresses, irrelevant observations dilute the model’s attention to the original instruction. The robot does things that are locally reasonable but globally wrong — it forgets what it was trying to do.

Generalization limits. The environment generalization is real but shallow. Change the physics of the task — not just the room — and success rates collapse. More demonstrations help less and less.

The common thread: a VLA is fundamentally reactive. It maps what it sees to what it should do next. It has no mechanism for asking “what happens if I do this?” before doing it. And that question, I’ve come to believe, is the whole game.

World Models: Learning to Predict, Not Just React

A world model learns to simulate: given the current state of the world and an action, predict what the world looks like next. Where a VLA asks “what should I do given what I see?”, a world model asks “what will happen if I do this?” — and can plan through imagined futures before committing to real ones.

Yann LeCun has been arguing for this direction longer than almost anyone, and his JEPA architecture is worth understanding because it’s a genuinely different bet from the transformer lineage above.

The obvious way to build a predictive model is to predict the future in pixel space — generate the next video frame. JEPA (Joint Embedding Predictive Architecture) rejects this. Its core idea: most of the pixel-level future is unpredictable and irrelevant. The exact ripple pattern of spilling water, the flutter of a curtain — no model can predict these, and no robot needs to. JEPA instead predicts in representation space: it learns abstract embeddings of world state and predicts how those embeddings evolve. It learns that the glass will fall without simulating every photon of the falling. That’s much closer to how humans model the world — we predict consequences, not video frames. LeCun’s blunt version of the argument: you cannot get to physical intelligence by scaling next-token prediction, because the world is not made of tokens. JEPA is now the foundation of his post-Meta startup, AMI Labs.

Fei-Fei Li’s World Labs comes at it from a different angle: spatial intelligence. Her June 2026 taxonomy splits world models into three functions — renderer (generate visual representations), simulator (model how objects respond to forces), and planner (reason over action sequences). Their product Marble generates persistent, navigable 3D environments: a robot training in one can approach the same shelf from different angles and find the same objects in the same places, because the geometry is consistent rather than regenerated per frame.

NVIDIA’s Cosmos attacks the problem I lived inside: data. Nobody has enough real robot demonstrations. Cosmos uses video world models to generate physically plausible synthetic training data. The model problem and the data problem become the same problem — better world models generate better data, which trains better robots.

The Philosophical Fork

Underneath the company logos, the field has split on one question: where does world understanding live?

The structure camp — LeCun, World Labs, Cosmos — says you must build it explicitly. A separate, inspectable model of how the world works, with policies plugged into it.

The scale camp — Physical Intelligence and the VLA lineage — believes world understanding emerges inside the weights if you train on enough diverse data, the same way LLMs developed reasoning nobody explicitly built. Some researchers claim to find world-model-like internal representations when they probe VLAs. I’m skeptical of how far that goes. A VLA translates observations into motions; nothing in its architecture simulates the world or checks a plan against predicted consequences. Calling that an implicit world model is, for now, the scale camp’s hope more than a demonstrated fact.

Google DeepMind’s Gemini Robotics is the most interesting data point, because it splits the difference architecturally. It pairs two models: Gemini Robotics-ER, an embodied reasoning model that plans multi-step tasks, calls tools, and can pull live context (their demo: sorting trash into bins using recycling rules fetched from the web), and the Gemini Robotics VLA, which turns each planned step into motor commands. To be precise, ER isn’t a world model — it doesn’t simulate physics; in Fei-Fei Li’s taxonomy it’s a planner, not a simulator. But the design concedes the structural point: reactive execution alone isn’t enough, and the reasoning layer has to be explicit and separate.

The open ecosystem is converging the same way. Hugging Face’s LeRobot has become the de facto open toolkit, NVIDIA’s GR00T models are openly available for post-training, and Cosmos integration is bringing world-model-generated synthetic data to teams that can’t collect real data at scale.

My Take: VLAs Are the Present, World Models Are the Future

VLAs work today and will keep improving. For well-defined tasks over short horizons, they’re a real solution now, and the laundry-folding demos aren’t tricks.

But the ceiling is structural, not incremental. Long-horizon planning, genuine generalization, acting safely in novel situations — these require predicting consequences before acting, and a reactive observation-to-action mapping has no place to put that capability. You can’t fine-tune your way to a simulator.

World models have their own hard problems — sim-to-real gaps, compute cost, learning accurate physics from video. But these look to me like engineering problems on the right path, whereas the VLA ceiling looks like the wrong abstraction for the long game.

My prediction: world models won’t replace VLAs — they’ll become the layer underneath them. World models handle prediction, planning, and synthetic data generation; VLAs handle dexterous real-time execution. Gemini Robotics’ reasoner-plus-executor split is an early, partial version of this stack, and I expect the rest of the field to converge on it whether or not anyone names it.

Watching what these models need from their training data is what convinced me. VLAs are data-hungry in a way that suggests they’re memorizing behavior rather than understanding the world — and the physical world is too varied to memorize. At some point, a robot has to stop pattern-matching and start predicting.

That’s what world models are built to do. That’s why I think they win.

AI Assistants Have Many Interfaces. Context Is the Real Product.

AI, , , , , ,

AI assistants are no longer just chat windows. The same assistant now appears as a web app, desktop app, mobile app, browser extension, IDE extension, command-line tool, local agent, and cloud worker.

That is powerful, but it creates a new problem: deciding which interface to use, and keeping context alive when moving between them.

This post is based mostly on my experience with OpenAI and Anthropic products: ChatGPT, Codex, Claude, Claude Code, and their web, desktop, IDE, CLI, mobile, browser, and cloud interfaces. I also touch briefly on Gemini and browser-extension-style workflows, because they represent another way people are starting to interact with AI.

The question I am interested in is not just which model is better. It is: which interface should I use, when should I use it, and why does context still get lost when I move between them?

The OpenAI Interfaces

InterfaceHow I think about it
ChatGPT webBest for general thinking, writing, research, analysis, and normal assistant workflows.
ChatGPT mobileUseful when I am away from my laptop. Also useful as a controller for connected Codex hosts.
ChatGPT Voice ModeExcellent for brainstorming. It feels like a real-time conversation, not just dictation.
Codex desktop appMy default for local agent work. Best when the task needs local files, terminal commands, browser sessions, or writing changes on my Mac.
Codex VS Code extensionUseful for bigger projects inside the IDE, especially when I want to work across multiple agents or keep the workflow inside one editor.
Codex CLIPowerful for terminal-native workflows, but I do not use it much because I prefer seeing code and diffs visually.
Codex Web / CloudUseful when the repo is on GitHub and I want a small bounded change, PR-style task, or cloud execution without relying on my laptop.

The Codex desktop app is the OpenAI interface I use most for local work. For anything that needs access to local files, local folders, terminal commands, browser sessions, or writing changes on my Mac, the desktop app is my default.

It gives me a practical local agent environment where I can see what is happening, approve actions, inspect changes, and let the assistant work inside my machine.

The Codex VS Code extension is useful when I am already inside the IDE, especially for bigger projects where I want a single editor surface and may work across multiple agents or threads.

The Codex CLI is powerful, but I personally do not use it much. I prefer the feel of seeing the work visually while changes are being made.

The Codex Web / Cloud mode is different. It is useful when the work is already in GitHub and I want to make a small change, run a bounded task, or delegate something in a PR-style workflow.

In this mode I do not need a local workspace, and execution does not happen on my laptop. The assistant works in the cloud against the repository.

That has obvious advantages. If my laptop is not available, or if I want something long-running to continue without depending on my machine staying awake, cloud execution makes sense. It also works well when the task is self-contained and the repository can build and test cleanly in a cloud environment.

But cloud is not a replacement for local work in every case. If the task depends on unpushed local files, local credentials, desktop apps, browser sessions, local databases, or my Mac setup, the desktop app is still more convenient.

The Anthropic Interfaces

InterfaceHow I think about it
Claude webGood for general Claude chat, writing, thinking, analysis, and projects.
Claude mobileUseful for mobile access and remote workflows, but not a full replacement for desktop/project context.
Claude desktop appUseful, but the experience feels split across Chat, Cowork, and Code.
CoworkUseful for local desktop-style tasks, especially for non-technical users, but I do not fully understand why it needs to be separate from Chat.
Claude Code CLIMy main serious Claude coding workflow, especially inside VS Code.
Claude Code in VS CodeUseful when I want Claude close to the code editor.
Claude Code web/cloudGood when I want execution to happen in the cloud rather than on my local machine.
Dispatch / Remote ControlUseful ideas, but they do not feel like one unified control layer yet.
Browser extensions / browser usageUseful, but not the best long-term workflow yet because the integration does not feel smooth enough.

Claude has a similar spread of interfaces, but the experience feels more fragmented to me.

There is Claude web for general chat and projects. There is the Claude mobile app. There is the Claude desktop app, which separates the experience into areas like Chat, Cowork, and Code. There is Claude Code CLI, Claude Code in VS Code, and Claude Code on the web.

For serious coding, my main Claude workflow is Claude Code CLI inside VS Code. That combination feels powerful because I get the capabilities of the CLI while still keeping the editor open and visible.

Claude’s other modes are useful too. Cowork can help with local desktop-style tasks. Claude Code web provides a cloud coding mode when I want the execution to happen away from my machine. Dispatch and Remote Control are useful ideas for sending or steering work from another device.

But the product feels more split. The pieces are good, but I often feel the boundaries between them.

Voice Is Another Interface

One interface I do not want to ignore is voice.

ChatGPT Voice Mode is one of the most useful non-coding interfaces for me. It is especially good for brainstorming. Speaking to the assistant and getting a real-time spoken response feels very different from typing, or even from using a dictation tool.

Tools like Wispr Flow are useful because they let me speak instead of type. But that is still mostly a better input method for a text conversation. It is not the same as a real-time voice conversation.

ChatGPT Voice Mode feels closer to a true conversational interface. It feels less like “generate text, then read the text aloud,” and more like a direct voice interaction.

Claude also has voice capabilities, but in my usage it does not feel as natural as ChatGPT Voice Mode. It feels more like speech-to-text followed by a spoken response. That may not be the exact implementation, but from a user experience standpoint the difference is noticeable. The delay and response style make it less useful for live brainstorming.

I would also like to see this kind of voice experience inside the Codex app. If I am already working in a local agent workspace, being able to brainstorm with Codex by voice would be very useful. I may not want voice for every coding task, but for planning, debugging, architectural discussion, and reviewing tradeoffs, it would be a natural interface.

Mobile As A Controller Interface

Another interface that I find useful is ChatGPT mobile as a controller for Codex.

From the ChatGPT mobile app, I can connect to Codex running on my Mac or Windows machine and access the projects and threads available on that connected host. I can continue work, send follow-up instructions, approve actions, and review results from my phone.

That is a powerful pattern. The phone is not trying to become the full development environment. It is controlling the Codex environment already running on my machine.

As long as that host is awake, online, paired, and signed in, I can continue threads, approve actions, and inspect results. The permissions still belong to the host-side Codex session.

This is different from Codex Web, where the work happens in the cloud against GitHub. Both are useful, but they solve different problems.

Claude has related ideas through Dispatch and Remote Control, but it does not feel the same to me. Dispatch is more like sending work from mobile to desktop. Remote Control is useful for steering a running Claude Code session. But the experience still feels more split between Claude mobile, Claude desktop, Claude Code, Cowork, and Claude Code web.

What I would like is a more unified control layer: mobile, web, desktop, local machines, and cloud environments should feel like different surfaces over the same underlying work context.

Local vs Cloud

The way I think about local and cloud is simple.

Use local when the machine matters.

Use cloud when the shared repository or online workspace is the source of truth.

Local is better when I need my files, my terminal, my browser, my desktop apps, my local setup, or visual feedback. This is why I use the Codex desktop app so much.

Even though it runs with sandboxing and permissions, once I allow the right operations, it can work with my Mac and browser fairly smoothly. Compared with some other local agent experiences, this makes Codex feel more convenient for my daily workflow.

Cloud is better when the task is centered around a shared online source such as GitHub. If the code is pushed, the task is bounded, and the assistant can work in a clean environment, cloud agents are very useful.

They are especially good for small fixes, dependency updates, tests, review follow-ups, PR-style tasks, and background work that should not depend on my laptop staying awake.

The hybrid model is probably the most realistic. I may explore and develop locally, push a branch, then ask a cloud agent to do a bounded follow-up. Or I may use cloud for a small GitHub change while continuing deeper work locally.

The key is discipline: local and cloud workflows work best when the shared source of truth is clean and the assistant is given a clear task.

My Current Workflow

My personal workflow today is roughly this.

For OpenAI, I mostly use the Codex desktop app when the task needs local access or local file changes. It gives me the best balance of visibility, control, and convenience.

For bigger projects inside the editor, I use the Codex VS Code extension, especially when I want to work across multiple agents or keep the whole workflow inside one IDE.

I use Codex Web / Cloud selectively. If something is already on GitHub and I want a small change or a bounded task, it is a good fit. I do not use it as my main development environment, but I see the value clearly.

For general thinking, writing, research, and brainstorming, I use ChatGPT web, mobile, and Voice Mode. Those are still very useful interfaces. But they are separate from the local Codex app context, and that separation matters.

For Claude, I primarily use Claude Code CLI inside VS Code for bigger coding projects. That feels like the strongest Claude coding workflow for me right now.

I also use browser-based tools across Claude, Gemini, and Codex-style workflows, but I see room for improvement there. The browser is important, but the current extension-style experience does not yet feel like the final form.

What OpenAI Gets Right

What I like about Codex is that it feels like a unified local agent workspace.

In the Codex desktop app, I can ask questions about the project, inspect files, make changes, run commands, review diffs, manage threads, and use local browser/computer tools from one place. That reduces the number of decisions I have to make before starting work.

The local desktop app is especially useful because it works with my actual machine. It is sandboxed, and permissions still matter, but once I approve the right operations, it can interact with my Mac and browser smoothly enough for real work.

ChatGPT mobile controlling connected Codex projects is also a strong pattern. It shows what a good cross-device AI interface can look like. The mobile app becomes a control surface over the environment where the work is actually happening.

ChatGPT Voice Mode is another strong interface. For brainstorming, it is one of the best ways to interact with an AI assistant.

What Anthropic Gets Right

Claude Code CLI is very strong. Used inside VS Code, it gives me a powerful workflow while still letting me see the project in the editor. For bigger projects, this works well.

Claude also has powerful separate modes. Chat, Cowork, Code, CLI, web, mobile, Dispatch, Remote Control, and IDE integration all have a reason to exist. The pieces are good.

Claude Code web/cloud is useful when I want execution to happen in the cloud rather than on my machine. Dispatch and Remote Control are also interesting because they recognize that users want to start or steer work from different devices.

What I Would Like To See Improved In Claude

My issue with Claude is not capability. It is product shape.

As a user, I would prefer one universal Claude experience where chat, cowork, and code feel like modes of the same workspace rather than separate places.

I can understand Code being a specialized mode because coding has its own environment, tools, permissions, and workflows. But the separation between Chat and Cowork is less obvious to me.

Claude Cowork seems designed to make agentic desktop work easier for non-technical users. That makes sense. Not everyone wants to use a terminal or think in terms of repositories, branches, commands, and diffs.

But if I am chatting with Claude and the discussion turns into a task, why should I need to move into a different mode? Ideally, Cowork would feel like a capability inside the same Claude workspace rather than a separate place. The assistant should be able to move from discussion to action naturally, while still asking for the right permissions when it needs to touch files, apps, or the computer.

I would also like Claude’s voice experience to feel more natural for live brainstorming. In my usage, ChatGPT Voice Mode feels closer to a real-time conversation, while Claude voice feels more like speech-to-text followed by a spoken response.

Where The Interfaces Still Break Down

The issue is not that there are many interfaces. Different interfaces are useful for different jobs.

Voice is good for brainstorming. Desktop is good for local work. IDE is good for deep project work. Cloud is good for background execution. Mobile is good for steering work.

The problem is that the context does not always travel with me.

A few examples:

ChatGPT voice to Codex

If I brainstorm an idea in ChatGPT Voice Mode on mobile or web, that conversation does not naturally appear inside the Codex desktop app. If the brainstorming leads to an implementation task, I need to manually restate the context in Codex.

Codex desktop to ChatGPT mobile

This works better. If Codex is running on my Mac and the machine is awake, online, paired, and signed in, I can access those Codex projects and threads from ChatGPT mobile. This is one of the best examples of a useful cross-device AI interface.

Codex desktop to ChatGPT web

This is where the continuity feels incomplete. I can control connected Codex hosts from ChatGPT mobile, but I do not get the same connected-host control surface from ChatGPT web. Since I often work from a browser too, I would like the web interface to become another control surface for the same Codex host context.

ChatGPT web or mobile to Codex desktop

The reverse direction is also incomplete. General ChatGPT conversations, projects, and voice brainstorms do not automatically become available as working context inside Codex. That matters because many tasks start as thinking or planning before they become implementation.

Claude Chat, Cowork, and Code

In Claude, the fragmentation feels different. Claude has Chat, Cowork, Code, Claude Code CLI, Claude Code web, mobile, Dispatch, and Remote Control. Dispatch is useful because I can send work from mobile to desktop, and Remote Control is useful for steering a session. But it does not feel like one shared workspace where the same context naturally follows me across Claude web, desktop, mobile, and Code.

Claude Code local vs cloud

Claude Code has both local and cloud-style workflows. Local Claude Code is useful when I want the work to happen inside my own machine or IDE. Claude Code web/cloud is useful when I want the task to run away from my machine, usually against a GitHub-backed environment.

That separation makes sense technically. Local work and cloud work have different permissions, files, tools, and execution environments. But from a user experience standpoint, I still want the context to move more naturally between them. If I plan something in Claude chat, start work in Claude Code CLI, and later move to Claude Code web, I do not want to reconstruct the whole task manually.

The Real Problem Is Context Continuity

The missing piece is not one universal interface. I actually want multiple interfaces.

What I want is a shared context layer underneath them.

If I brainstorm in voice, I should be able to continue in desktop. If I start work in a local agent, I should be able to inspect it from mobile and web. If I delegate work to the cloud, the result should be easy to pull back into the local or conversational context.

If I switch from Claude web to Claude Code, or from ChatGPT to Codex, I should not have to reconstruct the entire task history manually.

The best current example of this working is ChatGPT mobile controlling Codex projects on a connected machine. That shows the direction I want: mobile is not replacing the desktop environment; it is becoming a control surface for it.

The next step is making that idea more universal across web, desktop, mobile, voice, IDE, local agents, and cloud agents.

At the same time, permissions should remain local to the right environment. I do not want every interface to have every permission. Local files, desktop apps, browser sessions, and computer control should stay tied to the machine where permissions were granted. Cloud work should stay in the cloud. Mobile should control what it is allowed to control.

But the reasoning context, project context, and user intent should travel better across these surfaces.

Where I Think This Is Going

The future of AI tools is not just better models. The models will keep improving, but the interface and context layer may matter just as much.

The winning product will be the one that lets me move between local, cloud, IDE, web, browser, desktop, mobile, and voice without constantly re-explaining what I am doing.

For me, that is the real product: not just the assistant, not just the model, and not just another interface.

The real product is context continuity.