I have been thinking about how AI should fit into CI/CD. Not the part where a developer uses an AI coding agent before opening a pull request. That is important, but it is still developer workflow.

The part I wanted to test was what happens after the pull request is opened.

At that point, a normal CI pipeline already has several responsibilities:

• Run tests.
• Run static analysis.
• Run security checks.
• Possibly run dynamic or integration checks.
• Give reviewers useful evidence before a human approves the change.

AI can fit into this, but it should not replace every deterministic tool. Linters, type checkers, test frameworks, dependency scanners, and security scanners are still valuable because they are repeatable. The interesting question is where an AI reviewer adds judgment: finding logical defects, interpreting tool output, spotting missing tests, and giving a reviewer a concise explanation of risk.

This first experiment focuses only on AI code review for pull requests. I compared three approaches:

1. Hosted Codex Cloud code review
2. A repository-owned GitHub Action using Codex
3. CodeRabbit

The broader CI question — static analysis, dynamic analysis, security tooling, and testing with AI interpreting the results — is covered in Part 2: AI in CI Should Not Replace Your Tools. It Should Read the Evidence.

The Question I Wanted to Answer

The starting point was simple: if an AI tool can write code, can a different AI tool review it?

My instinct was that the reviewer should be independent from the coding agent. If the same model, prompt, and context wrote the change and then reviewed it, the review is less convincing. It may repeat the same assumptions. Independence matters.

But independence does not automatically require a dedicated code review product. A coding agent can also act as a reviewer if it is run with a different prompt, different constraints, and a review-only role.

So the actual comparison is not just “model vs product.” It is:

• How much review quality comes from the model?
• How much comes from the prompt and repository instructions?
• How much comes from operational product features such as GitHub integration, inline comments, deduplication, review status, dashboards, and configuration?
• How much control does the repository owner retain?
• How much effort is required to make the workflow reliable?

The Test Repository

I used a real repository for the experiment:

Repository: smakam/doctor-parser

The repository setup was introduced in PR #1. The repository-owned inline-review publisher was added in PR #3. I then used controlled experiment PRs for the actual comparison rather than reviewing the setup changes themselves.

The application has a backend and frontend, including code paths for uploaded doctor nameboard extraction and review/correction flows. That gave us enough realistic surface area to test authorization and data-mapping bugs.

To keep the comparison fair, I used two controlled defects.

Defect 1: Authorization Check Inversion

In backend/app/routers/nameboard.py, the access check was changed from denying access when neither ownership identifier matched to denying access when the session identifier did match.

-    if uploader != requesting_user_id and record.session_id != requesting_user_id:
+    if uploader != requesting_user_id and record.session_id == requesting_user_id:

The expected finding was a high-priority authorization issue: a requester whose ID differs from both the uploader and session ID could access or mutate another record, while a matching guest session could be incorrectly denied.

Defect 2: Incorrect Correction Mapping

In backend/app/services/review_service.py, the correction mapping for medical_registration_no was changed to write into qualifications.

-        "medical_registration_no": "medical_registration_no",
+        "medical_registration_no": "qualifications",

The expected finding was a data integrity issue: submitting a registration-number correction would overwrite qualifications and leave the actual registration number unchanged.

These are useful defects for this kind of experiment because they are not formatting issues. They require the reviewer to understand intent, data flow, and security impact.

Shared Review Guidance

One important design choice was to make the review guidance repository-owned.

The shared policy lives in .github/review/code-review.md.

The policy tells the reviewer to prioritize:

• Authorization and access-control regressions
• PII exposure
• Security-sensitive behavior
• Logic defects introduced by the pull request
• Incorrect persistence or field mapping
• Missing validation that changes behavior

It also sets a threshold: only report actionable defects introduced by the PR. Do not produce a generic style review.

This is an important lesson from the experiment. The model matters, but the review role matters just as much. If you want an AI reviewer, you need to define what “good review” means for your repository.

How the Three Review Paths Fit Together

All three approaches start with the same pull request and the same repository-owned review policy. What changes is who runs the review and who owns the operational layer around it.

The review policy is shared. The execution and publication layers differ.

Option 1: Hosted Codex Cloud Code Review

The first approach used hosted Codex Cloud code review.

In this mode, the repository uses AGENTS.md as the durable instruction entry point. For PR reviews, AGENTS.md routes the reviewer to the shared review policy.

There is no GitHub Actions workflow involved in this path. The hosted Codex integration reviews the pull request and posts inline GitHub review comments.

In our experiment, hosted Codex posted inline comments as chatgpt-codex-connector on PR #4.

Hosted Codex posted native inline findings on the changed code.

What Worked Well

The setup cost was low. Once the repository guidance was in place, the hosted reviewer could use that context and post inline comments directly on the PR.

It found the important seeded defects, including the authorization issue and the incorrect field mapping.

This is the simplest path if you want an AI reviewer without building a review pipeline.

What You Give Up

The tradeoff is control.

You do not own the workflow in the same way you own a GitHub Action. You have less control over:

• Trigger rules
• Output schema
• Deduplication behavior
• Review gating
• Model/provider substitution
• Combining the review with other CI signals
• Custom publishing logic

For many teams, that may be fine. But if the long-term goal is a larger AI-assisted CI system, this approach may become only one review source rather than the central orchestration layer.

Option 2: Repository-Owned GitHub Action Using Codex

The second approach used a GitHub Action.

The GitHub Action does not use a separate definition of what good review means. It uses the same AGENTS.md and shared .github/review/code-review.md policy as hosted Codex. The Action prompt explicitly tells Codex to read AGENTS.md, which routes it to that common policy.

On top of that shared layer, the repository owns four Action-specific files:

• .github/workflows/codex-pr-review.yml: trigger, permissions, execution, and publishing jobs.
• .github/codex/prompts/pr-review.md: Action-specific review procedure and output contract.
• .github/codex/schemas/pr-review.schema.json: machine-readable response schema.
• .github/codex/scripts/publish-inline-review.cjs: diff anchoring, deduplication, and GitHub review publication.

This separation is deliberate. The shared code-review.md file defines review priorities, repository invariants, and the finding threshold. The Action-specific pr-review.md file defines how this particular automated run should execute and how it must structure its output. It does not duplicate the review policy.

The workflow is label-gated. A pull request is reviewed only when the codex-action-review label is present. It also runs again when a new commit is pushed to the PR branch, as long as the label remains present.

The action asks Codex to produce structured JSON with sections such as:

• summary
• findings
• validation
• coverage
• uncertainty

That structure is not mainly for humans. It is an execution contract between the reviewer and the publishing script. The script needs predictable fields so it can decide which findings can become inline comments and which ones should fall back to a summary.

The GitHub Action then publishes review comments as github-actions[bot] on the same PR #4.

The repository-owned Action produced native inline comments after we built the publishing layer.

What We Had to Build

This approach required the most custom work.

To make the review useful, we had to build:

• A review prompt with strict output expectations
• A JSON schema for review results
• Patch parsing to understand changed lines
• Inline comment publishing through the GitHub API
• Fallback summary behavior for findings that cannot be anchored to a changed line
• Fingerprinting to avoid duplicate comments on repeated runs
• A safer permission model where the trusted publisher code comes from the base branch

This is the biggest practical difference between a coding agent and a code review product. The model can identify findings, but the product workflow around those findings still has to exist somewhere.

The Fingerprint Problem

Fingerprinting is specific to the repository-owned Action path.

If a workflow runs every time a commit is pushed, the reviewer may find the same issue more than once. Without deduplication, the PR gets noisy. The same comment can be posted repeatedly.

The fingerprint solves that by creating a stable identity for a finding, usually based on fields such as:

• File path
• Line number
• Rule or issue title
• Finding body or normalized explanation

Before posting a new comment, the publisher checks whether an equivalent finding has already been posted. If yes, it avoids posting a duplicate.

Dedicated review products usually handle this operational detail internally. In the GitHub Action approach, the repository owner has to implement it.

What Worked Well

The GitHub Action found both controlled defects and posted inline comments.

It also gave the most control:

• The model can be swapped.
• The prompt can be changed.
• The trigger policy can be changed.
• The output schema is owned by the repository.
• The review can later be combined with static-analysis, security, or test output.

This is the strongest architecture if the goal is a scalable, model-agnostic AI review pipeline.

What You Give Up

You own the operational burden.

Inline comments, deduplication, schema validation, retries, permissions, security hardening, and future maintenance are all your responsibility.

If you only want PR review, this may be too much. If you want an AI-assisted CI orchestration layer, this is probably the most extensible foundation.

Option 3: CodeRabbit

The third approach used CodeRabbit, a dedicated AI code review product.

For this experiment, CodeRabbit was configured with .coderabbit.yaml.

The configuration used:

• The assertive review profile
• Code guidelines pointing to .github/review/code-review.md
• Path instructions for backend routers and services

The CodeRabbit configuration was first merged independently in PR #5. The clean CodeRabbit test was PR #6. It was created as a non-draft PR, with the configuration already present on main, and without the competing Codex Action label.

CodeRabbit posted inline comments as coderabbitai[bot].

CodeRabbit produced native inline comments in the clean, non-draft experiment.

What Worked Well

Under clean conditions, CodeRabbit found both seeded defects:

• The authorization bypass in backend/app/routers/nameboard.py
• The incorrect registration-number mapping in backend/app/services/review_service.py

The product experience is clearly stronger than a bare GitHub Action. CodeRabbit provides more than model output:

• Inline comments
• PR walkthroughs and summaries
• Review status
• Pre-merge checks
• Configuration through YAML
• Code guidelines
• Path instructions
• Knowledge-base behavior
• Learnings
• Autofix and docstring generation depending on plan
• Unit test generation on higher plans
• Analytics and reports
• IDE and CLI review options
• Integrations such as Jira and Linear
• Linter and SAST tool support

That is the real value of a dedicated code review product. It is not just that it has a model. It packages the operational workflow around the model.

The Inconsistent Run

Before the clean PR #6 test, we had the noisier PR #4.

That PR was draft at one point, had hosted Codex comments, GitHub Action comments, manual CodeRabbit review commands, and CodeRabbit configuration changes inside the same PR.

In that noisy run, CodeRabbit selected the relevant files but did not publish the two expected actionable inline comments. Later, a CodeRabbit chat response acknowledged that the two issues should have been flagged.

I would not treat that as proof that draft PRs are unsupported or that CodeRabbit cannot find these defects. The cleaner conclusion is more limited:

CodeRabbit’s behavior was inconsistent in a noisy mixed-bot/draft/config-changing experiment, but it worked correctly in a clean non-draft PR with configuration already present on main.

That distinction matters. The blog should not overclaim from a messy experiment.

Feature Comparison

This was a controlled comparison, not a complete market survey. GitHub Copilot Code Review, Cursor Bugbot, Greptile, Qodo, and other review products may be worth evaluating, but I did not include tools that I had not tested on the same defects.

Cost Comparison

The pricing models differ, so the right comparison is not just “which one is cheaper per review.” The better comparison is how predictable the cost is and what operational work is included.

Hosted Codex Cloud

Hosted Codex review cost depends on the user’s ChatGPT/Codex plan and Codex credit model. It is not exposed as a simple repository-level per-PR API bill. OpenAI’s current Codex pricing documentation includes cloud-based automatic code review in supported ChatGPT plans, while API-key usage is separately token-billed and does not include the hosted GitHub code-review feature.

For an individual or small team already using Codex, this can be the lowest-friction option because it fits into the existing product experience. For a formal engineering cost model, it is harder to forecast per PR than an API-backed GitHub Action.

GitHub Action + Codex

The GitHub Action path has two costs:

1. GitHub Actions runner minutes
2. Model API usage

Runner cost is usually not the dominant factor for a small review job. The model cost is the meaningful part.

For a concrete API-cost reference, the table below uses the public GPT-5-Codex rate of $1.25 per 1M input tokens and $10.00 per 1M output tokens, with cached input at $0.125 per 1M tokens. The model page now marks GPT-5-Codex as deprecated, so these numbers should be read as a reproducible reference calculation, not as a recommendation to start a new production workflow on that model.

The experiment workflow did not pin a model, and the estimates below are not a measurement of its actual bill. A production workflow should explicitly choose an available model and recalculate with that model’s current prices.

The cost per review depends on:

• Size of the diff
• Amount of repository context included
• Prompt length
• Whether cached input applies
• Number and length of findings
• Whether the workflow reruns on every push

For planning, here is a rough uncached model-cost estimate:

These calculations assume uncached input and use the GPT-5-Codex API token prices above. Actual cost can be lower with prompt caching and higher if the workflow loads excessive repository context or reruns aggressively.

At a monthly level, the model-cost side looks like this:

GitHub Actions runner cost is additional, but for a lightweight Linux review workflow it is usually smaller than the model cost unless the workflow is slow or run at very high volume.

This approach can be very cost-efficient if reviews are scoped carefully. But if the workflow loads too much context or reruns aggressively on every commit, cost and PR noise can grow quickly.

CodeRabbit

CodeRabbit is primarily seat-based, not line-of-code based.

Public pricing lists:

• Pro: $24/mo/user billed annually
• Pro Plus: $48/mo/user billed annually

The docs describe per-developer rolling review limits:

• Pro: 5 PR reviews per developer per hour
• Pro+: 10 PR reviews per developer per hour

The current docs list a limit of 300 files per review for both Pro and Pro+.

Each review run consumes review allowance, including automatic incremental reviews after pushes and manual review commands such as @coderabbitai review or @coderabbitai full review.

This means CodeRabbit is not priced like “X cents per line reviewed.” You are paying for a productized review workflow, plus plan limits and optional usage-based expansion.

For a five-developer team, the annual-billing subscription comparison is:

So the GitHub Action path can be materially cheaper in direct model spend. But that is not the full cost. The engineering time to build and maintain inline publishing, deduplication, schema validation, security hardening, retries, and future integrations is real.

CodeRabbit costs more than the raw model calls, but it includes the product workflow. That is the main economic tradeoff.

Configuration Lesson: Keep One Review Policy

The best scalable pattern is not to duplicate long instructions across every tool.

I would organize the repository around four linked layers:

• AGENTS.md
• .github/review/code-review.md
• .github/codex/prompts/pr-review.md
• .coderabbit.yaml

The responsibilities should be different:

• AGENTS.md: tells coding agents and hosted Codex where the project guidance and review policy live.
• The shared review policy contains the durable, provider-neutral priorities and finding threshold.
• The Action prompt contains the GitHub Action execution contract, including JSON output requirements.
• The CodeRabbit YAML contains CodeRabbit-specific configuration, review profile, code-guideline references, and short path instructions.

The important point is that the CodeRabbit YAML should not become a second full review policy. But I also would not remove path instructions entirely. In the successful CodeRabbit run, its inline comments cited path instructions as the source. So the better approach is to keep path instructions short and targeted while keeping the broader policy in the shared review-policy file.

My Recommendation

If I were adopting this in a real engineering workflow, I would not choose only one answer for all teams.

For a small team that wants quick value, I would start with hosted Codex Cloud review or CodeRabbit.

For a team that wants a polished productized PR review experience, I would evaluate CodeRabbit seriously. Its value is not only the model. Its value is the review workflow around the model.

For a platform team building a broader AI-assisted CI system, I would invest in the GitHub Action approach. It is more work, but it becomes the orchestration point. The same pipeline can later combine:

• AI code review
• Static-analysis output
• Security scan output
• Test failures
• Coverage changes
• Dependency risk
• Runtime or dynamic-analysis signals

Then AI can do what it is better at: interpret multiple signals, identify risk patterns, explain impact, and help the human reviewer make a better decision.

What I Would Not Do

I would not ask a generic coding agent to “review this PR” with no policy, no structured output, and no integration plan.

That may produce useful comments sometimes, but it is not a CI/CD architecture.

For AI review to work inside CI, it needs:

• A clear review role
• Repository-specific guidance
• A defect threshold
• Inline comments or structured output
• Deduplication
• A trigger model
• A human review handoff
• Cost controls
• A way to evolve the policy over time

The model is only one part of the system.

Conclusion

This experiment changed how I think about AI code review.

A dedicated code review product is not necessarily better because it has a different model. A repository-owned coding agent workflow is not automatically better because it is more flexible.

The real tradeoff is operational.

Hosted Codex Cloud gives a low-friction review path. A GitHub Action gives maximum control and model flexibility, but requires custom engineering. CodeRabbit gives a productized review workflow, but the internal decision-making is less transparent and the buyer does not choose the exact model stack.

For now, my preferred direction for a scalable CI architecture is:

1. Keep review policy in the repository.
2. Use productized review tools where they reduce operational burden.
3. Use a repository-owned action when orchestration, model flexibility, and CI integration matter.
4. Treat AI as a reviewer and interpreter, not as a replacement for deterministic CI tools.

That fourth point is the bridge to the next article: how deterministic static analysis, security checks, and tests can produce evidence, and how AI can turn that evidence into a concise report for the final human reviewer. In that larger design, AI code review becomes one input to the decision rather than the decision itself.

References

• CodeRabbit plans and limits: docs.coderabbit.ai/management/plans
• CodeRabbit pricing: coderabbit.ai/pricing
• OpenAI GPT-5-Codex API pricing: developers.openai.com/api/docs/models/gpt-5-codex
• Codex pricing overview: chatgpt.com/codex/pricing
• Experiment repository: github.com/smakam/doctor-parser
• Setup PR: PR #1 — scalable Codex review integration experiment
• Inline publisher PR: PR #3 — publish Action findings as inline reviews
• Mixed hosted/Action experiment: PR #4 — controlled inline-review experiment
• CodeRabbit configuration: PR #5 — add CodeRabbit configuration
• Clean CodeRabbit experiment: PR #6 — CodeRabbit-only review experiment