Category Archives: AI

Multi-Agent Design Patterns: When Do They Actually Help? Three small diagrams: one agent with tools; a coordinator with three sub-agents; a generator, reviewer and correction flow.

Multi-Agent Design Patterns: When Do They Actually Help?

AI, , , , ,

Objective

The goal of this post is to show, with real numbers from a working system, how to decide whether an agentic system should be a single agent or multiple agents working together.

I’ve read about cases where multi-agent systems helped, for example:

  • Anthropic’s research system: a lead agent sends several research agents to investigate different parts of a question in parallel, then combines what they find.
  • Cognition’s reviewer pattern: a second agent reviews the first agent’s work and sends back what needs fixing.
  • LangChain’s language comparison: to compare Python, JavaScript and Rust for web development, a separate agent researches each language and a coordinator combines the results.
  • OpenAI’s Navier–Stokes proof: thousands of agents explored different approaches to a long-open math problem in parallel, sharing their best findings across groups.

I wanted to know whether any of these patterns would help my own marketplace support agent: an AI agent that answers support staff’s questions about orders, payouts, refunds and policies in a small simulated marketplace, using read-only SQL and policy search. More details are in the use case below. This post walks through the patterns, how I looked for one that fits a support agent, the two designs I tested, and what I found.

On my agent:

  • Splitting the work across seller agents did not help. A coordinator agent with three seller agents gave the same answers as one agent with parallel tools, but took longer and cost about 2.5× more.
  • A reviewer agent did help, but only when it was a stronger model. GPT-5.4 reviewing GPT-5 nano’s answers removed every incorrect answer at about half the cost of running GPT-5.4 alone. When nano reviewed nano, nothing improved.

The lesson: As per my understanding, multi-agent helps in two situations below, there can be more.

  • The work splits into independent pieces that can run in parallel, each with its own context. When the steps are mostly sequential, or one query already covers the work, extra agents don’t help.
  • A second agent can see what the first one can’t.

Agent Design Patterns

Before comparing design patterns, it helps to ask one question: who controls the next step, the code or the model?

  • Workflow: in a workflow, code defines the path.
  • Agent: an agent’s model decides the next step, such as which tool to call.
  • Combinations: real systems often combine code-controlled and model-controlled steps. For example, code fixes the order of the steps, while agents decide how to carry out each one.

More model calls, different models or parallel tool calls don’t by themselves make a system multi-agent.

These are the patterns I considered:

PatternTypeHow it flowsExample
Workflow (chaining, routing, parallel steps)WorkflowCode defines each stepMy model-routing posts: classify the question, then send it to a cheap or strong model
Single agent with toolsAgentOne model loop decides which tool to call nextMy support agent: catalog → SQL → policy search → answer
Coordinator with sub-agentsMulti-agentEach agent works autonomously: a coordinator agent splits the task, sub-agents work in parallel in their own contexts, and the coordinator combines the resultsAnthropic’s research system; LangChain’s language comparison
Generator–evaluatorWorkflow + agentsCode fixes the order (draft → review → revise if needed); a generator agent drafts and an evaluator agent checksCognition’s reviewer; Anthropic’s verification subagent
HandoffWorkflow + agentsCode defines which specialist agents exist and who can hand off to whom; an agent decides when to hand the conversation over, and the specialist takes overA triage agent hands a refund request to a refunds agent, which continues with the user
Independent agents across systemsMulti-agent across organizationsAgents owned by different organizations exchange tasks and results over a protocol such as A2AA marketplace’s support agent asks a payment provider’s agent to investigate a failed payout

Why add agents at all? Anthropic’s January 2026 post on when to use multi-agent systems gives three reasons:

  • context protection: keep irrelevant detail out of an agent’s context
  • parallelization: investigate independent parts at the same time
  • specialization: focused tools and prompts

It also warns that multi-agent systems typically use 3–10× more tokens, and advises starting with the simplest approach that works.

Three examples show when splitting pays off:

  • Anthropic’s research system: a lead agent breaks a research question into parts, several research agents search in parallel, each in its own context, and the lead agent combines their findings.
  • OpenAI’s Navier–Stokes proof: on the order of 10,000 agents worked in groups, each trying a different variant or approach, and Codex shared the best findings across groups. They reached a proposed proof of the 90-year-old Millennium Prize problem in 88 hours, using about 130 billion output tokens, and a separate Lean step verified it. Mathematicians have not yet accepted it.
  • LangChain’s language comparison: for “Compare Python, JavaScript, and Rust for web development”, a coordinator sends each language to its own agent, which researches it with about 2,000 tokens of language-specific material. The coordinator then combines the three results.
    • Separate agents: 5 model calls and about 9K tokens.
    • Handoffs: passing control from one agent to the next takes 7+ calls and about 14K tokens, because the steps run one after another.
    • One agent holding all three languages’ material: about 15K tokens.

LangChain credits the 67% fewer tokens to “context isolation”.

All three have the same shape: independent pieces of work, each needing its own context, combined at the end. Cognition’s reviewer is a different idea: a second agent with fresh context checks the first agent’s work.

Not everyone finds that more agents help. Google’s research on scaling agent systems finds the benefit depends on how well the task decomposes, and a 2026 paper found single agents outperform multi-agent systems on multi-hop reasoning when both get the same thinking budget.

The Use Case

The agent is an internal customer support agent I built over a small simulated marketplace with buyers, sellers, orders, shipments, seller payouts, returns, refunds and versioned policies. Support staff ask it questions in plain English. It answers by calling read-only tools (a data catalog, SQL over the business data, order history and a policy-document search) and cites the evidence it used. It runs on the OpenAI Agents SDK behind a LiteLLM gateway.

I use two models: GPT-5 nano, the lower-cost model, and GPT-5.4, the stronger one. More background on the agent is in my model routing posts.

Looking for a Multi-Agent Fit

My first idea was a SQL agent and a RAG agent: one for the business data, one for the policy documents. It didn’t make sense.

  • The two are mostly sequential. The records SQL finds decide which policy to look up: which payout, which date, which return.
  • Combining contexts adds complexity. Splitting them means passing one agent’s findings to the other, then combining both into one answer, with nothing gained.
  • SQL and documents are tools, not separate jobs. One agent can use both.

The second idea looked more promising: one agent per seller. When payouts don’t go through, each seller can have different reasons: an item not yet delivered, a hold that hasn’t passed, an active return, a failed provider attempt. Investigating each seller separately looked like independent branches, the parallelization case. That became the first experiment.

The second experiment came from a different observation. In my earlier evaluations, nano’s wrong answers often had the right evidence available but drew the wrong conclusion. A second agent checking the work seemed worth testing.

Experiment 1: A Coordinator With Seller Agents

The task was a single investigation: “Investigate all sellers’ payouts, explain supported reasons, include paid and zero-due controls, and summarize unpaid totals by seller.” A fixed test dataset had 3 sellers and 11 payouts covering seven outcomes:

  • not delivered
  • under the 24-hour hold
  • active return
  • eligible but not yet released
  • failed provider attempt
  • fully refunded (nothing due)
  • paid

I compared two designs, both using GPT-5.4 and the same five read-only tools:

  • One agent with parallel tool calls allowed. This is not multi-agent: it’s a single agent that can run several tool calls at the same time.
  • A coordinator agent with three seller agents. Each seller agent has its own tool loop and context. The coordinator delegates, then combines their findings.
Two designs side by side. Left: one support agent, a single model loop that chooses the next tool, calls read-only tools: catalog, SQL for all payouts, ten order histories at once, then policy search, and returns the answer. Right: a coordinator agent receives the question, delegates to three seller agents that each have their own tools and context and return findings, then combines the findings into the answer.
The same investigation, done by one agent with parallel tools or by a coordinator with three seller agents.

Each design ran three times, in rotated order. An LLM reviewer checked every answer against the expected payouts, sellers and amounts. Here are the averages; one coordinator run did not complete, and its recorded usage is included.

DesignMean timeMean cost (cache-adjusted)Mean model requests
One agent, parallel tools82.0 s$0.1626.0
Coordinator + 3 seller agents108.1 s$0.40119.7

The answers were equally right on the business facts: every payout covered, the right seller and order, correct amounts and totals. The difference was time and cost. The coordinator design used about 3× the model requests and cost about 2.5× more.

The reasons are visible in the traces:

  • One query covered every seller. The single agent found all the payouts with one SQL query, so there was no heavy independent branch to hand off. Each seller’s investigation was small.
  • The single agent already worked in parallel. After finding the order IDs, it fetched 10 order histories at once in two of the three runs. Parallel tool calls gave concurrency without extra agents.
  • Sub-agents add overhead: each one repeats the instructions and catalog lookup, then packages its findings and evidence for the coordinator, which then has to combine them.

This is the opposite of LangChain’s language example. There, each language needs its own research and its own context; here, one query already held everything.

The limits matter. This was one task, three runs and three sellers, all under a common payout policy, and I didn’t give both designs equal budgets. With many more sellers, or seller-specific policies that need real independent investigation, sub-agents might pay off. Not shown here is the honest conclusion, not “sub-agents never help”.

Experiment 2: The Generator–Evaluator Pattern

The second design adds a reviewer. Anthropic calls this pattern evaluator-optimizer; I’ll call it generator–evaluator.

  1. A generator agent (nano) drafts the answer with tools.
  2. A reviewer agent sees the question, the draft and the full evidence the generator used. It has no answer key, and in this experiment no tools. It returns “no changes needed” or specific feedback.
  3. If the reviewer flags problems, a correction agent (nano) revises the answer once, with tools available.

The application controls the order of the steps; the generator and correction agents run their own tool loops.

A left-to-right flow. A generator agent (GPT-5 nano) drafts the answer with tools and passes it to a reviewer agent (nano or GPT-5.4), which checks the draft against the evidence. If no changes are needed, the draft becomes the final answer. If changes are needed, feedback goes to a correction agent (GPT-5 nano), which makes one correction with tools and produces the final answer.
The generator–evaluator flow: draft, review, and at most one correction.

I ran it on the same 18 support cases I use for regression testing: order lookups, current and historical policy, payout eligibility, refund histories, reconciliation, an ambiguous name and a request the agent must refuse. Two cases are short follow-up conversations, giving 20 answers in total. An LLM reviewer graded every final answer against the expected results, separately from the reviewer agent’s own verdict. These grades are not human-adjudicated.

How it unfolded. I first ran it on 8 hard cases, with GPT-5.4 reviewing nano’s answers. Results moved from 3 correct / 3 partial / 2 incorrect to 5 / 3 / 0. It looked as if the pattern worked.

But was it the pattern, or a stronger model doing the reviewing? So I tested nano reviewing nano on the same 8 cases. It asked for no changes, and nothing improved. Then I ran both reviewers across all 18 cases:

DesignCorrect / partial / incorrectEstimated cost
One agent: nano11 / 4 / 3$0.037
Nano generator + nano reviewer11 / 4 / 3$0.053
Nano generator + GPT-5.4 reviewer14 / 4 / 0$0.582
One agent: GPT-5.417 / 1 / 0$1.210

Costs are model-usage estimates at uncached list prices, including nano’s drafts and the review and correction calls.

Scatter chart of cost against fully correct cases for the four designs. One agent nano: 11 of 18 at $0.037. Nano generator plus nano reviewer: 11 of 18 at $0.053, marked as dominated. Nano generator plus GPT-5.4 reviewer: 14 of 18 at $0.58. One agent GPT-5.4: 17 of 18 at $1.21. A line connects the three non-dominated designs.
Same-model review added cost without changing results. A stronger reviewer moved nano most of the way toward GPT-5.4 at about half its cost.

What the results show:

  • A reviewer can only catch what it can see. Across all 18 cases, nano as reviewer asked for three corrections, and none changed a grade. It shares the generator’s blind spots.
  • The stronger reviewer removed all three incorrect answers. Three answers became correct and one improved from incorrect to partial. The total came in about 52% below running GPT-5.4 alone, with three fewer fully correct answers.
  • Detection isn’t correction. GPT-5.4 asked for 13 corrections, and 4 answers improved. Several times nano was told what evidence was missing and still didn’t fetch it.
  • The gain comes from the reviewer’s capability. The pattern is how you spend that capability selectively: GPT-5.4 reads and judges nano’s work instead of doing all of it.

On a cost-versus-accuracy chart, nano reviewing nano is dominated: the same accuracy as nano alone, at higher cost. The useful choices are nano alone, nano with a GPT-5.4 reviewer, and GPT-5.4 alone. Which one is right depends on your accuracy bar and budget.

The limits: one pass over 18 known cases, and LLM-reviewer grades. This doesn’t show that self-review never works; it shows it didn’t work here.

When Multiple Agents Make Sense Across Systems

The experiments above looked at splitting one application’s work across agents. Multiple agents also make sense when the work crosses a boundary between systems, especially systems owned by different companies. The marketplace and a payment provider are two systems run by two different companies. The marketplace’s support agent could ask the provider’s agent to investigate a failed payout. Each side keeps its own tools, data and permissions, and they exchange only the task and the result.

That’s what the A2A (Agent2Agent) protocol is for: delegating work to an agent you don’t control. Google introduced it, and it’s now an open project under the Linux Foundation. Inside your own system, tools exposed through function calls or MCP are usually enough. I haven’t implemented A2A here, so I’m not claiming a measured benefit.

Lessons Learned

  • Look for independent branches, not tools. SQL and document search were mostly sequential, so separate agents would only have added a context-merging step.
  • One good query can beat a team. When a single bulk query covered every seller, sub-agents added time, cost and model calls without better answers.
  • Parallel tools give you concurrency without extra agents. The single agent fetched ten order histories at once, though it still had to find the orders first.
  • A reviewer is only as good as what it can see. Same-model review found nothing. A stronger reviewer removed every incorrect answer, which makes review a way to spend a strong model only where it matters.
  • Detection isn’t correction. Specific feedback helped most, but the correction agent didn’t always fetch the evidence it was told was missing.

A Checklist: Before You Add Multiple Agents

  1. Can one agent with the right tools, and parallel tool calls, do it?
  2. Are each agent’s task and context separate and isolated, or would the agents need each other’s details?
  3. Can the agents’ work run in parallel, or is it mostly sequential?
  4. Is one agent’s context actually overloaded? Measure it.
  5. For a reviewer: is it more capable than the generator on these errors? Test same-model review as a control.
  6. Compare against single-agent baselines on accuracy, cost, latency and failures.
  7. Across system boundaries, such as another team or another company, consider independent agents and A2A.

Limits

  • Both experiments are small: one investigation task with three runs, and 18 known cases with one pass.
  • Grades come from an LLM reviewer against expected answers, not human adjudication.
  • Costs are model-usage estimates, not invoices.

Summary

The lesson is simple: add multiple agents only for a reason you can name and measure. On my support agent, a coordinator with seller agents gave the same answers as one agent with parallel tools, at about 2.5× the cost, because one query already covered every seller. A reviewer agent helped only when it was stronger than the generator: GPT-5.4 reviewing nano removed every incorrect answer at about half the cost of GPT-5.4 alone, while nano reviewing nano changed nothing. Multi-agent pays off when the work splits into independent pieces that can run in parallel, each with its own context, or when a second agent can see what the first one can’t.

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude Code, which also helped me research the multi-agent patterns and plan the experiments. The experiments were built and run with GPT-6 Astra, which also graded the answers against the expected results.

References

Patterns and Evidence

Earlier Posts

Who pays the writer when the reader is an agent? A writer's page sends content to an AI agent, with a dashed return arrow and a dollar coin asking whether the agent pays the writer.

Who Pays the Writer When the Reader Is an Agent?

AI, , , , , , , , ,

Objective

The goal of this post is to show, with data from Cloudflare and TollBit, what is already happening to the web’s content economy now that AI agents do much of the reading for people. Today, AI companies pay a handful of large publishers through private licensing deals, while most other creators get little or nothing. I walk through why the old bargain broke, what the industry is building to replace it, why micropayments might finally work this time, and what still has to happen for creators to keep a reason to publish on the open web.

The Bargain That Built the Web

For about 25 years, the open web ran on an unwritten deal. You published something, a search engine crawled it, and in return the search engine sent you readers. What you did with those readers was up to you. Some creators showed ads. Some put the good parts behind a paywall and converted readers into subscribers. Many simply wanted to be read: to build a reputation, find customers, or share what they knew.

All three payoffs depended on one thing: a human visiting the page. The crawl was the price; the visit was the payment.

Two diagrams. Left: creator, search engine and reader in a loop where links bring visits back to the creator. Right: creator feeds an AI agent, the agent answers the reader directly, and only a dashed "rare clicks" arrow returns to the creator.
In the search web, the visit paid the creator. In the agent web, the answer stays with the agent, and so do the ads.

How the Bargain Broke

When people ask ChatGPT, Claude, Perplexity or Google’s AI Overviews instead of clicking through search results, the crawl still happens, but the visit mostly doesn’t.

Cloudflare sees a large share of web traffic and measures this directly as a crawl-to-referral ratio: how many pages a platform fetches for every visitor it sends back. In June 2025, Google crawled roughly 14 pages per referral, OpenAI about 1,700, and Anthropic about 73,000.

Bar chart on a log scale showing pages crawled per visitor referred in June 2025: Google 14 to 1, OpenAI 1,700 to 1, Anthropic 73,000 to 1.
Search engines still send visitors back. AI platforms mostly don’t.

These numbers move a lot, and that matters for how you read them. Anthropic’s ratio was around 286,000:1 in January 2025 and fell to about 38,000:1 by July, after Claude added web search with clickable citations. In the same Cloudflare sample, Mistral sent back more referrals than it made crawls. The ratio depends heavily on whether a platform’s crawling is for training, which never sends anyone back, or for answering live questions with links.

TollBit measures the same problem from the publisher side, using the sites that run its analytics. At the end of 2024, it found click-through rates from AI chatbots were 95.7% lower than from traditional Google search. Through 2025 it kept getting worse.

Line chart of click-through rate from AI apps in 2025. Sites with AI licensing deals fell from 8.8% in Q1 to 1.33% in Q4. All TollBit partner sites fell from 0.8% in Q2 to 0.27% in Q4.
Licensing deals bring money, not readers. Even publishers with deals saw click-through fall about 85% in a year.

The line I keep coming back to is the top one. It tracks how often people clicked through from AI apps to the sites of publishers that had signed AI licensing deals. Those publishers started 2025 with far better click-through than everyone else, at 8.8%, and by the end of the year it had fallen to 1.33%. A deal pays you, but it doesn’t bring the readers back.

Bots are also becoming a meaningful share of traffic in their own right. By Q4 2025, TollBit’s partner sites saw one AI bot visit for every 31 human visits, up from one in 50 earlier that year. One publisher, Digital Trends, measured 966 bot scrapes for every human referral.

The effect on a large publisher is concrete. People Inc. (formerly Dotdash Meredith) said Google search went from 54% of its traffic two years earlier to 24%, and blamed AI Overviews.

Ads didn’t disappear. They moved.

It is tempting to say the ad model is dying. It isn’t. It moved up to the agent layer. OpenAI launched ads in ChatGPT in early 2026, and less than 200 days later announced a $1 billion annualized run rate. That is a projection from one month of revenue, not money collected, but the direction is clear. Someone is still looking at ads. The difference is that the interface owner sells them, and nothing flows down to the people whose work made the answer possible.

This also explains why GEO (generative engine optimization) is a symptom rather than a fix. GEO helps your content get cited in an AI answer. That’s good for reach and reputation, but a citation without a click earns a publisher nothing. It is optimizing for a currency that doesn’t pay yet.

Not All Content Breaks the Same Way

The damage isn’t uniform. I find it useful to split the web into four kinds of content, because each one breaks differently.

Content that wants to be found. Product documentation, marketing pages, open-source READMEs. These sites never monetized the visit directly. Agents are simply a new distribution channel for them, and they will happily optimize for it.

Commerce. Here the ad was really a toll on discovery, and agents are already replacing it with a commission. OpenAI and Stripe’s Agentic Commerce Protocol (ACP) lets you buy from a merchant inside ChatGPT without visiting the merchant’s site. The merchant stays the seller of record, Stripe issues a payment token limited to that merchant and that cart total, and the merchant pays OpenAI a small fee on completed sales. The money flows from merchant to agent, because the agent brought the buyer. Commerce will mostly sort itself out: pay-per-click becomes pay-per-sale.

The one thing to watch in commerce is paid influence. If a merchant can pay to rank higher in an agent’s shortlist, and the agent buys on your behalf without showing you the alternatives, the agent quietly becomes the merchant’s salesperson. The line I’d draw is simple: influence that changes the deal for the buyer (a lower price, faster delivery) is competition; influence that changes the ranking without changing the deal is a conflict of interest. OpenAI says ads do not influence ChatGPT’s answers today. That is a promise worth holding it to.

User-generated content platforms. Reddit and Stack Overflow have sold their archives to AI companies; Reddit’s OpenAI deal is reportedly worth about $70 million a year. I don’t think bulk licensing is a durable model, because it undermines its own supply. Stack Overflow’s monthly questions fell from over 200,000 at the 2014 peak to under 50,000 by late 2025, and by December 2025 were back at 2008 levels. Over the same period its revenue reportedly doubled, partly from AI licensing. The platform is selling the stock of past answers while the flow of new ones dries up.

Content-as-product. News, analysis, reviews, reference, and the work of independent creators. This content was paid for by the visit, and there is no commission to replace it. This is where the hole is, and it is the focus of the rest of this post.

What Is Being Built

Over the past year, a fairly complete stack has emerged for letting a publisher charge an agent. Cloudflare is the most visible builder, but most of the pieces are open standards.

Four stacked layers: Declare (robots.txt, Content Signals, RSL), Identify (Web Bot Auth), Enforce (CDN or WAF edge, app middleware) and Pay (pay per crawl or per use, x402). A side panel notes that the edge can enforce access, but use after fetch depends only on law and contracts.
Each layer is weak alone. Together they make a working toll booth for honest agents.

Declare: what may be done with the content. Cloudflare’s Content Signals Policy extends robots.txt with three uses a site can allow or refuse separately: search (a traditional index with links), ai-input (using the content to generate an answer at query time) and ai-train (training a model). RSL (Really Simple Licensing) 1.0, released in December 2025, goes further and lets a publisher state licensing and payment terms in a machine-readable file. Over 1,500 media organizations support it, and it is managed by the RSL Collective, a nonprofit modeled on music royalty bodies like ASCAP and BMI. The catch: no major AI company has committed to honoring it.

Identify: which bot is asking. Today a crawler just claims an identity in a header, and anyone can fake it. Web Bot Auth, a Cloudflare-led IETF draft built on HTTP Message Signatures (RFC 9421), has bots sign their requests so the server can verify who is asking. It is still a draft, but Cloudflare, Anthropic and OpenAI moving to production together has made it the de facto standard. You can’t charge someone you can’t identify, so this is the foundation for everything else.

Enforce: allow, block or charge. Enforcement happens wherever the traffic flows. A CDN or firewall at the edge is the natural place, because it sees every request before your server does: it checks the signature, applies the rule for that bot, and returns HTTP 402 Payment Required if the bot hasn’t paid. Cloudflare says its network now returns over a billion 402 responses to AI crawlers per day. The same logic can also run as middleware inside the application, which is slower to reject unwanted traffic but knows things the edge doesn’t, such as whether an article is premium or breaking news. I expect most publishers to use both: the edge for coarse filtering and identity, the application for fine-grained pricing.

Pay: settle the charge. Cloudflare launched pay per crawl in July 2025, acting as merchant of record and letting a site set a flat price per request. In July 2026 it announced Pay Per Use, which pays publishers when their content actually creates value, for example when it appears in an AI answer, rather than when it is fetched. It also announced a Monetization Gateway that can charge for any web page, dataset, API or MCP tool, settling in stablecoins over x402, an open payment protocol that Coinbase created and handed to the Linux Foundation in 2026.

From September 15, 2026, Cloudflare also blocks “mixed-use” crawlers by default on ad-supported pages. That change is aimed squarely at Google, whose crawler collects for both search and AI Overviews in one pass. Until now, a publisher couldn’t refuse one without disappearing from the other.

Who pays whom

Once you look at real flows, there isn’t one payment model. There are three, and they differ in how far the user is from each payment.

Three columns on a spectrum. User's card: agent shops for you, the statement shows the merchant and the merchant knows the buyer. User-funded agent wallet: agent spends your top-up on small digital purchases, the seller sees only the agent. Agent operator's wallet: agent pays for its own inputs, you pay a subscription and the publisher sees only the agent.
In commerce the user still pays directly. In the content economy, the agent pays and the user stays anonymous to the seller.

With the user’s card, as in ChatGPT’s checkout today, the agent orchestrates but the money is yours: the merchant appears on your statement and knows who you are. With a user-funded agent wallet, you top up a balance and the agent spends it on small things on your behalf, like a paid article you want to read or a dataset for a research task. With the agent operator’s wallet, the agent pays for its own inputs, such as the 20 articles it reads to answer your question, and recovers the cost through your subscription. The content economy mostly lives in the last two, which is why it needs new rails rather than card networks.

Why Micropayments Could Work This Time

Paying a few cents per article has been tried since the 1990s and has failed every time. The usual explanation is fees, and fees were real: one recent analysis notes that traditional payment processors struggle with anything under about $0.50. But the deeper reason was named by Nick Szabo in the late 1990s. He argued that the binding cost of a micropayment isn’t technological, it’s mental: every tiny purchase forces a person to decide whether it’s worth it, and that decision costs more attention than the item is worth. So people chose subscriptions, or free content with ads.

An agent doesn’t have that cost. Given a budget and a policy, it can make thousands of small purchase decisions without fatigue. Szabo himself anticipated this. He suggested micropayments might work if a person could hand their budget and preferences to software that completes the purchasing for them. That is a reasonably precise description of an AI agent with a spending policy.

Stablecoins remove the other half of the problem. A card payment carries a fixed fee that swamps a one-cent charge. A stablecoin transfer on a fast network costs a tiny fraction of a cent (one source puts a Solana transfer at about $0.0001) and settles in seconds, globally, without each publisher needing a merchant account. That is why x402 settles in stablecoins. In the 30 days before the x402 Foundation’s July 2026 launch, the protocol processed 75.41 million transactions worth $24.24 million, an average of about $0.32 per transaction. These are mostly API and data payments, not article fees, but they show that machine-to-machine micropayments are already running at scale.

I don’t want to overstate this. Stablecoins don’t carry the chargeback protections people are used to on cards, regulation varies a lot by country, and for anything large the user’s card with a signed authorization will remain the better choice. But for the specific case of an agent buying a fraction of a cent’s worth of content, both of the old obstacles, the fee and the decision, are gone at the same time. That has not been true before.

The Hard Problems

Access is not use

robots.txt and Content Signals are declarations, not locks. A well-behaved bot chooses to comply; nothing stops one that doesn’t. TollBit found that AI scrapes bypassing robots.txt grew by over 40% in a single quarter, and in August 2025 Cloudflare removed Perplexity from its verified bot list for disguising its crawlers to evade blocks.

So enforcement comes from three places working together: the publisher declares, the edge enforces access, and the law covers use. But no protocol can see what happens after a page is fetched. Content can be cached and reused indefinitely, and training is the ultimate cache: one crawl, permanent value. That is why training is fought in court and settled with lump sums, while live answers are where per-request pricing can hold. It is also why the mixed-use crawler matters: if you allow a crawler for search, you can’t technically stop its owner using the same content for AI answers.

Who sets the price

Cloudflare’s Matthew Prince offered a useful picture: an AI model’s knowledge is a block of Swiss cheese. Content that fills a hole, such as new reporting, niche expertise or first-hand data, is worth far more than the hundredth article repeating what the model already knows. Pricing by new knowledge instead of traffic would reward originality over the clickbait the ad model encouraged.

But it is hard to measure. Only the AI company can see what its model already knows, the same article may fill a hole for one model and not another, and the first source captures nearly all the value. Supply and demand won’t fix this on their own: there are only a handful of buyers, only the buyer knows which content was useful, and content is trivially copyable. TollBit’s CEO put it bluntly: AI companies say they can’t pay every publisher the asking rate.

The toll booths are concentrating too. Every protocol here is open, but the practical position belongs to the few CDNs already in front of much of the web. Open protocols don’t guarantee an open market.

The toll booth is built, but few drive through

Cloudflare’s Pay Per Use launched with two partners and the Monetization Gateway is still a waitlist. Of the more than 7,000 publisher sites on TollBit’s network, nearly 20% are earning revenue from its bot paywall, ranging from hundreds to tens of thousands of dollars a month. The real money still moves through private deals like News Corp’s with OpenAI, reportedly worth more than $250 million over five years, which only large publishers can negotiate.

There are signs the open model can work. People Inc. said blocking non-paying bots brought almost everyone to the table, and it joined Microsoft’s pay-per-use marketplace. But for most creators, the traffic either turns around at the toll booth or passes on someone else’s negotiated pass.

What Needs to Happen

  • Separate crawlers by purpose, Google included. Most AI companies now use different crawlers for training, search and live user requests, so a publisher can treat each one differently. Google’s dual-purpose crawler is the largest exception, and until it splits, no publisher can refuse AI answers without losing search.
  • Separate enforcement from settlement. Enforcement has to sit where the traffic flows, which favors CDNs. Payment doesn’t. Publishers and AI companies should insist on open settlement through x402 or similar, so whoever controls access doesn’t automatically control the money.
  • Build a standard for usage reporting. Pay Per Use only works if AI companies report honestly when and how content was used. Today that depends on trust. This is the missing piece of the stack, and the one that makes the Swiss cheese idea measurable.
  • Make collective licensing the default for the long tail. An independent writer can’t negotiate with OpenAI. A collective like the RSL Collective can, the same way ASCAP lets songwriters get paid for radio play without counting every broadcast.
  • License flow, not stock, and share it with contributors. UGC platforms should sell ongoing access to fresh content rather than one-time dumps of their archives, and pass part of the revenue to the people producing it. Otherwise the contributors stop contributing.
  • Invest in spending-policy infrastructure. Users will only let agents spend on their behalf if setting, auditing and revoking limits is simple: a budget, a per-transaction cap, allowed categories, and a threshold above which the agent must ask. This is the layer Szabo was describing, and it gets far less attention than payment protocols.

Summary

The lesson is uncomfortable: the web’s old bargain can’t be bought back, so payment has to replace traffic. AI platforms crawl thousands of pages for every visitor they send back, click-through from AI apps fell to 0.27% by the end of 2025, and even publishers with licensing deals lost most of theirs. The pieces of a replacement exist: signed agent identity, machine-readable terms, 402 responses at the edge, and stablecoin micropayments that agents can make without the decision fatigue that killed micropayments before. But participation is uneven. Google hasn’t separated its crawlers, AI companies prefer private deals to open marketplaces, and small creators have no bargaining power. The rails are being built faster than the incentives, and creators who leave rarely come back. The real question is whether they will still be here when the market is ready.

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude.ai, which helped me research and check the sources, brainstorm the structure, draft the text and produce the figures.

References

Cloudflare

Standards and protocols

Data

Industry developments

Background

Model routing is three decisions, made on your own data: 1. Model pool (GPT-5 nano and GPT-5), 2. Classification (SIMPLE to nano, COMPLEX to GPT-5), 3. Scope (per conversation or per question).

Model Routing, Part 2: Choosing the Scope

AI, , , ,

Objective

In Part 1, I compared ways to choose which model should answer a support request. This post continues that experiment with a second decision: how long should that model choice last—a whole conversation or just one question? The small pilot illustrates a method, not a general recommendation.

I compared four policies on six new three-question conversations: always GPT-5 nano, always GPT-5, per-conversation routing and per-question routing. Nano and both routing policies got 4 of 6 conversations fully correct; GPT-5 got all six. Routing cost about nine times nano, without improving the fully-correct-conversation count. Neither scope clearly won.

The Agent and Classifier

The agent answers marketplace support questions using read-only SQL and policy search, through the OpenAI Agents SDK and LiteLLM, with Langfuse tracing. Part 1 compared classifiers on 18 cases. Here I reuse its domain-aware nano classifier: SIMPLE → nano, COMPLEX → GPT-5.

Three boxes: 1. Model pool (a fixed, approved set of models; ours are GPT-5 nano and GPT-5), 2. Classification (label each request, then map each label to a model; ours map SIMPLE to nano and COMPLEX to GPT-5), 3. Scope (per conversation, per question, per model call, named stage). Below, one box lists what a gateway such as LiteLLM can manage: allowed models, built-in classifiers, per-conversation selection through session affinity, per-question selection through user_turn and per-model-call selection through every_request. Another box lists what the application must supply: session identity, structured messages and named stages. A note says that in the conversation-granularity experiment the author's own runner did the classification, selection and pinning, and LiteLLM forwarded each request to the selected model.
Part 1 covered the model pool and classification. This post covers the third decision, scope.

My Starting Expectation

I expected per-conversation routing to preserve context and cache warmth, while per-question routing could save money on easy follow-ups. I also assumed switching would lose context or start cold. These were assumptions to test, not established properties.

Ways to Scope a Routing Decision

A gateway can manage most of these for you. LiteLLM’s documentation describes three built-in options:

  • Per conversation: session affinity pins the model chosen on the first turn of a session and skips reclassification on later turns.
  • Per question: classification_mode: user_turn classifies only new user questions, and carries that decision through the tool calls and continuation turns within the question.
  • Per model call: classification_mode: every_request (the default) classifies each request the agent sends to the gateway.

For any of these to work, the application supplies the session identity (LiteLLM reads a session_id from the request metadata) and well-structured messages, so the gateway can tell a new user question from a tool result.

Two more options need the application’s knowledge:

  • Named stages: a deliberate rule such as “nano gathers the evidence, GPT-5 writes the final answer”. The gateway can reclassify every call, but only the application knows which call is “gathering” and which is “writing”.
  • Escalation: start on nano and switch up when a check fails. This needs a reliable signal that an answer is wrong.

Here is what a single question looks like inside the agent:

Illustrative sequence for one question, "Why hasn't Cedar been paid for order-1?", in five steps: model call 1 decides to query payout records; a tool runs read-only SQL; model call 2 reads the rows and decides to search the payout policy; a tool searches the policy documents; model call 3 writes the answer with citations. Notes below explain that per-question routing uses one model for all three calls (LiteLLM user_turn), automatic per-call routing reclassifies each call (every_request), and deliberate stage selection is decided by the application.
An illustrative question. Automatic per-call routing reclassifies each call; deliberate stage selection needs the application to name the stages.

I tested per conversation and per question, against the two fixed models.

How I Tested It

What I implemented. My own runner did the classification, the model selection and the pinning. LiteLLM simply forwarded each request to the model the runner had explicitly selected. So these results test the two routing policies, not LiteLLM’s native session-affinity or user_turn implementations. Every internal model call for a question used the model chosen for that question.

The conversations. I wrote 6 new conversation scenarios of 3 questions each, with expected answers written in advance. They cover:

  • simple questions leading to hard ones, and the reverse
  • a clarification
  • a purchase that changes the data mid-conversation
  • a clock change that lets a payout hold pass
  • simple lookups only

The policies. Each scenario ran once under four policies:

  • always nano
  • always GPT-5
  • per-conversation routing
  • per-question routing, where the classifier also sees the earlier conversation

Six scenarios × three questions × four policies is 72 answers, but only 6 distinct scenarios, not 72 independent tests. I couldn’t reuse saved answers as in Part 1, because each answer shapes the next question’s context.

Grading. An LLM reviewer, GPT-6 Astra, reviewed every answer against the expected facts, and I adjudicated the borderline cases. Three borderline partial grades were flagged; I reviewed and accepted them. A conversation counts as fully correct only if all three of its answers are correct, because a support conversation with one bad answer is still a failed conversation.

One Conversation, Two Scopes

Here is one scenario, “simple lookup to payout investigation”, under the two routing policies:

#QuestionPer conversationPer question
1“For order-2, list the purchased product, quantity and seller.”nano (pinned for the conversation) ✅nano ✅
2“For that same order, explain why its seller has not received the payout and what must happen before release.”nano ⚠️ partialGPT-5 ✅
3“What exact amount is currently payable to that seller for this order, after the marketplace fee?”nano ✅ ₹1,618.20GPT-5 ✅ ₹1,618.20

The first question looked simple, so per-conversation routing pinned the whole conversation to nano, including the harder second question. Per-question routing classified question 2 as COMPLEX and sent it to GPT-5. The classifier’s reason: “Determining why a seller payout is not completed requires tracing allocations, payments, shipments, returns/refunds, and payout status, plus policy-based eligibility.”

The Results

The headline measure is conversations with all three answers correct. Costs are cache-adjusted usage estimates for these newly generated answers. They are reconstructed from the token and cached-token counts the provider reported, at fixed list rates.

PolicyConversations fully correctCache-adjusted usage estimate, 6 conversationsvs always GPT-5
Always nano4 / 6$0.014−94.9%
Always GPT-56 / 6$0.271—
Per conversation4 / 6$0.126−53.4%
Per question4 / 6$0.127−53.2%
Two bar charts titled "Same fully-correct-conversation count for nano and both routing policies; 6 conversation scenarios, one pass each". Left: conversations with all three answers correct: always nano 4 of 6, always GPT-5 6 of 6, per conversation 4 of 6, per question 4 of 6. Right: cache-adjusted usage estimate for six conversations: always nano $0.014, always GPT-5 $0.271, per conversation $0.126, per question $0.127.
Nano and both routing policies had the same fully-correct-conversation count; only GPT-5 alone got all six. Six scenarios, one pass each.
  • Nano and both routing policies each got 4 of 6 conversations fully correct; GPT-5 got 6 of 6.
  • Routing cost about nine times nano, and it did not improve the fully-correct-conversation count.
  • Routing did reduce incorrect individual answers. Out of 18 answers per policy, nano got 15 correct, 1 partial and 2 incorrect; both routing policies got 15 / 2 / 1. Fewer wrong answers did not produce more fully correct conversations.
  • Neither granularity established a compelling advantage. Per conversation and per question got the same conversation score at almost the same cost.

Both routing policies chose nano for 12 of the 18 questions and GPT-5 for 6, though not always the same questions. Per-question routing switched models four times across the six conversations.

Right vs Wrong: Two Conversations in Detail

1. Per-question routing caught the harder question. In the scenario above, question 2 asks why the seller hasn’t been paid. The correct answer is that payment is captured, the shipment is still pending, and the payout is held until delivery is confirmed and a 24-hour hold has passed.

  • Per-conversation routing (pinned to nano): said the payout “is currently held” until “delivery/fulfillment conditions are satisfied”. It left out the 24-hour hold, so it was graded partial.
  • Per-question routing (sent to GPT-5): “Policy requires delivery confirmation plus a 24-hour hold… No delivery confirmation is recorded yet.” Correct.

Routing the hard question to the stronger model helped here. But there is a twist: in the separate always-nano run, nano answered this same question correctly. Same model, same question, different run, different grade. One run is not proof.

2. When every question looks simple, no scope helps. This scenario was about a name that means two things: “Cedar” is both a seller (Cedar & Co.) and a product (Cedar Desk).

  • “Please list purchases associated with Cedar.” Nano correctly asked whether I meant the seller or the product.
  • “The seller Cedar & Co., not the desk product.” Nano invented a “today only” date filter nobody asked for and replied that there were no purchases. In fact there was one: order-1, with two Canvas Weekenders. Graded incorrect.
  • “Within those purchases, show only items that have not been delivered.” Nano answered “none”. That happens to be the right result, since the order had been delivered, but only because the previous answer had filtered everything out. Right answer, wrong reason, graded partial.

The classifier labeled all three questions SIMPLE, and on the surface they are: a list, a clarification, a filter. So both routing policies used nano throughout and made the same mistakes. GPT-5 alone got all three right. Per-question routing only helps if the classifier can tell that a question is risky, and follow-ups often look simpler than they are. Once a wrong answer enters the conversation, later answers build on it.

What It Costs

The pilot also recorded latency and caching. Full-price estimates are shown next to the cache-adjusted estimates so the effect of caching is visible.

PolicyCache-adjusted estimateFull-price estimateCached share of inputLatency p50 / p95Model switches
Always nano$0.014$0.02050.6%10.3 s / 20.2 s0
Always GPT-5$0.271$0.46064.0%13.2 s / 34.8 s0
Per conversation$0.126$0.20654.0%10.4 s / 33.1 s0
Per question$0.127$0.21957.9%13.4 s / 38.8 s4
  • Caching changes the estimate a lot. For always-GPT-5, the cache-adjusted usage estimate was about 40% below the full-price estimate. Neither figure is an invoice.
  • Report cache differences, don’t explain them away. Per-question routing switched models four times, and its cached share was 57.9% against 54.0% for per-conversation. I am not attributing that difference to model switching, or to anything else, because two things make it hard to interpret:
    • Provider caches were shared and could already be warm across runs.
    • One conversation was interrupted mid-run when my API credit ran out. It resumed after funding, which may have changed cache warmth.
  • Treat the latency figures as rough. p95 over 18 answers per policy is an indicator, not a production estimate.

What Happened to My Expectation

I expected per-conversation routing to win on context and caching, with per-question routing trading some of both for lower cost. Here is how each part held up:

  • Context: my assumption was wrong. A model switch doesn’t lose the conversation. The application passes the conversation history with every call, so the model answering question 2 sees question 1 and its answer, whichever model it is. What carries over from one question to the next is the content of earlier answers, including their mistakes, not the model.
  • Caching: I didn’t observe the penalty I expected. The per-question policy’s cached share was higher, not lower, and that held within each model too: nano 51.5% versus 47.8%, GPT-5 66.9% versus 61.6%. With shared cache warmth, the funding interruption and a single pass, this observation doesn’t tell me why.
  • Cost: effectively the same. $0.126 for per-conversation, $0.127 for per-question.
  • Accuracy: the same conversation score, with different mistakes. Per-question routing rescued one question the pinned conversation got partly wrong. In another conversation, neither scope helped, because every question looked simple.

So the pilot didn’t confirm my expectation, and it didn’t reverse it either. It was inconclusive. The useful part is seeing that the differences between the scopes were smaller and more situational than I assumed.

Pros and Cons of Each Scope

No scope is best in general. Each one trades something:

ScopeProsConsConsider it when
Per conversationOne classification per conversation; one model; simplest to run and explainThe first question decides for all later ones; a simple opener can leave a hard follow-up on nanoConversations tend to stay at one level of difficulty
Per questionAdapts when a conversation gets harder or easier; conversation history still carries overA classification on every question; follow-ups often look simpler than they are; more model switches to monitorConversations often mix quick lookups with investigations
Per model call (automatic)The gateway can reclassify every call with no application changesCalls within one question may land on different models; more classifications; the classifier sees the call, not the stageYou want fine-grained routing without writing stage logic
Named stages (application-defined)Can reserve the stronger model for the stage that needs it, such as writing the final answerThe application must identify the stages; the final stage often carries the most context, so it may not save muchFailures cluster in one stage, such as interpreting evidence
EscalationPays for the stronger model only when the cheaper one failsNeeds a reliable signal that an answer is wrong; fluent wrong answers slip through; slower when it escalatesYou have strong automatic checks on answers

A Checklist for Choosing Routing

Steps 1–4 are from Part 1; steps 5 and 6 come from this post.

  1. Shortlist models on benchmarks, domain fit, cost and latency, from approved providers that support your agent’s needs.
  2. Run evals on your golden set, with single questions and full conversations, to narrow the shortlist. Then fix the pool and pin the versions.
  3. Weigh cost savings against accuracy. If the cheaper model meets your accuracy bar, use it. If only the stronger one does and its cost is acceptable, use it. Route only when routing saves meaningful cost while keeping accuracy within your bar.
  4. Design the classification. Choose labels that matter for your traffic and map them to models. Start with keyword rules, then try semantic or LLM-based classifiers built from real user journeys.
  5. Choose the granularity on real conversations. Compare per conversation and per question (and per call or named stages if they fit your agent) on multi-question conversations, watching conversation-level correctness, cost, latency and caching.
  6. Validate and monitor. Re-test on fresh scenarios and repeat unstable cases before deciding, and rerun your evals whenever a model changes or retires.

Limits and What’s Next

This was a small pilot, and I would not treat it as a benchmark:

  • Sample size: 6 conversation scenarios of 3 questions, one pass per policy.
  • Grading: an LLM reviewer plus my adjudication of borderline cases.
  • Classifier: a prompt developed on known cases.
  • Implementation: my own runner, not the gateway’s native routing.
  • Caching: shared cache warmth and a funding interruption; the figures are observations, not causal proof.

The “different run, different grade” twist above shows how much one pass can vary. Next, I want to:

  • run more conversations, with repeated passes
  • repeat the test with LiteLLM’s native session affinity and user_turn
  • try named-stage routing, with nano gathering evidence and GPT-5 writing the answer
  • test escalation, once I have a reliable signal that an answer is wrong
  • run controlled cache tests

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude Code, which also helped me research the routing landscape and plan the experiments. The application and experiments were built and run with GPT-6 Astra, which also reviewed answers against the expected results.

References

This Series

Earlier Posts

Routing Tools

  • LiteLLM Auto Routing: session affinity, classification_mode (every_request and user_turn) and classifier configuration.
Model routing is three decisions, made on your own data: 1. Model pool (GPT-5 nano and GPT-5), 2. Classification (SIMPLE to nano, COMPLEX to GPT-5), 3. Scope (per conversation or per question).

Model Routing, Part 1: Choosing a Classifier

AI, , , , , ,

Objective

How do you choose a model router for your own application? I tested six classifiers on a marketplace support agent to compare cost against answer quality. This small study demonstrates the decision process, not a general ranking of products.

Both GPT-5 nano and GPT-5 answered 18 support cases: 16 single-turn and two two-turn cases, or 20 answer turns per model. The LLM classifier and JEV each selected fully correct answers for 88.9% of cases, versus 94.4% for GPT-5 alone and 50% for nano, at an estimated 55–58% of GPT-5’s cost.

Part 2 examines how long a routing decision should last across a conversation.

The Use Case

I built a simulated marketplace with buyers, sellers, orders, shipments, payouts, returns, refunds and versioned policies. Its internal support agent answers questions using read-only SQL and policy search, citing its evidence. It runs on the OpenAI Agents SDK, through LiteLLM, with Langfuse tracing; payments and business data are simulated.

The workload mixes quick lookups—“What products are in order-2?”—with investigations such as “Why hasn’t this seller been paid?” The latter can require payment records, delivery events, policy and a time calculation.

Routing Is Three Decisions

Model routing is not one setting. It is three decisions:

Three boxes: 1. Model pool (a fixed, approved set of models; ours are GPT-5 nano and GPT-5), 2. Classification (label each request, then map each label to a model; ours map SIMPLE to nano and COMPLEX to GPT-5), 3. Scope (per conversation, per question, per model call, named stage). Below, one box lists what a gateway such as LiteLLM can manage: allowed models, built-in classifiers, per-conversation selection through session affinity, per-question selection through user_turn and per-model-call selection through every_request. Another box lists what the application must supply: session identity, structured messages and named stages. A note says that in the conversation-granularity experiment the author's own runner did the classification, selection and pinning, and LiteLLM forwarded each request to the selected model.
Routing is three decisions. A gateway can manage most of them; the application supplies session identity and any named stages.
  1. Model pool: which approved models the router may choose between.
  2. Classification: how each request is labeled, and which model each label maps to.
  3. Scope: how long a routing decision lasts, such as a whole conversation, a single question or a single model call. That is Part 2.

Choosing and Fixing the Model Pool

Shortlist approved models on benchmarks, domain fit, cost and latency, then run evals on your own golden set. Candidates must support the agent’s tool calls and output formats. Cost per correct answer matters more than token price alone.

My evaluated pair was GPT-5 nano, the lower-cost candidate, and GPT-5, the stronger performer. Both use the same provider behind one gateway. I did not run public benchmarks myself; the choice rested on workload evals, cost and latency.

Keep routing within the evaluated, approved pool and pin model versions for reproducibility. Even OpenRouter Auto was restricted to these two candidates.

How I Checked the Quality

A router can only be judged as well as the grades behind it, so grading came first.

I built a golden set: cases with known business data behind them and expected answers written in advance. A case counts as fully correct only if every requested fact is right, in every turn of the case.

Grading combined human review and LLM review:

  • Calibration, earlier: in an earlier model comparison, I graded 36 answers by hand and had Claude Opus 5.5 grade them as an LLM judge. After clarifying the grading rules, the judge agreed with my grade on 31 / 36 (86%). That calibration applies to that judge on those answers.
  • This comparison: the nano grades are existing accepted grades. The GPT-5 answers were reviewed by an LLM reviewer, GPT-6 Astra, against the expected answers, and I adjudicated the borderline cases. The Claude calibration does not validate that reviewer; the adjudication is what anchors those grades.

Baselines: What Routing Has to Beat

First, the two models on their own over the 18 cases. The cases cover order lookups, policy questions (current and historical), payout eligibility, refund histories, incidents, reconciliation, an ambiguous name, and a request the agent must refuse.

ModelCases correct / partial / incorrectFully correctEstimated cost, all 18 cases
GPT-5 nano9 / 7 / 250.0%$0.023
GPT-517 / 1 / 094.4%$0.456

GPT-5 was far more accurate and cost about 20× more. That gap is what makes routing worth considering. The question routing has to answer is simple: how much cost can we save, and how much accuracy do we give up?

Classification: Labels Are a Design Choice

A classifier assigns a label to each request. A separate mapping turns that label into a model. Both are your choices.

  • Labels don’t have to be about difficulty. They can be task type (lookup, investigation, action request), business domain (payments, returns, fulfillment), risk, language, or a mix.
  • “Simple” doesn’t have to mean “cheapest”. A short but risky request such as “refund this order now” could be labeled simple and still be mapped to the strong model.
  • A pool can have more than two tiers.

For this study, I used the simplest useful setup for a two-model pool: SIMPLE → GPT-5 nano and COMPLEX → GPT-5, for the classifiers I configured. LiteLLM’s built-in tiers also include MEDIUM and REASONING; I mapped everything above SIMPLE to GPT-5. OpenRouter’s Auto Router is the exception: it does not use my labels at all (see below).

The Six Classifiers and How I Configured Them

Every rule, example and prompt below was written from the support-user journeys in my specification: the kinds of questions support staff actually ask. They were frozen before testing. The one exception is a separately versioned refinement of the semantic examples, which I made after seeing the first results. It is reported on its own and is not part of the results table.

How each one ran. For the first five, my own runner produced or collected the classification and chose the model. OpenRouter Auto made its own choice. None of this used a gateway’s automatic routing on live traffic.

  • Keyword rules, heuristic scorer and semantic similarity: LiteLLM’s native classification code, exercised separately. It was not routing live traffic through my gateway.
  • LLM classifier: GPT-5 nano called through my LiteLLM gateway.
  • JEV: called directly through OpenRouter’s Decisions API.
  • OpenRouter Auto: called through OpenRouter.

Keyword Rules

LiteLLM’s keyword rules map words to tiers without calling any model. My lists:

keyword_tier_rules:
  - tier: SIMPLE
    keywords: [list, show, quantity, stock, balance, price]
  - tier: COMPLEX
    keywords: [why, investigate, reconcile, history, policy, eligible, eligibility,
               refund, failed, incident, cause, "can you", "can the", "can we"]

Simple words signal a direct lookup. Complex words signal explanation, investigation, policy, eligibility, money movement, or a request to act. If both lists match, the higher tier wins. If nothing matches, the question falls back to the heuristic scorer below.

Heuristic Scorer

This is LiteLLM’s default classifier, and I used its defaults unchanged on purpose, to see how a generic heuristic behaves out of the box. It scores seven text signals that were built with software questions in mind:

  • code words (weight 0.30)
  • reasoning phrases like “step by step” (0.25)
  • technical terms (0.25)
  • length (0.10)
  • simple phrases like “what is” (0.05)
  • “first… then” patterns (0.03)
  • several question marks (0.02)

LiteLLM lets you add your own technical keywords, which is how you would adapt it to a domain.

Semantic Similarity

LiteLLM can also match questions by meaning. I wrote 14 example sentences, embedded with text-embedding-3-small. A question takes the label of its closest example if the similarity is at least 0.5; otherwise it falls back to the heuristic. A few of the examples:

  • SIMPLE (6 examples): “Show the products purchased in this order and their quantities.” · “What is the listed price of this product?” · “List the sellers and their current outstanding amounts.”
  • COMPLEX (8 examples): “Explain why the buyer paid but the merchant has not received the money.” · “Determine whether delivery and the waiting period permit a payout now.” · “Can you approve a refund and transfer the money on my behalf?”

LLM Classifier

Here a small model, GPT-5 nano, reads a routing prompt and returns SIMPLE or COMPLEX with a one-line reason. The prompt has five parts:

  1. Purpose: classify the evidence and reasoning a correct answer needs, “not sentence length or how easy the answer sounds”.
  2. The application and its tools: read-only SQL, policy search, event history.
  3. “What makes business questions deceptively difficult”: eight domain traps. Examples: “Buyer payment, seller payout and buyer refund are different money movements”; “Current payout/refund rows can hide failed attempts preceding a success”; “Recorded state, eligibility and successful execution differ.”
  4. Decision rules: SIMPLE only for direct, bounded lookups. COMPLEX for causal explanation, event reconstruction, policy applied to live data, time-boundary calculations, reconciliation or ambiguity. “A short question can be COMPLEX. A long factual list can be SIMPLE.”
  5. Generic data definitions: the application’s entities, relationships and metrics, without any test data.

A typical pair of decisions:

  • “For order-2, list the product, quantity and seller” → SIMPLE: “Direct bounded retrieval… No complex inference or policy.”
  • “Explain why its seller has not received the payout” → COMPLEX: “requires tracing allocations, payments, shipments, returns/refunds, and payout status, plus policy-based eligibility.”

JEV

JEV is TypeSafe’s decision model, which I wrote about in Right Model, Right Job. I gave it the same routing rubric, adapted to JEV’s typed API. The rubric text went in as the instructions of a typed SIMPLE/COMPLEX choice question, with the first line changed to “Choose SIMPLE or COMPLEX” instead of asking for a reason. That keeps the two close to a like-for-like comparison.

JEV returns a typed choice and a probability for each option:

{"tier": {"type": "choice", "choice": "COMPLEX",
          "probabilities": {"SIMPLE": 0, "COMPLEX": 1}}}

That was its answer for “Why has Cedar not been paid for order-1?”

Both the LLM classifier and JEV ran in standalone mode rather than through LiteLLM’s built-in auto-routing. LiteLLM supports both an LLM classifier and JEV as built-in options, so the same setup can be configured inside the gateway.

OpenRouter Auto

OpenRouter’s Auto Router works differently from the other five. There is no prompt or rule set to write, and it doesn’t use my SIMPLE/COMPLEX labels. OpenRouter applies its own task classification, then ranks candidate models using its market data.

  • I restricted it to only my two models (allowed_models: openai/gpt-5-nano, openai/gpt-5).
  • I supplied the cost tier myself, trying low and medium.
  • Both settings selected nano for every case.

That is a finding about this restricted two-model configuration. It is not evidence that OpenRouter’s router performs poorly in general.

The Results

Here is how each approach did on the same 18 cases. Costs are replay estimates for all 18 cases at full list price, and they include the classifier’s own cost. The keyword and semantic rows include their heuristic fallback.

ApproachCases correct / partial / incorrectFully correctnano / GPT-5 picksEstimated costSaving vs always GPT-5
Always GPT-5 nano9 / 7 / 250.0%18 / 0$0.02394.9%
Always GPT-517 / 1 / 094.4%0 / 18$0.456—
Keyword rules15 / 2 / 183.3%8 / 10$0.24047.4%
Heuristic (default)9 / 7 / 250.0%17 / 1$0.03891.7%
Semantic similarity13 / 4 / 172.2%13 / 5$0.13869.8%
LLM classifier16 / 1 / 188.9%9 / 9$0.24945.3%
JEV16 / 1 / 188.9%8 / 10$0.26641.7%
OpenRouter Auto (low and medium)9 / 7 / 250.0%18 / 0$0.02394.9%
Scatter plot titled "18-case saved-answer replay: case-level correctness vs estimated cost". Always GPT-5 is top right at 94.4% and $0.456. The LLM classifier and JEV sit at 88.9% around $0.25 to $0.27, keyword rules with heuristic fallback at 83.3% and $0.240, semantic similarity with heuristic fallback at 72.2% and $0.138. The default heuristic, always nano and OpenRouter Auto all sit at 50% near $0.02 to $0.04.
On these 18 cases, classifiers written for the domain came closest to GPT-5’s case-level correctness at about half its estimated cost.

On this tested set:

  • The domain-aware classifiers did better than the generic ones. The LLM classifier and JEV share one domain-aware routing rubric. They kept 16 of GPT-5’s 17 fully correct cases at 55–58% of its estimated cost. The default heuristic and the restricted OpenRouter configuration selected nano almost every time and scored the same as nano alone.
  • Two short keyword lists got surprisingly close: 83.3% against 88.9%, at similar cost and with no model call.
  • Separate the classifier’s cost from the route’s cost. JEV’s 18 classification calls cost less than the LLM classifier’s (about $0.0038 versus $0.0054), and each decision was faster (about 0.7 s versus 2.7 s). Its route still cost more overall, because JEV selected GPT-5 for 10 cases against 9. The extra one was a case nano had already answered correctly.
  • Semantic matching depends on its examples. 10 of 18 cases matched no example closely enough and fell back to the heuristic. In the separately versioned refinement, I added more examples: fallbacks dropped to 6, but correctness did not improve and cost went up.
  • Routing did not outperform fixed GPT-5 here. That isn’t a law. Routing can beat the best fixed model when different models succeed on different cases. In this pair, GPT-5 was also right on every case where nano was right, so there was nothing for routing to gain over GPT-5 alone.

Right vs Wrong: Three Examples

Averages hide the interesting part, so here are three cases in detail.

1. A short question isn’t a simple question. “Why has Cedar not been paid for order-1?” is only eight words, and the default heuristic scored it as simple and sent it to nano. The real answer needs the payout records: the seller’s payout had failed because of a provider timeout. Nano’s answer did not mention the timeout and was graded partial. GPT-5 identified the recorded provider timeout and was correct. The keyword rules (“why”), the LLM classifier and JEV all sent this question to GPT-5. Generic signals measure how a question looks; domain-aware ones try to measure what answering it takes.

2. Two identical questions, routed differently. Two questions differed only in the order ID: “Explain the refund history for ord-6f3604a5fe.” and “Explain the refund history for ord-aaa1768930.” In the refined semantic version, the first scored 0.529 and the second 0.493 against the closest example, on opposite sides of the 0.5 threshold.

  • The first went to GPT-5, although nano’s answer to it was already fully correct. That was wasted cost.
  • The second fell back to the heuristic and went to nano, whose answer was partial while GPT-5’s was correct. That was lost accuracy.

It was wrong in both directions, because of an order ID. Similarity thresholds can be sensitive to details that don’t change the task.

3. A keyword list is only as good as its coverage. “After returning one Canvas Weekender, what remains payable to Cedar?” is a money calculation: take the seller’s share of the sale, then adjust for the refunded item and the fee. Its wording (“returning”, “payable”) wasn’t in either keyword list, so it fell back to the heuristic and went to nano. Nano’s answer was partial; GPT-5 computed the correct amount, ₹2,249.10. Rules are cheap and transparent, but this gap led to a routing mistake. A fallback can sometimes still choose correctly; here it didn’t.

Lessons Learned

Domain-aware configurations did better on this set. Two keyword lists reached 83.3%, versus 88.9% for the LLM classifier and JEV. That makes simple rules a useful baseline, with their coverage gaps tested explicitly.

The LLM classifier and JEV agreed on 17 of 18 choices. JEV was faster and cheaper per decision, but agreement does not establish which part of the setup caused the benefit. Separate classifier cost from selected-model cost: a cheap classifier can still produce an expensive route by choosing GPT-5 more often.

A Checklist for Choosing a Classifier

  1. Shortlist models on benchmarks, domain fit, cost and latency, from approved providers that support your agent’s needs.
  2. Run evals on your golden set to narrow the shortlist. Then fix the pool and pin the versions.
  3. Weigh cost savings against accuracy. If the cheaper model meets your accuracy bar, use it. If only the stronger one does and its cost is acceptable, use it. Route only when routing saves meaningful cost while keeping accuracy within your bar.
  4. Design the classification. Choose labels that matter for your traffic and map them to models. Start with keyword rules, then try semantic or LLM-based classifiers built from real user journeys, not from test questions. Compare them on the same golden set, and look at both classifier cost and route cost.

Part 2 adds the remaining steps: choosing the scope on real conversations, and validating before you deploy.

Limits and What’s Next

This was a small study, and I would not treat it as a benchmark:

  • Sample size: 18 cases (20 answer turns per model), one run each.
  • Grading: the earlier calibrated judge, existing accepted grades, and an LLM reviewer with my adjudication of borderline cases.
  • Tuning: the rules, examples and prompts were written by someone who knew the test cases.
  • Costs: replay estimates at full list price.

The point is the process, not the percentages. Next, I want to:

  • test on fresh held-out cases, with repeated runs
  • add a middle-tier model
  • try routers I haven’t tested yet: Not Diamond, which learns from your graded data; Aurelio’s semantic-router; RouteLLM; Arch-Router; and the cloud platforms’ own model routers

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude Code, which also helped me research the routing landscape and plan the experiments. The application and experiments were built and run with GPT-6 Astra, which also reviewed answers against the expected results. Claude Opus 5.5 served as the calibrated LLM judge in the earlier model comparison.

References

This Series

Earlier Posts

Routing Tools

For Future Work

Right Model, Right Job: A Daily AI Digest with Jev and Claude at 1/10th the Cost

AI, , ,

Objective

The goal of this post is to show, with real numbers from a working system, how to pick the right model for each step of an AI workflow:

  • Where a decision model like Jev fits, and where an LLM is still the right tool.
  • How much that split saves compared to using an LLM for everything, on cost and on speed.
  • How to check that the cheaper model is good enough, by comparing its decisions against a strong LLM on the same inputs.
  • How to run it on a schedule in the cloud, using either Claude Code routines or GitHub Actions, and what each one is like to operate.

The example is a daily AI blog digest I built for myself. I used Jev, TypeSafe’s decision model, to judge every post, and Claude only to summarize the few that made the cut. The whole run costs about four cents a day. The pattern applies to any workflow that judges a lot of items and writes about a few.

The code is on GitHub: smakam/ai-blog-digest.

The Use Case

I follow around two dozen AI blogs through Feedly: frontier labs, research groups, a few people whose writing I trust, and some general tech and startup news. On a typical day that is 60 new posts. Maybe five of them are worth my time. The problem is not reading; it is deciding what to read.

So I built a small daily job that does the deciding for me. The requirements were short:

  1. Input: my Feedly subscriptions, exported as OPML. Only posts from the last day or so.
  2. Classify each post on two questions: is it about AI, and is it worth reading?
  3. Summarize only the best posts, 4–5 lines each: what it is, why it matters, the key takeaway.
  4. Deliver one Telegram message a day at 7 AM: top picks with summaries, a few more as plain links.
  5. Never send the same post twice, and never fail silently.
  6. Run in the cloud, without my laptop on.

One detail shaped the classifier: the worthiness judgment is about content only. Who wrote the post is not an input (there is a hook for an author boost later, but it is off).

The Question I Wanted to Answer

Can a purpose-built decision model replace an LLM for the high-volume, low-glamour part of an AI workflow, and what do you actually give up?

A digest is a good test case because it has a very lopsided shape:

  • Classification runs on every post. About 60 a day, each one a full article that has to be read in full.
  • Generation runs on almost nothing. Five summaries a day.

If you use one LLM for both, you pay LLM prices for the 55 posts you throw away. That is where nearly all the cost goes.

System One and System Two: Picking the Right Model for Each Job

TypeSafe describes Jev as a System One model. It does not generate text. You give it some state (here, the post’s title and text) and a set of typed questions, and it returns typed answers with probabilities:

  • a noul (yes/no) returns the probability of yes;
  • a score places the state on an ordered scale you describe in words and returns the expected level plus a probability per level;
  • a choice picks one option from a set.

I think of the LLM as the System Two side of this system (that is my framing, not TypeSafe’s): slower, more expensive, and the right tool when you need language out, not a decision.

JobVolumeWhat it needsModel
Is this post about AI? How worthwhile is it?Every post (~60/day)A calibrated decision, fast and cheapJev (System One)
Summarize the best postsTop 5/dayFluent, specific writingClaude Haiku 4.5 (LLM)

I used Haiku 4.5 for summaries because it is the cheapest Claude model and the summaries are short. The summarizer is a single model ID in the config, so Claude Sonnet or Opus can be dropped in when you want richer summaries. I ran Sonnet 5 for part of the testing; the cost difference is in the table further down.

Both are reached through OpenRouter, so one API key covers the whole pipeline. Jev uses OpenRouter’s Decisions API rather than chat completions. I call it through TypeSafe’s official Python SDK with the base URL pointed at OpenRouter, so moving to TypeSafe’s native API later is a base-URL and key change.

How It Works

How the daily digest works
  1. Fetch. Parse the OPML, fetch every feed, and keep posts from the last 36 hours that haven’t been processed before. (36 rather than 24, so a late or skipped run doesn’t lose posts; the seen-post record prevents repeats.)
  2. Get the text. Use the full text from the feed if it’s there, otherwise fetch the article and extract the main text, otherwise fall back to the title and feed summary. On a typical day, 62 of 64 posts had full text.
  3. Classify with Jev. One call per post, two questions. Text is capped at about 20K tokens because Jev’s context is 32K.
  4. Apply the policy in code. This is deliberately not in the model (details below).
  5. Summarize the top 5 High posts with Claude Haiku 4.5.
  6. Send one Telegram message, save the seen-post record and a per-post log, and send a short notice if anything failed.

The Classification Criteria

This is the actual question and rubric Jev gets. The whole “prompt engineering” for the classifier fits on one screen.

Is it about AI? (noul)

Is this blog post primarily about artificial intelligence: machine learning, LLMs, generative AI, AI agents, AI infrastructure, AI research, or AI products? Embodied AI, robotics, self-driving hardware, and humanoids count as NO.

With the two outcomes described as: yes means the post’s main subject is AI/ML software, models, research, tooling, or products; no means it is not mainly about AI, or it is about robotics or embodied AI.

So embodied AI, robotics, self-driving hardware, humanoids, and anything else where the AI lives in a physical machine are classified as no. Any other topic is also classified as no. I specifically called out embodied AI and Robotics as these were in my feedly.

How worthwhile is it? (score, lowest to highest)

How worthwhile is this post for an experienced AI engineer who wants technical insight and significant launches, not news noise? Judge the content only.

LevelDescription
0Not worth reading: funding or acquisition news, listicles, prompt-tip posts, generic opinion or hype pieces, pricing changes, marketing fluff.
1Low value: minor feature updates, shallow news recaps, or thin commentary with no technical substance.
2Moderate: a tutorial on known techniques, a modest product update, or informed commentary with some concrete specifics.
3High value: architecture detail, agent infrastructure, LLMOps practice, or a significant product launch (a new model, a new agent or developer framework, a major platform capability).
4Exceptional: genuinely new technical insight, such as original research results, deep architecture write-ups, or first-hand engineering lessons.

Jev returns a probability-weighted level, so a post can land at 2.8 rather than exactly 3. I divide by 4 to get a 0–1 worthiness score.

The call itself is short:

from typesafe_sdk import Noul, Score, TypeSafeClient
client = TypeSafeClient(api_key=OPENROUTER_API_KEY,
                        base_url="https://openrouter.ai/api", model="jev-1.13")
result = client.system_one(
    state={"title": post.title, "text": post.text[:80_000]},
    questions={"is_ai": IS_AI, "worthiness": WORTHINESS},
)
is_ai = result.nouls["is_ai"].noul                  # e.g. 0.97
worthiness = result.scores["worthiness"].score / 4  # e.g. 0.75

No JSON parsing, no “respond only with…”, no retries because the model added a sentence before the JSON. The answer is typed.

What the Script Does on Top of Jev

Jev answers the questions. The decisions stay in code, where they are easy to see and change:

  • Drop non-AI posts: is_ai below 0.5.
  • Bucket the rest by worthiness, with configurable thresholds:
BucketDefault thresholdWhat happens
High≥ 0.7Top 5 get a summary and link; the rest are listed as links
Medium0.4 to 0.7Title and link
Low< 0.4Not delivered (but logged)
  • Cap summaries at five. That keeps the digest to one Telegram message and caps LLM cost. High posts beyond the fifth are still delivered, as links.
  • Never send twice. Seen-post IDs are saved after each successful delivery. Posts Jev failed on are not marked seen, so the next run retries them.
  • Never fail silently. A broken feed, a Jev error, a summary error, or a failed delivery sends a short notice to Telegram.
  • Log every post with title, URL, source, both scores, bucket, where the text came from, latency, and cost. That log is what made the cost and quality comparisons below possible.

What It Costs

All numbers below come from real runs on my 24 feeds, priced by OpenRouter’s usage.cost field, not estimated from token counts.

Classification: Jev vs Claude Sonnet 5

I gave Jev and Claude Sonnet 5 exactly the same 59 posts, the same two questions, and the same rubric, and ran both through the same policy.

JevClaude Sonnet 5
Price (per million tokens)$0.042 input, output free$2 input, $10 output
Cost for 59 posts$0.006$0.41
Cost per 1,000 posts~$0.10~$6.95
Median latency per post0.34 s2.6 s

That is about 70× cheaper and 8× faster for the classification step.

Sonnet is a mid-tier model; a larger model would widen the gap. Cheaper LLMs narrow it but do not close it: from list prices, I estimate Gemini 3.5 Flash-Lite at around $0.06 for the same run (I did not measure this one), still roughly 10× Jev. And with a general-purpose LLM you also own the output format, the parsing, and the calibration.

The Whole Pipeline

SetupClassifySummarize (5 posts)Per runPer month
Jev + Claude Haiku 4.5 (what I run)$0.006$0.032$0.038~$1.15
Jev + Claude Sonnet 5$0.006$0.088$0.094~$2.80
Claude Sonnet 5 for both$0.41$0.088~$0.50~$15

So the split gives roughly 13× savings end to end, and 70× on the step that runs on every post.

At my volume these are small numbers; nobody will notice $15 a month. The point is the shape. Classification scales with how much you read; generation scales with how much you keep. If this were 10,000 posts a day for a team or a product, classification is the line that grows, and it is the line Jev takes off the bill.

How I Checked the Quality

Cheap is only interesting if the answers are good. I didn’t have hand-labeled data on day one, so I used Sonnet as a reference point. It is not ground truth, but it is a strong model reading the same text with the same rubric.

On the same 59 posts:

AgreementResult
Is it AI?58 / 59
Delivered or not (High/Medium vs dropped)55 / 59
Same bucket (High / Medium / Low / Not AI)50 / 59

Bucket by bucket (rows are Jev, columns are Sonnet):

HighMediumLowNot AI
High7110
Medium3330
Low0080
Not AI00132

The nine disagreements split into two kinds.

Four were near-misses. Posts that Jev scored between 0.53 and 0.69 and Sonnet put at High, or a funding story both of them dropped under different labels. Part of this is mechanical: Sonnet almost always picked a whole level (0, 0.25, 0.5, 0.75, 1), while Jev returns a probability-weighted position anywhere in between. Anything near a threshold will flip.

Five were real differences, and in every one Sonnet was the stricter judge. Jev put two Gemini product launches at High where Sonnet said Medium or Low, and put a Claude Code release note and two startup-news pieces at Medium where Sonnet said Low. You could argue either side on the launches (my own rubric counts “major platform capabilities” as High). On release notes and business news, Sonnet was closer to what I asked for.

The practical reading: Jev is slightly more permissive, and it never dropped anything Sonnet rated High. The worst case is a couple of extra links in the Medium list, not a missed post. If I want Jev closer to Sonnet, raising the Medium threshold from 0.4 to about 0.5 would catch some of it. I am going to decide that after a week of my own manual review, not from one day.

This was one day and 59 posts. I would not treat it as a benchmark. It was enough to convince me the cheap model is not trading away the part I care about.

Scheduling: Claude Code Routines vs GitHub Actions

The requirement was to run in the cloud with my laptop closed. I set up two options and ran the identical script on both, so any difference is the runtime, not the digest.

GitHub ActionsClaude Code routine
What runs the jobGitHub runs python -m digest directlyA Claude session (Haiku 4.5) reads a short instruction and runs the same command
Schedulecron, 07:00 ISTcron, 07:15 IST
SecretsRepository secretsCloud environment variables
Where state is savedCommits to mainCommits to its own claude/digest-routine branch
Cost to runFree within Actions minutesUses some of your Claude plan
DebuggingJob logSession transcript
Runtime (script only)~26 s~28–41 s, plus about a minute of session start-up

What Worked Well

Both delivered the same digest. Each run’s data cost was identical ($0.038) because both call the same models. GitHub Actions needed nothing beyond a workflow file and three secrets. The routine was easy to create and, once configured, followed its instructions exactly: it didn’t try to fix anything or peek at secrets, and it reported failures clearly.

What You Give Up with the Routine

With GitHub Actions, the schedule starts the script directly. With a routine, the schedule starts a Claude session, and Claude reads my instructions and then runs the script. That extra step is useful when a job needs judgment, but here the job is a fixed command, so the agent adds a moving part (and some Claude usage) without adding anything the digest needs. The routine’s cloud environment also has constraints that GitHub Actions doesn’t:

  • GitHub access is scoped to the configured repo. One of my feeds was a GitHub releases feed; from the routine it returned 403 with “sessions are bound to their configured repositories.”
  • Some sites block its outbound IP. One feed’s host served a CAPTCHA page to the routine instead of the feed.
  • State needs a home. Each run starts on a fresh machine, so anything not pushed back to git is gone. By default routines can only push to claude/ branches, which turned out to be a nice constraint: the routine keeps its state on its own branch and never touches main.
  • A missing secret can’t notify you. My first routine run failed because the keys were in a different environment, and with no Telegram token it couldn’t report that to Telegram. It was visible only in the session transcript.

My Recommendation

For a fixed script on a schedule, GitHub Actions is the simpler and more predictable choice. Reach for a routine when the job actually benefits from an agent: when it needs to read something and decide what to do, not just run a command. I am running both for a week to measure reliability properly, but I expect Actions to be the one I keep.

Lessons Learned Along the Way

A few general notes that weren’t about models at all:

  • Feedly exports aren’t all public feeds. Nine of the 31 feeds in my export failed on the first fetch outside Feedly. Three were feedproxy.feedly.com URLs that only work inside Feedly, five had moved, and one was just slow. Check every feed before you trust an OPML file.
  • Cap and check LLM output length. Every summary request sets a hard output limit (max_tokens) so a runaway response can’t blow up cost or the Telegram message. My first summaries asked for “4–5 lines” but came back around 8 lines and hit the limit (400 tokens at the time), so they were cut off mid-word. Two changes fixed it: a tighter prompt (“4–5 short sentences, 90 words at most”, with the limit lowered to 250 tokens to match), and trimming any reply that still hits the limit back to its last complete sentence. The trimming needed its own fix: a summary cut inside “2.66x” ended in “2.” and looked complete.
  • Decision models are not perfectly deterministic at the boundary. The same post, with identical text, scored 0.6975 in one run and 0.705 in another, either side of the 0.7 threshold. Small, but visible if you compare runs.
  • Strip what an unattended agent doesn’t need. When I created the routine through the API, it attached every connector on my account by default, including Gmail and Drive. A job that reads arbitrary web pages has no business holding my email. I removed them all.
  • Log everything you’ll want to compare. A JSONL line per post (scores, bucket, content source, latency, cost) turned every question in this post into a one-line query.

Summary

The lesson is simple: use the right model for the right job.

Most AI workflows have a high-volume judging step and a low-volume generating step. An LLM can do both, but you pay generation-model prices to make decisions, and with a general-purpose LLM you also get free-form output to parse. A System One model like Jev is built for the judging: typed answers, probabilities you can threshold in code, about 70× cheaper and 8× faster than Sonnet in my test, and in agreement with Sonnet on what to deliver for 55 of 59 posts. The LLM then does what it is uniquely good at, on the handful of posts that earned it.

For my digest that meant about four cents a day instead of fifty. At a larger scale, it is the difference between a feature you can afford to run on everything and one you ration.

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude Code, which also helped me build the digest and run the cost and quality comparisons described here.

References

Code

Jev and TypeSafe

Scheduling

7 Common Misconceptions When Choosing an AI Model

AI, , , , , , , , , ,

Choosing an AI model often appears straightforward: compare the leaderboard, check the token price, look at the context window, and select the winner.

In production, it is rarely that simple.

One resource I regularly use is Artificial Analysis, which compares models across intelligence, speed, and cost per task. For an actual product, I translate this into three broad dimensions:

  • Intelligence for the capabilities my system needs
  • End-to-end latency
  • Cost per successful task

Even these dimensions cannot be evaluated in isolation. A model only becomes useful when it is placed inside a system with prompts, context, tools, validation, recovery mechanisms, and real users.

Here are seven common misconceptions I have seen people make when choosing an AI model.

The benchmark figures in this article reflect published results available in August 2026 and may change as evaluations and models are updated.

1. At similar intelligence, a lower token price means a cheaper model

Token price and task cost are not the same thing.

Consider Kimi K3 and GPT-5.6 Sol. Kimi K3’s output-token price was $15 per million tokens, compared with $30 for GPT-5.6 Sol.

Based on the headline price, Kimi K3 appeared to be 50% cheaper.

However, Artificial Analysis estimated their cost per Intelligence Index task at:

  • Kimi K3: $0.94
  • GPT-5.6 Sol: $1.04

That is only about a 10% difference in task cost.

The reason is token efficiency. A model with cheaper tokens may consume more reasoning and output tokens to complete the same work. Input tokens, cache reads and writes, and answer tokens also contribute to Artificial Analysis’s task-cost calculation.

In a production system, the calculation can become even broader. Retries, failed outputs, tool calls, validation steps, and human corrections can all affect the final cost.

The lesson is:

Among models that provide the required intelligence, compare cost per successful task—not merely cost per token.

Read the Kimi K3 analysis and the GPT-5.6 Sol analysis.

2. The benchmark leader is the best model

Intelligence is not one universal capability.

A model may excel at coding but perform less well in automation, knowledge work, tool use, visual reasoning, instruction following, or presentation.

Kimi K3 illustrates this difference. It scored 57 on the overall Artificial Analysis Intelligence Index, behind GPT-5.6 Sol at 59. But Kimi K3 took the leading position on AutomationBench-AA at launch, scoring 53%.

Which model was better?

It depended on the task.

There is another reason to treat public leaderboards carefully: benchmark contamination. Evaluation questions or closely related material may appear in training data, potentially inflating the reported performance. Models may also be repeatedly optimized against familiar public evaluations without those improvements generalizing to new workloads.

This does not make benchmarks useless. They remain valuable for discovery and shortlisting. But the final decision should come from private evaluations based on:

  • Your actual prompts and data
  • The capabilities your workflow depends on
  • Your tools and system instructions
  • Known failure conditions
  • Required output formats
  • Your definition of a successful task

Research on benchmark-data contamination explains why public scores may not always represent performance on unseen tasks.

The lesson is:

Use public benchmarks to create the shortlist. Use workload-specific evaluations to select the model.

3. Open-weight models are cheaper

Open-weight models can be cheaper, particularly when there is enough sustained demand to use the underlying infrastructure efficiently.

But self-hosting also introduces costs:

  • GPUs and infrastructure
  • Capacity planning and utilization
  • Deployment engineering
  • Monitoring and reliability
  • Security
  • Model upgrades
  • Operational support

Cost may not even be the strongest reason to choose an open-weight model.

Control, privacy, deployment flexibility, data residency, offline operation, and independence from a hosted provider may be more important.

A recent Hugging Face security incident provides a powerful example. During its investigation, Hugging Face needed to analyze real attack commands, exploit payloads, and command-and-control artifacts. The commercial models it initially tried blocked those requests through their safety guardrails.

Hugging Face instead ran the open-weight GLM 5.2 on its own infrastructure. This also ensured that attacker data and referenced credentials did not leave its environment.

Importantly, Hugging Face did not identify which model powered the attacker’s system. Its conclusion was about defensive readiness: organizations may need a capable model they can operate within their own environment when hosted services cannot support the workflow.

Read the Hugging Face security-incident disclosure.

The lesson is:

Choose open-weight models for control and operational independence—not because they are automatically cheaper.

4. Domain-specific knowledge requires fine-tuning

Fine-tuning is one way to build a domain-specific system. It is not the only way.

Depending on the problem, domain knowledge can be introduced through:

  • Detailed prompting
  • In-context examples
  • Retrieval-augmented generation
  • SQL or structured-data retrieval
  • APIs and tool calling
  • Knowledge graphs
  • Deterministic business rules
  • User or session memory
  • Fine-tuning
  • A combination of these approaches

A useful distinction is to separate knowledge from behavior.

If the information already exists in documents, databases, or business systems—and changes regularly—it may be better to retrieve it at runtime.

If the model must consistently learn a new task pattern, classification boundary, response structure, tool-selection behavior, or specialized style, fine-tuning may be appropriate.

AWS explored this using Amazon Nova models and AWS-specific questions. In its experiment, both RAG and fine-tuning improved the average evaluated response score by approximately 30% over the base Nova Lite model. Combining fine-tuning with RAG produced the strongest improvement.

This was a limited experiment involving ten domain-specific questions and LLM-based judges, so the percentages should not be treated as a universal rule. The valuable result is that the appropriate architecture depends on the problem—and sometimes the best answer is a combination.

Read the AWS comparison of RAG, fine-tuning, and a combined approach.

The lesson is:

Domain-specific knowledge does not automatically require fine-tuning. First identify whether you are solving a knowledge problem, a behavior problem, or both.

5. The biggest model is always the best model

The most capable frontier model may produce the strongest answer, but that does not mean it creates the best user experience.

For bounded tasks such as classification, extraction, autocomplete, short summarization, or on-device assistance, a smaller model may provide:

  • Lower latency
  • Better privacy
  • Offline availability
  • Predictable cost
  • Reduced network dependence
  • Sufficient intelligence for the task

Apple’s Foundation Models guidance provides a practical example. Apple positions its on-device model for lightweight, latency-sensitive, privacy-sensitive, and offline tasks. When an application needs deeper reasoning or a larger context window, it can use a more capable server model through Private Cloud Compute.

The point is not that smaller models are always faster or better. Their performance still depends on the device, optimization, task, and model architecture.

The principle is:

Use the smallest model that reliably meets the intelligence, latency, and operational requirements of the task.

Read Apple’s documentation on on-device Foundation Models and server-side intelligence through Private Cloud Compute.

6. The same context-window size means the same context understanding

A context window describes how much information a model can accept. It does not guarantee how effectively the model can use that information.

NVIDIA’s RULER benchmark makes this distinction concrete.

Qwen3-235B-A22B and Mistral-Large-2411 both advertise 128K-token context windows. At 128K on RULER, their reported scores were:

  • Qwen3-235B-A22B: 90.6
  • Mistral-Large-2411: 48.1

RULER classified Qwen’s effective context length as greater than 128K, while Mistral-Large-2411’s effective length was approximately 64K.

The input could fit inside both models. Their ability to use it was substantially different.

Long-context performance can depend on whether a model can:

  • Find relevant details
  • Preserve instructions from earlier in the prompt
  • Connect information across distant sections
  • Ignore irrelevant material
  • Perform aggregation and multi-hop reasoning

The exact RULER results are one benchmark rather than a universal ranking. They nevertheless demonstrate why advertised context capacity should not be treated as effective context intelligence.

Review the NVIDIA RULER benchmark and results.

The lesson is:

Context capacity tells us what a model can accept. Effective context tells us how much it can use reliably.

7. Using the best model produces the best product

Users do not interact with a model in isolation. They interact with a complete system.

The harness around a model may manage:

  • Prompt and context construction
  • File and repository access
  • Tools and permissions
  • State and memory
  • Validation
  • Retries and recovery
  • Stopping conditions
  • Output formatting
  • Observability and human escalation

A coding-agent benchmark called Claw-SWE-Bench demonstrates how much this can matter.

The researchers evaluated OpenClaw with the same GLM 5.1 model using two different adapters.

With a minimal coding adapter, it scored 19.1% Pass@1. With a full repository-editing adapter, it scored 73.4%.

The full adapter placed the agent in the correct repository workspace, allowed it to edit files directly, extracted the resulting patch from the repository state, removed unrelated artifacts, and produced the format required by the evaluator.

The model did not become more intelligent. The surrounding system became better at converting its intelligence into a valid result.

The dramatic difference partly reflects correct integration with the benchmark’s patch-submission contract, so it should not be interpreted as proof that every harness improvement will produce a fourfold gain. However, the study’s broader controlled comparisons also found meaningful performance differences when the model was held fixed and the harness changed.

Read the Claw-SWE-Bench paper.

The lesson is:

Model intelligence is potential. The harness determines how effectively that potential becomes completed work.

A more practical model-selection process

Instead of asking, “Which model is best?”, start with a more specific set of questions:

  1. What intelligence does this workload require?
  2. Which failures are unacceptable?
  3. What end-to-end latency can users tolerate?
  4. What is the cost per successful task?
  5. How effectively does the model use the context we provide?
  6. Does the model work reliably with our tools and harness?
  7. What privacy, deployment, and operational controls do we need?
  8. How does it perform on our own evaluation set?

The best model is rarely the one with the most impressive number in a single column.

It is the model—and the surrounding system—that delivers the required intelligence, latency, reliability, control, and task economics for the application being built.

What misconceptions or unexpected quirks have you encountered while selecting models for production?

Why We Built Model Academy: Understanding How AI Models Produce Answers

AI, , , , , , , ,

Over the past few years, most of my work and learning in AI has been closer to the application layer—understanding models, building AI applications, and exploring how businesses can use them.

I knew the foundational ideas behind models, weights, prompts, and tokens, but I had not explored inference infrastructure and GPU execution in sufficient depth. I wanted to understand what really happens after an application sends a prompt to a model.

How is the model loaded and prepared for inference? What happens during prefill and decode? How are multiple GPUs used? What is a GPU kernel? How do warps, schedulers, compute units, and memory work together to produce the next token?

These concepts are often explained separately. You might find one resource about model weights, another about inference serving, and a highly technical tutorial about GPU internals. It is much harder to find a single learning journey that connects all these layers—from a trained model to the answer displayed in an application.

That gap became the motivation for creating Model Academy.

Two complementary perspectives

My background is primarily in networking and, more recently, AI applications. Systems concepts are familiar to me, but I had not previously gone deeply into modern AI infrastructure or the internal execution model of GPUs.

Ritesh Dhoot has spent considerably more time working with AI infrastructure and understands this part of the stack in much greater depth. I brought the perspective of an application builder trying to understand what happens beneath the interfaces we use every day.

That combination worked particularly well.

As we discussed each concept, we repeatedly asked two questions:

  1. Is this technically accurate?
  2. Can someone without a deep GPU or infrastructure background understand it?

The second question was just as important as the first. If I found a concept difficult to connect to the larger story, it was a useful signal that the explanation needed to be simplified, visualized, or placed in better context.

One connected journey

Model Academy follows the complete path from models to answers.

It begins with how models are built, what their weights contain, and what a model release actually provides. It then follows a request through an inference system: request preparation, model loading, batching, prefill, decode, KV cache, and multi-GPU execution.

From there, the journey moves inside the GPU to explain kernels, thread blocks, warps, scheduling, compute, and memory movement. It concludes by replaying one prompt from the initial request to the final streamed answer.

The academy is intended for engineers, architects, technology leaders, and curious learners who want a practical mental model of the complete system. The goal is not to turn every reader into a GPU programmer. It is to help people understand how the layers fit together and why different architectural decisions matter.

Learning through interaction

We strongly believe that complex systems are easier to understand when you can see them operating.

For that reason, Model Academy is not only a collection of written chapters. We have tried to use animations and interactive exercises throughout the learning journey.

Instead of merely reading that temperature changes a token distribution, you can change it and observe the result. Instead of reading a list of inference stages, you can follow a prompt through request preparation, model execution, and token generation. You can also zoom inside a GPU and explore how kernels, blocks, warps, schedulers, compute units, and memory work together.

The academy provides a Beginner mode that explains the complete journey in approachable language. Expert mode progressively exposes the mechanisms underneath without requiring a separate curriculum.

The first part of a larger learning series

This academy covers the journey from models and weights through inference systems and into GPU execution. But we see it as the first part of a broader learning series.

Over time, we would like to explore more of the systems operating “under the hood” of AI: model architectures, training infrastructure, inference optimization, distributed systems, networking, accelerators, and other topics that connect AI applications to the platforms beneath them.

For now, we hope Model Academy makes an intimidating subject more approachable, visual, and connected.

Explore the academy at ModelAcademy.tech. The project is also available as an open-source repository on GitHub.

We welcome your feedback and hope the academy helps you understand—and confidently explain—how AI models produce answers.

Model Academy is created and maintained by Sreenivas Makam and Ritesh Dhoot.

AI in CI Should Not Replace Your Tools. It Should Read the Evidence.

AI, Programming, , , , , , , , , , ,

In Part 1, I compared three approaches to AI-assisted code review: hosted Codex review, a repository-owned GitHub Action using Codex, and CodeRabbit. That experiment intentionally focused on one stage of the pull request lifecycle: review comments on code.

This article moves one step wider.

Once a pull request is opened, code review is only one part of the continuous integration story. A real CI pipeline also has static analysis, security checks, unit/API tests, maybe dynamic analysis, and some final judgement about whether the PR is ready for a human reviewer.

The question I wanted to answer was not:

Can AI replace CI tools?

That is the wrong framing.

The better framing is:

Can deterministic tools produce evidence, and can AI turn that evidence into a useful reviewer-facing judgement?

That is the architecture I tested.

The design principle

The core design principle is simple:

Deterministic tools produce evidence.
AI interprets the evidence.
The human reviewer makes the final decision.

AI is not the linter. AI is not the security scanner. AI is not the test runner. AI should not be trusted as the source of truth for whether code actually passed.

Instead:

  • The static-analysis tool should decide whether linting and formatting passed.
  • The security scanner should report known security patterns.
  • The test framework should execute tests.
  • AI can generate additional targeted tests based on the PR diff.
  • pytest should still execute those AI-generated tests.
  • AI can then read the outputs and produce a final review summary.

In my experiment, those tools were Ruff, Bandit, and pytest. In another company, they could be different. A Java service might use Checkstyle, SpotBugs, PMD, Semgrep, Maven, Gradle, JUnit, or Snyk. A JavaScript service might use ESLint, TypeScript, npm audit, Playwright, Jest, or Vitest. A mobile team might use SwiftLint, Detekt, XCTest, Espresso, or Appium.

The specific tool is not the point. The point is that each tool should do the thing it is built to do. AI should assist around those tools: fill gaps, generate targeted probes, correlate outputs, and produce a reviewer-facing summary.

That gives us a more scalable pattern than asking one model to “review everything.”

The experiment

I used the same repository from the code-review experiment:

The base workflow is implemented in .github/workflows/ci-evidence-report.yml. During the blocked-path experiment, I added a deterministic evidence summary and a separate readiness gate; that refined seven-stage version is preserved in the PR #8 experiment commit.

The AI prompts live in:

The base workflow has five core jobs. The refined experiment adds two orchestration jobs, producing seven distinct stages:

  1. Static analysis
  2. Security analysis
  3. Backend unit/API tests
  4. AI-generated targeted tests
  5. Deterministic CI evidence summary
  6. AI final reviewer report
  7. CI readiness gate

The workflow also uploads artifacts between jobs. That matters because the AI final-report job should not rely on vague impressions. It should read concrete outputs: Ruff JSON, Ruff format output, Bandit JSON, pytest XML/output, and generated-test output. GitHub Actions artifacts are designed for exactly this kind of cross-job evidence sharing.

What the AI was instructed to do

There were two separate AI responsibilities, with two separate prompts.

The first prompt, targeted-test-generation.md, asks the AI to inspect the PR diff and generate temporary pytest tests only when the changed behavior justifies it. The important constraints are:

  • focus on risky changed backend behavior;
  • generate tests under the allowed temporary path only;
  • do not modify production code;
  • do not treat the model’s judgement as the test result;
  • let pytest execute the generated tests.

The second prompt, ci-final-report.md, asks the AI to read the evidence artifacts and produce a PR comment for the manual reviewer. It is not supposed to invent results. It reads files like:

  • static-analysis-results/ruff-check.json
  • static-analysis-results/ruff-format.txt
  • security-analysis-results/bandit.json
  • backend pytest output
  • AI-targeted pytest output

That final report should answer a different question from the test-generation prompt:

Given all the evidence, is this PR ready for manual review, and where should the reviewer focus?

What the pipeline looked like

At a high level, the pipeline looked like this:

Deterministic checks control readiness. AI turns their evidence into a reviewer-facing explanation.

The diagram shows the refined architecture. The separate evidence-summary and readiness-gate jobs were added while iterating on PR #8; the earlier clean PR #11 run used the five-job base workflow.

The “AI code review” box is the bridge to the first article. In the actual Blog 2 workflow, I focused on CI evidence. But in a complete implementation, code-review findings should also become one more input to the final reviewer report.

The most important detail is the separation between generation and execution.

For AI-generated tests, Codex is allowed to write a temporary pytest file under a controlled path:

ci-generated-tests/backend/test_pr_targeted.py

But Codex does not decide whether the tests pass. pytest does.

That distinction is important. The model can suggest a hypothesis. The test runner proves or disproves it.

Static analysis: let the deterministic tool do its job

For static analysis, I used Ruff:

ruff check backend --output-format=json
ruff format --check backend

Ruff is fast, deterministic, and well suited to CI. This is exactly the kind of work AI should not replace.

In another stack, replace Ruff with the tool the team already trusts. The workflow remains the same: run the tool, store the output, and let AI interpret that output alongside other CI signals.

One practical lesson showed up immediately: the existing repo baseline was not fully Ruff-clean. That created noise. So I first created a baseline cleanup PR:

That PR is not the core blog experiment. It was housekeeping. But it was necessary because PR-level CI evidence becomes much clearer when the baseline is already clean.

This is a general lesson: before you use CI evidence to judge a PR, decide whether your tools are checking the whole repo, changed files, or changed lines. If the baseline is noisy, the AI report will also become noisy.

Security analysis: deterministic scanner first, AI second

For security analysis, I used Bandit:

bandit -r backend/app -f json -o ci-results/security/bandit.json

Bandit is a static security scanner for Python. In this experiment, it was not meant to prove the application is secure. It was meant to provide a deterministic signal for known classes of Python security issues.

That distinction matters.

If Bandit finds a high-signal issue, the AI report can explain the impact and connect it to the PR. If Bandit finds nothing, the AI report should not overstate that as “the code is secure.” It should say something closer to:

Bandit did not report findings in the scanned backend application code. This does not prove runtime authorization, data access, or business-logic security.

That is the kind of nuance AI is useful for.

Testing: existing tests plus AI-generated targeted tests

There are three different testing layers that are easy to confuse:

  1. Tests the developer writes before opening the PR.
  2. Existing CI tests that protect known behavior.
  3. Temporary AI-generated targeted tests created from the PR diff.

This experiment focused on the second and third.

The normal backend test job ran the existing pytest suite. Separately, the AI-targeted-test job asked Codex to inspect the PR diff and generate a temporary pytest file for risky changed behavior.

The generated tests were not committed back to the repository. They were CI evidence.

That is a useful pattern. AI-generated tests do not have to become permanent tests every time. They can still be valuable as temporary review probes:

  • Does this PR change an authorization path?
  • Does this PR change field mapping?
  • Does this PR change input normalization?
  • Does this PR change error behavior?

If the generated test exposes a defect, the PR should be blocked. If the generated test is genuinely useful long term, the developer can convert it into a committed regression test.

That last point matters. AI-generated tests should not remain temporary forever if they expose a real gap. Once a targeted generated test catches an important issue, the baseline test suite should be updated with a durable version of that test. Otherwise the pipeline has to rediscover the same risk again in a future PR.

The bad-path PR

To test whether the pipeline could catch real problems, I created an intentionally bad PR:

This PR intentionally included multiple defects:

  • an access-control regression;
  • a registration-correction mapping bug;
  • a security-pattern issue for Bandit/static security evidence.

The result was what we wanted: deterministic jobs produced evidence, the AI-generated targeted tests failed, and the CI readiness gate blocked the PR.

The blocked path: evidence was produced successfully, but the readiness gate failed.

The static-analysis and security-analysis jobs appear green because they completed their work and uploaded evidence instead of stopping the workflow immediately. Their tool-status files still recorded failures. The readiness gate read those internal statuses and blocked the PR. This allowed the AI report to be generated even when scanners found problems.

The important part is not just that something failed. The important part is that the workflow made the failure explainable.

The final report could say:

  • Ruff/Bandit/backend tests produced their respective signals.
  • The AI-generated targeted tests found blocking evidence.
  • The PR is not ready for merge.
  • The manual reviewer should focus on the authorization and mapping behavior.

That is much more useful than a generic failed check.

The AI report explains why the PR is blocked and where the reviewer should focus.

There is an important status distinction here. The AI final reviewer report job can be green even when the PR is blocked. Green means the report was generated and published successfully. It does not mean the report recommends merging. The separate CI readiness gate represents whether the underlying evidence passed.

The good-path PR

I also created a clean PR:

This PR added a small, legitimate backend behavior change:

  • normalize medical registration corrections;
  • add a focused unit test for that normalization.

The result was the good path:

  • Static analysis passed.
  • Security analysis passed.
  • Existing backend tests passed.
  • AI-generated targeted tests passed.
  • AI final reviewer report passed.

The clean path: the five-job base workflow completed successfully.

This clean run predates the separate readiness-gate job added on PR #8, so I am not claiming that PR #11 executed that later check. It demonstrates the clean evidence path and a “ready for manual review” AI report; PR #8 demonstrates the explicit gate behavior.

The final AI report did not merely say “all green.” It summarized the evidence and still told the human reviewer where to focus:

Even on the clean path, the report gives the human reviewer a concrete focus area.

This is the workflow I want from AI in CI: not blind approval, not replacement of the reviewer, but a compact evidence report.

Constrain the AI job

There is one security point that should not be treated as an implementation detail.

If an AI agent runs inside CI, it should run with limited permissions.

In this experiment:

  • deterministic jobs used read-only repository access;
  • the AI test-generation job could write only inside the runner workspace;
  • generated tests were restricted to one expected path;
  • the final report job had permission to write a PR comment, but not to merge;
  • expensive AI jobs were label-gated;
  • the workflow used sandboxing and privilege reduction options exposed by openai/codex-action.

The enforcement happens in the workflow file itself: .github/workflows/ci-evidence-report.yml.

The practical controls are:

  • GitHub Actions permissions blocks, such as contents: read, so jobs do not receive unnecessary repository permissions.
  • pull-requests: write only on the final-report job, because that job needs to post a PR comment.
  • sandbox: workspace-write for Codex so it can write generated tests only inside the checked-out workspace.
  • safety-strategy: drop-sudo so Codex does not keep elevated runner privileges.
  • codex-args: '["--ephemeral"]' so the Codex run does not persist unnecessary state.
  • a scope-verification step that checks git status and fails if Codex modifies anything outside the expected generated-test file.
  • label gating with ci-ai-report, so expensive AI jobs run only when intentionally requested.

That is the right default posture.

An AI job in CI should not inherit broad write permissions just because it is convenient. Treat it like any other automation: give it the minimum access required for the specific job.

What does this architecture cost?

There are four different costs in this design:

  1. The deterministic tools.
  2. GitHub Actions runner time and artifact storage.
  3. AI model usage.
  4. Engineering time to build and maintain the orchestration.

Ruff, Bandit, and pytest are open source, so this experiment did not add license fees for those tools. An enterprise could substitute commercial scanners or test platforms, which would add their own licensing costs without changing the architecture.

The workflow has two AI calls when the ci-ai-report label is present:

  • one to generate targeted tests;
  • one to create the final reviewer report.

For a concrete planning example, assume those two calls together use 150,000 input tokens and 10,000 output tokens for one labeled PR run. That is an illustrative workload, not a measurement of the experiment’s actual bill. The workflow did not pin a model, and real usage depends on diff size, repository context, generated tests, evidence volume, caching, and retries.

Using current public API prices, that workload would cost approximately:

Reference modelInput price per 1M tokensOutput price per 1M tokensIllustrative cost per labeled PR run
GPT-5.4 mini$0.75$4.50About $0.16
GPT-5.4$2.50$15.00About $0.53

The corresponding uncached monthly model-cost estimates are:

Labeled AI-assisted CI runs per monthGPT-5.4 mini referenceGPT-5.4 reference
100About $16About $53
250About $39About $131
500About $79About $263

GitHub runner time is additional. GitHub currently lists a standard two-core Linux runner at $0.006 per minute beyond any included allowance. If the complete workflow consumes an illustrative 20 billable runner-minutes across its jobs, that is about $0.12 per PR run, or $12 for 100 runs. Public-repository rules, included minutes, job-level rounding, artifact storage, and the actual duration of each job can change that number.

If the Part 1 AI code review is also executed as another model-backed job inside the same orchestration, its token and runner usage must be added. Dynamic analysis would similarly add application-startup time, scanner time, and possibly commercial tool licensing.

The direct usage cost can therefore be modest. The larger cost in the repository-owned approach is engineering ownership: prompts, schemas, permissions, artifacts, inline publication, deduplication, retries, gates, and ongoing maintenance.

Could CodeRabbit replace this GitHub Actions workflow?

CodeRabbit can cover parts of this workflow. Its Pro plan includes AI pull-request review plus linter and SAST-tool support. Pro+ adds capabilities around the review process, including unit-test generation and other finishing actions.

But I would not describe CodeRabbit and this GitHub Actions pipeline as exact substitutes.

The Blog 2 experiment used GitHub Actions for the complete evidence flow. I did not run an equivalent end-to-end CodeRabbit experiment combining the same Ruff, Bandit, pytest, temporary-test, artifact, readiness-gate, and final-report stages. The comparison here is therefore architectural and economic, not a claim that both implementations were tested identically.

AreaRepository-owned GitHub ActionsCodeRabbit
OrchestrationFully controlled in the repositoryProduct-managed with configuration controls
Tool selectionAny tool the runner can executeSupported product integrations and features
Model choiceTeam chooses the model; changing provider means replacing the AI step or adapterCodeRabbit controls model routing
Evidence formatTeam owns artifacts, schemas, gates, and report formatProduct-defined workflow and reporting
Custom test executionFully programmableProduct capability, with unit-test generation on Pro+
Dynamic analysisCan add any start-and-scan jobDepends on supported integrations and product workflow
Operational effortHighestLower because the product owns the review machinery
Pricing shapeRunner usage + model usage + engineering timePrimarily per-developer subscription, with plan limits and add-ons

Current annual-billing prices list CodeRabbit Pro at $24 per developer per month and Pro+ at $48 per developer per month. For a five-developer team, that is $120 or $240 per month before any applicable usage add-ons.

For comparison, 100 labeled GitHub Actions runs under the assumptions above would be roughly $16 to $53 in model usage, plus an illustrative $12 in billable Linux runner time. That makes the repository-owned path look cheaper in direct usage—but it excludes the engineering time required to build and operate everything CodeRabbit packages as a product.

My preference for this broader CI architecture is GitHub Actions because it provides the maximum flexibility. The team can select the deterministic tools, choose the AI model, define permissions, decide what blocks the PR, add dynamic analysis later, and control exactly what the final reviewer sees.

That does not mean CodeRabbit has no place. A practical hybrid is to use GitHub Actions as the CI evidence backbone and feed CodeRabbit—or another dedicated reviewer’s findings—into the final evidence report as the code-review signal from Part 1.

A useful edge case: no generated tests

One of the most useful lessons came from a failure in the workflow itself.

On a formatting-only PR, Codex correctly decided that no targeted tests were needed. It created a placeholder pytest file explaining that no targeted backend test was generated.

That was logically correct.

But pytest collected zero tests and exited with code 5. According to pytest’s documented exit codes, exit code 5 means no tests were collected. GitHub Actions treated that as a failed job.

So the AI decision was right, but the CI wrapper was wrong.

The workflow now treats this specific case as non-blocking:

if [ "$pytest_status" -eq 5 ] && \
   grep -q "collected 0 items" ci-results/ai-targeted-tests/pytest-output.txt; then
  pytest_status=0
fi

That keeps the semantics clean:

  • generated tests fail: block the PR;
  • no generated tests needed: pass;
  • scope violations: block the PR;
  • real pytest failures: block the PR.

This is a small implementation detail, but it is an important design lesson: AI-generated CI steps need explicit no-op semantics. Otherwise a correct AI decision can become a false CI failure.

What about dynamic analysis?

Dynamic analysis is still important. Tools like OWASP ZAP can scan a running application and catch issues that static analysis cannot see.

But I did not include dynamic analysis in this experiment.

That does not change the architecture.

Dynamic analysis would simply become another deterministic evidence producer:

Start app in CI
Run dynamic scan
Upload scan output
AI final report reads scan output
Human reviewer gets the combined summary

The reason I skipped it here is practical scope. Dynamic analysis introduces extra operational concerns:

  • starting the application in CI;
  • configuring test-safe environment variables;
  • database or dependency setup;
  • authentication/session handling;
  • seed data;
  • scan tuning to avoid noisy findings;
  • deciding whether the scan is advisory or merge-blocking.

Those are real topics, but they do not change the core AI-assisted CI workflow. They are a next extension, not a prerequisite for proving the pattern.

For this article, Ruff + Bandit + pytest + AI-generated targeted tests were enough to validate the architecture.

What this means for AI-assisted CI

The experiment made the boundary clearer for me.

AI is useful in CI when it does things deterministic tools cannot do well:

  • read multiple tool outputs together;
  • connect failures back to PR intent;
  • explain whether a finding is likely introduced by the PR;
  • generate targeted tests for changed behavior;
  • tell the manual reviewer where to focus;
  • summarize uncertainty instead of hiding it.

AI is weaker when it tries to replace purpose-built tools:

  • linting;
  • formatting;
  • dependency scanning;
  • security pattern matching;
  • test execution;
  • coverage calculation;
  • merge gating.

The scalable approach is not “AI does CI.”

The scalable approach is:

CI tools produce structured evidence.
AI reads the evidence.
The reviewer gets a decision-ready summary.

Practical recommendations

If I were applying this pattern across projects, I would start with these rules.

1. Keep deterministic checks deterministic

Use existing tools for static analysis, security analysis, and test execution. Do not ask the model to decide whether formatting passed or whether a test passed.

2. Store outputs as artifacts

The AI report should read artifacts, not screenshots or vague logs. GitHub Actions artifacts make this straightforward.

3. Separate AI generation from execution

Let AI generate targeted tests, but run them with the normal test framework.

4. Label-gate expensive AI jobs

In the experiment, AI-generated tests and the final AI report were gated behind a label:

ci-ai-report

That gives control over cost and avoids running expensive AI jobs on every trivial PR.

5. Create a clear no-op path

If no targeted tests are needed, that should be a successful outcome, not a failed job.

6. Keep the final human reviewer in control

The AI report should recommend readiness. It should not silently merge code.

7. Pin and measure the production model

Choose the model deliberately, record actual token and runner usage, and set a budget per AI stage. Planning estimates are useful, but production cost controls should be based on measured runs from the team’s real PR distribution.

Final takeaway

The best role for AI in CI is not replacing the pipeline.

It is making the pipeline readable.

Static analysis, security scanners, and tests already know how to produce signals. The problem is that those signals are often scattered across logs, artifacts, comments, and check statuses.

AI can turn those signals into a coherent review brief:

  • what passed;
  • what failed;
  • what matters;
  • what is probably introduced by the PR;
  • what the human reviewer should inspect next.

That is a practical use of AI in CI.

Not magic. Not autonomous merging. Just better evidence flow.

Experiment references

Implementation files

Tool references

AI Code Review in CI: Codex Cloud vs GitHub Actions vs CodeRabbit

AI, Programming, , , , , , , ,

I have been thinking about how AI should fit into CI/CD. Not the part where a developer uses an AI coding agent before opening a pull request. That is important, but it is still developer workflow.

The part I wanted to test was what happens after the pull request is opened.

At that point, a normal CI pipeline already has several responsibilities:

  • Run tests.
  • Run static analysis.
  • Run security checks.
  • Possibly run dynamic or integration checks.
  • Give reviewers useful evidence before a human approves the change.

AI can fit into this, but it should not replace every deterministic tool. Linters, type checkers, test frameworks, dependency scanners, and security scanners are still valuable because they are repeatable. The interesting question is where an AI reviewer adds judgment: finding logical defects, interpreting tool output, spotting missing tests, and giving a reviewer a concise explanation of risk.

This first experiment focuses only on AI code review for pull requests. I compared three approaches:

  1. Hosted Codex Cloud code review
  2. A repository-owned GitHub Action using Codex
  3. CodeRabbit

The broader CI question — static analysis, dynamic analysis, security tooling, and testing with AI interpreting the results — is covered in Part 2: AI in CI Should Not Replace Your Tools. It Should Read the Evidence.

The Question I Wanted to Answer

The starting point was simple: if an AI tool can write code, can a different AI tool review it?

My instinct was that the reviewer should be independent from the coding agent. If the same model, prompt, and context wrote the change and then reviewed it, the review is less convincing. It may repeat the same assumptions. Independence matters.

But independence does not automatically require a dedicated code review product. A coding agent can also act as a reviewer if it is run with a different prompt, different constraints, and a review-only role.

So the actual comparison is not just “model vs product.” It is:

  • How much review quality comes from the model?
  • How much comes from the prompt and repository instructions?
  • How much comes from operational product features such as GitHub integration, inline comments, deduplication, review status, dashboards, and configuration?
  • How much control does the repository owner retain?
  • How much effort is required to make the workflow reliable?

The Test Repository

I used a real repository for the experiment:

Repository: smakam/doctor-parser

The repository setup was introduced in PR #1. The repository-owned inline-review publisher was added in PR #3. I then used controlled experiment PRs for the actual comparison rather than reviewing the setup changes themselves.

The application has a backend and frontend, including code paths for uploaded doctor nameboard extraction and review/correction flows. That gave us enough realistic surface area to test authorization and data-mapping bugs.

To keep the comparison fair, I used two controlled defects.

Defect 1: Authorization Check Inversion

In backend/app/routers/nameboard.py, the access check was changed from denying access when neither ownership identifier matched to denying access when the session identifier did match.

-    if uploader != requesting_user_id and record.session_id != requesting_user_id:
+    if uploader != requesting_user_id and record.session_id == requesting_user_id:

The expected finding was a high-priority authorization issue: a requester whose ID differs from both the uploader and session ID could access or mutate another record, while a matching guest session could be incorrectly denied.

Defect 2: Incorrect Correction Mapping

In backend/app/services/review_service.py, the correction mapping for medical_registration_no was changed to write into qualifications.

-        "medical_registration_no": "medical_registration_no",
+        "medical_registration_no": "qualifications",

The expected finding was a data integrity issue: submitting a registration-number correction would overwrite qualifications and leave the actual registration number unchanged.

These are useful defects for this kind of experiment because they are not formatting issues. They require the reviewer to understand intent, data flow, and security impact.

Shared Review Guidance

One important design choice was to make the review guidance repository-owned.

The shared policy lives in .github/review/code-review.md.

The policy tells the reviewer to prioritize:

  • Authorization and access-control regressions
  • PII exposure
  • Security-sensitive behavior
  • Logic defects introduced by the pull request
  • Incorrect persistence or field mapping
  • Missing validation that changes behavior

It also sets a threshold: only report actionable defects introduced by the PR. Do not produce a generic style review.

This is an important lesson from the experiment. The model matters, but the review role matters just as much. If you want an AI reviewer, you need to define what “good review” means for your repository.

How the Three Review Paths Fit Together

All three approaches start with the same pull request and the same repository-owned review policy. What changes is who runs the review and who owns the operational layer around it.

The review policy is shared. The execution and publication layers differ.

Option 1: Hosted Codex Cloud Code Review

The first approach used hosted Codex Cloud code review.

In this mode, the repository uses AGENTS.md as the durable instruction entry point. For PR reviews, AGENTS.md routes the reviewer to the shared review policy.

There is no GitHub Actions workflow involved in this path. The hosted Codex integration reviews the pull request and posts inline GitHub review comments.

In our experiment, hosted Codex posted inline comments as chatgpt-codex-connector on PR #4.

Hosted Codex posted native inline findings on the changed code.

What Worked Well

The setup cost was low. Once the repository guidance was in place, the hosted reviewer could use that context and post inline comments directly on the PR.

It found the important seeded defects, including the authorization issue and the incorrect field mapping.

This is the simplest path if you want an AI reviewer without building a review pipeline.

What You Give Up

The tradeoff is control.

You do not own the workflow in the same way you own a GitHub Action. You have less control over:

  • Trigger rules
  • Output schema
  • Deduplication behavior
  • Review gating
  • Model/provider substitution
  • Combining the review with other CI signals
  • Custom publishing logic

For many teams, that may be fine. But if the long-term goal is a larger AI-assisted CI system, this approach may become only one review source rather than the central orchestration layer.

Option 2: Repository-Owned GitHub Action Using Codex

The second approach used a GitHub Action.

The GitHub Action does not use a separate definition of what good review means. It uses the same AGENTS.md and shared .github/review/code-review.md policy as hosted Codex. The Action prompt explicitly tells Codex to read AGENTS.md, which routes it to that common policy.

On top of that shared layer, the repository owns four Action-specific files:

This separation is deliberate. The shared code-review.md file defines review priorities, repository invariants, and the finding threshold. The Action-specific pr-review.md file defines how this particular automated run should execute and how it must structure its output. It does not duplicate the review policy.

The workflow is label-gated. A pull request is reviewed only when the codex-action-review label is present. It also runs again when a new commit is pushed to the PR branch, as long as the label remains present.

The action asks Codex to produce structured JSON with sections such as:

  • summary
  • findings
  • validation
  • coverage
  • uncertainty

That structure is not mainly for humans. It is an execution contract between the reviewer and the publishing script. The script needs predictable fields so it can decide which findings can become inline comments and which ones should fall back to a summary.

The GitHub Action then publishes review comments as github-actions[bot] on the same PR #4.

The repository-owned Action produced native inline comments after we built the publishing layer.

What We Had to Build

This approach required the most custom work.

To make the review useful, we had to build:

  • A review prompt with strict output expectations
  • A JSON schema for review results
  • Patch parsing to understand changed lines
  • Inline comment publishing through the GitHub API
  • Fallback summary behavior for findings that cannot be anchored to a changed line
  • Fingerprinting to avoid duplicate comments on repeated runs
  • A safer permission model where the trusted publisher code comes from the base branch

This is the biggest practical difference between a coding agent and a code review product. The model can identify findings, but the product workflow around those findings still has to exist somewhere.

The Fingerprint Problem

Fingerprinting is specific to the repository-owned Action path.

If a workflow runs every time a commit is pushed, the reviewer may find the same issue more than once. Without deduplication, the PR gets noisy. The same comment can be posted repeatedly.

The fingerprint solves that by creating a stable identity for a finding, usually based on fields such as:

  • File path
  • Line number
  • Rule or issue title
  • Finding body or normalized explanation

Before posting a new comment, the publisher checks whether an equivalent finding has already been posted. If yes, it avoids posting a duplicate.

Dedicated review products usually handle this operational detail internally. In the GitHub Action approach, the repository owner has to implement it.

What Worked Well

The GitHub Action found both controlled defects and posted inline comments.

It also gave the most control:

  • The model can be swapped.
  • The prompt can be changed.
  • The trigger policy can be changed.
  • The output schema is owned by the repository.
  • The review can later be combined with static-analysis, security, or test output.

This is the strongest architecture if the goal is a scalable, model-agnostic AI review pipeline.

What You Give Up

You own the operational burden.

Inline comments, deduplication, schema validation, retries, permissions, security hardening, and future maintenance are all your responsibility.

If you only want PR review, this may be too much. If you want an AI-assisted CI orchestration layer, this is probably the most extensible foundation.

Option 3: CodeRabbit

The third approach used CodeRabbit, a dedicated AI code review product.

For this experiment, CodeRabbit was configured with .coderabbit.yaml.

The configuration used:

  • The assertive review profile
  • Code guidelines pointing to .github/review/code-review.md
  • Path instructions for backend routers and services

The CodeRabbit configuration was first merged independently in PR #5. The clean CodeRabbit test was PR #6. It was created as a non-draft PR, with the configuration already present on main, and without the competing Codex Action label.

CodeRabbit posted inline comments as coderabbitai[bot].

CodeRabbit produced native inline comments in the clean, non-draft experiment.

What Worked Well

Under clean conditions, CodeRabbit found both seeded defects:

  • The authorization bypass in backend/app/routers/nameboard.py
  • The incorrect registration-number mapping in backend/app/services/review_service.py

The product experience is clearly stronger than a bare GitHub Action. CodeRabbit provides more than model output:

  • Inline comments
  • PR walkthroughs and summaries
  • Review status
  • Pre-merge checks
  • Configuration through YAML
  • Code guidelines
  • Path instructions
  • Knowledge-base behavior
  • Learnings
  • Autofix and docstring generation depending on plan
  • Unit test generation on higher plans
  • Analytics and reports
  • IDE and CLI review options
  • Integrations such as Jira and Linear
  • Linter and SAST tool support

That is the real value of a dedicated code review product. It is not just that it has a model. It packages the operational workflow around the model.

The Inconsistent Run

Before the clean PR #6 test, we had the noisier PR #4.

That PR was draft at one point, had hosted Codex comments, GitHub Action comments, manual CodeRabbit review commands, and CodeRabbit configuration changes inside the same PR.

In that noisy run, CodeRabbit selected the relevant files but did not publish the two expected actionable inline comments. Later, a CodeRabbit chat response acknowledged that the two issues should have been flagged.

I would not treat that as proof that draft PRs are unsupported or that CodeRabbit cannot find these defects. The cleaner conclusion is more limited:

CodeRabbit’s behavior was inconsistent in a noisy mixed-bot/draft/config-changing experiment, but it worked correctly in a clean non-draft PR with configuration already present on main.

That distinction matters. The blog should not overclaim from a messy experiment.

Feature Comparison

AreaHosted Codex CloudGitHub Action + CodexCodeRabbit
Setup effortLowHighMedium
Inline commentsYesYes, after custom publisherYes
Comment authorchatgpt-codex-connectorgithub-actions[bot]coderabbitai[bot]
Uses shared repo guidanceYes, via AGENTS.md routingYes, via prompt and shared policyYes, via code guidelines/path instructions
Workflow ownershipMostly product-ownedRepository-ownedProduct-owned with YAML configuration
Model flexibilityLimited from repo owner perspectiveStrongestProduct-controlled
DeduplicationProduct-managedMust buildProduct-managed
Trigger controlProduct settings / manual invocationFull GitHub Actions controlProduct settings / comments / YAML
Best use caseLow-friction AI PR reviewScalable AI-assisted CI orchestrationProductized AI code review
Main weaknessLess pipeline controlOperational burdenLess transparency into internal model decisions
Controlled-defect resultFound the seeded defectsFound the seeded defectsFound the seeded defects in the clean PR; inconsistent in the noisy mixed-bot PR

This was a controlled comparison, not a complete market survey. GitHub Copilot Code Review, Cursor Bugbot, Greptile, Qodo, and other review products may be worth evaluating, but I did not include tools that I had not tested on the same defects.

Cost Comparison

The pricing models differ, so the right comparison is not just “which one is cheaper per review.” The better comparison is how predictable the cost is and what operational work is included.

Hosted Codex Cloud

Hosted Codex review cost depends on the user’s ChatGPT/Codex plan and Codex credit model. It is not exposed as a simple repository-level per-PR API bill. OpenAI’s current Codex pricing documentation includes cloud-based automatic code review in supported ChatGPT plans, while API-key usage is separately token-billed and does not include the hosted GitHub code-review feature.

For an individual or small team already using Codex, this can be the lowest-friction option because it fits into the existing product experience. For a formal engineering cost model, it is harder to forecast per PR than an API-backed GitHub Action.

GitHub Action + Codex

The GitHub Action path has two costs:

  1. GitHub Actions runner minutes
  2. Model API usage

Runner cost is usually not the dominant factor for a small review job. The model cost is the meaningful part.

For a concrete API-cost reference, the table below uses the public GPT-5-Codex rate of $1.25 per 1M input tokens and $10.00 per 1M output tokens, with cached input at $0.125 per 1M tokens. The model page now marks GPT-5-Codex as deprecated, so these numbers should be read as a reproducible reference calculation, not as a recommendation to start a new production workflow on that model.

The experiment workflow did not pin a model, and the estimates below are not a measurement of its actual bill. A production workflow should explicitly choose an available model and recalculate with that model’s current prices.

The cost per review depends on:

  • Size of the diff
  • Amount of repository context included
  • Prompt length
  • Whether cached input applies
  • Number and length of findings
  • Whether the workflow reruns on every push

For planning, here is a rough uncached model-cost estimate:

Review shapeInput tokensOutput tokensApprox model cost
Small PR50,0005,000About $0.11
Medium PR150,00010,000About $0.29
Large PR400,00025,000About $0.75

These calculations assume uncached input and use the GPT-5-Codex API token prices above. Actual cost can be lower with prompt caching and higher if the workflow loads excessive repository context or reruns aggressively.

At a monthly level, the model-cost side looks like this:

Monthly PR review runsSmall PR estimateMedium PR estimateLarge PR estimate
100 reviews/monthAbout $11About $29About $75
250 reviews/monthAbout $28About $72About $188
500 reviews/monthAbout $56About $144About $375

GitHub Actions runner cost is additional, but for a lightweight Linux review workflow it is usually smaller than the model cost unless the workflow is slow or run at very high volume.

This approach can be very cost-efficient if reviews are scoped carefully. But if the workflow loads too much context or reruns aggressively on every commit, cost and PR noise can grow quickly.

CodeRabbit

CodeRabbit is primarily seat-based, not line-of-code based.

Public pricing lists:

  • Pro: $24/mo/user billed annually
  • Pro Plus: $48/mo/user billed annually

The docs describe per-developer rolling review limits:

  • Pro: 5 PR reviews per developer per hour
  • Pro+: 10 PR reviews per developer per hour

The current docs list a limit of 300 files per review for both Pro and Pro+.

Each review run consumes review allowance, including automatic incremental reviews after pushes and manual review commands such as @coderabbitai review or @coderabbitai full review.

This means CodeRabbit is not priced like “X cents per line reviewed.” You are paying for a productized review workflow, plus plan limits and optional usage-based expansion.

For a five-developer team, the annual-billing subscription comparison is:

OptionMonthly platform/model cost for 5 developersWhat this does not include
CodeRabbit Pro$120/monthUsage add-ons, if needed
CodeRabbit Pro Plus$240/monthUsage add-ons, if needed
GitHub Action + Codex, 100 medium reviews/monthAbout $29/month model costBuild/maintenance time, GitHub runner cost
GitHub Action + Codex, 250 medium reviews/monthAbout $72/month model costBuild/maintenance time, GitHub runner cost
GitHub Action + Codex, 500 medium reviews/monthAbout $144/month model costBuild/maintenance time, GitHub runner cost

So the GitHub Action path can be materially cheaper in direct model spend. But that is not the full cost. The engineering time to build and maintain inline publishing, deduplication, schema validation, security hardening, retries, and future integrations is real.

CodeRabbit costs more than the raw model calls, but it includes the product workflow. That is the main economic tradeoff.

Configuration Lesson: Keep One Review Policy

The best scalable pattern is not to duplicate long instructions across every tool.

I would organize the repository around four linked layers:

The responsibilities should be different:

  • AGENTS.md: tells coding agents and hosted Codex where the project guidance and review policy live.
  • The shared review policy contains the durable, provider-neutral priorities and finding threshold.
  • The Action prompt contains the GitHub Action execution contract, including JSON output requirements.
  • The CodeRabbit YAML contains CodeRabbit-specific configuration, review profile, code-guideline references, and short path instructions.

The important point is that the CodeRabbit YAML should not become a second full review policy. But I also would not remove path instructions entirely. In the successful CodeRabbit run, its inline comments cited path instructions as the source. So the better approach is to keep path instructions short and targeted while keeping the broader policy in the shared review-policy file.

My Recommendation

If I were adopting this in a real engineering workflow, I would not choose only one answer for all teams.

For a small team that wants quick value, I would start with hosted Codex Cloud review or CodeRabbit.

For a team that wants a polished productized PR review experience, I would evaluate CodeRabbit seriously. Its value is not only the model. Its value is the review workflow around the model.

For a platform team building a broader AI-assisted CI system, I would invest in the GitHub Action approach. It is more work, but it becomes the orchestration point. The same pipeline can later combine:

  • AI code review
  • Static-analysis output
  • Security scan output
  • Test failures
  • Coverage changes
  • Dependency risk
  • Runtime or dynamic-analysis signals

Then AI can do what it is better at: interpret multiple signals, identify risk patterns, explain impact, and help the human reviewer make a better decision.

What I Would Not Do

I would not ask a generic coding agent to “review this PR” with no policy, no structured output, and no integration plan.

That may produce useful comments sometimes, but it is not a CI/CD architecture.

For AI review to work inside CI, it needs:

  • A clear review role
  • Repository-specific guidance
  • A defect threshold
  • Inline comments or structured output
  • Deduplication
  • A trigger model
  • A human review handoff
  • Cost controls
  • A way to evolve the policy over time

The model is only one part of the system.

Conclusion

This experiment changed how I think about AI code review.

A dedicated code review product is not necessarily better because it has a different model. A repository-owned coding agent workflow is not automatically better because it is more flexible.

The real tradeoff is operational.

Hosted Codex Cloud gives a low-friction review path. A GitHub Action gives maximum control and model flexibility, but requires custom engineering. CodeRabbit gives a productized review workflow, but the internal decision-making is less transparent and the buyer does not choose the exact model stack.

For now, my preferred direction for a scalable CI architecture is:

  1. Keep review policy in the repository.
  2. Use productized review tools where they reduce operational burden.
  3. Use a repository-owned action when orchestration, model flexibility, and CI integration matter.
  4. Treat AI as a reviewer and interpreter, not as a replacement for deterministic CI tools.

That fourth point is the bridge to the next article: how deterministic static analysis, security checks, and tests can produce evidence, and how AI can turn that evidence into a concise report for the final human reviewer. In that larger design, AI code review becomes one input to the decision rather than the decision itself.

References

Constrain, Adapt, Evaluate: Rethinking Coding Assessments for the AI Era

AI, , , , ,

AI is becoming part of everyday software engineering. Developers use it to understand unfamiliar code, explore alternatives, generate tests, debug failures, and accelerate implementation. A technical assessment that asks candidates to pretend these tools do not exist is becoming less representative of the work we actually expect engineers to do.

But the opposite approach creates a different problem. If a candidate receives an unrestricted coding agent that can inspect the entire task, identify every defect, rewrite complete files, and prepare the final explanation, what exactly is the assessment measuring?

This tension led us to build SignalLoop, an AI-native candidate evaluator for software engineering hiring. SignalLoop is a collaboration between me, Sreenivas Makam, and my project partner, Ritesh Dhoot. The ideas, product direction, and implementation grew through our work together.

SignalLoop started as a proof of concept. It has grown into a runnable reference implementation with a hosted pilot, but it is not a production-grade hiring system. We are publishing it to demonstrate and test the assessment model, make the design discussable, and invite others to help improve it—not to suggest that it is ready to be adopted unchanged for consequential hiring decisions.

SignalLoop includes the things any usable assessment platform needs: employer and candidate experiences, administration, invitations, test execution, dashboards, scoring, and evidence reports. Those components matter, but they are not the central idea.

The differentiated part of SignalLoop rests on three design choices:

  1. Constrain AI collaboration without pretending AI does not exist.
  2. Adapt the assessment to the company and role while preserving comparability across candidates.
  3. Evaluate how the candidate worked, not only the final output.

We think of these as Constrain, Adapt, and Evaluate.

SignalLoop is also a work in progress. Some parts of this model are implemented and running today. Others are foundations or future directions that we are actively exploring. This article describes both, because the open questions are as important as the software already built.

The architecture behind the three ideas

The portals and dashboards are delivery surfaces around a deeper assessment loop:

Company context + role + JD
              ↓
Comparable role-level assessment blueprint
              ↓
Candidate works with a constrained AI collaborator
              ↓
Code, tests, prompts, snapshots and decisions are captured
              ↓
Technical evaluation + FAVO interpretation
              ↓
Evidence-based employer report

Each layer is designed to preserve a different kind of signal: relevance, candidate ownership, and evidence about the engineering process.

1. Constrain: allow AI collaboration without allowing AI delegation

There are two easy positions to take on AI in coding assessments.

The first is to prohibit it. That preserves the familiar assessment format, but it increasingly creates an artificial environment. A candidate may be evaluated under conditions that no longer resemble how the employer expects engineers to work.

The second is to allow an unrestricted assistant. That is realistic in one sense, but it can erase the distinction between a candidate who directs and verifies the work and one who delegates the entire task.

SignalLoop takes a position between these extremes. The candidate receives an AI collaborator inside the assessment workspace, but that collaborator is designed to be a coach, not an autopilot.

The assistant can:

  • explain candidate-visible code and public test output,
  • clarify concepts,
  • suggest a general debugging approach,
  • help reason through one candidate-identified issue,
  • compare tradeoffs the candidate has already framed,
  • provide bounded implementation help as the candidate demonstrates understanding.

It cannot:

  • enumerate every defect in the assessment,
  • produce the complete solution,
  • rewrite whole files,
  • reconstruct hidden tests,
  • reveal scoring internals or evaluator material,
  • make the candidate’s design decisions,
  • write the final explanation on the candidate’s behalf.

This boundary is enforced in multiple layers. A deterministic pre-gate catches obvious protected requests. A policy-classification component evaluates the intent of the interaction. A separate response-generation component produces help within the allowed boundary. Progressive disclosure lets the assistant become more specific when the candidate has identified the problem and articulated an approach.

SignalLoop also applies an anti-decomposition rule. A candidate should not be able to turn one disallowed request into a full solution by splitting it into a sequence of smaller prompts such as “list every issue,” followed by “give me the code for each issue,” followed by “write every missing test.” The system considers the combined effect of the interaction, not only each message in isolation.

There is another boundary that matters just as much: the AI receives only candidate-visible information. Hidden tests, seeded issue lists, reference solutions, evaluator notes, and scoring internals are never part of the assistant’s context.

This constrained collaborator is implemented today. It is not perfect, and policy evaluation will require continued testing and calibration, but it lets us ask a more useful question than “Did the candidate use AI?”

The better question is:

How did the candidate divide responsibility between themselves and the AI?

A prompt such as “Find every bug and fix the project” tells us something different from “The duplicate-email test is failing; I think normalization is missing. What should I inspect before I change the behavior?”

The interaction itself becomes evidence.

2. Adapt: tailor the assessment to the role, not arbitrarily to each candidate

Most engineering assessments face a relevance problem. A single generic coding exercise is easy to administer and compare, but it may be only loosely related to the role. A fully personalized assessment may be more relevant, but it can destroy comparability.

SignalLoop’s design principle is:

Adapt across companies and roles. Preserve consistency within a hiring cohort.

The intended role-level blueprint is derived from inputs that belong to the hiring decision:

Company context
+ hiring area
+ role and seniority
+ job description
+ cognitive areas to emphasize
+ target duration
→ role-level assessment blueprint

Company context matters because the same job title can imply very different engineering work.

A backend engineer at a fintech company may need stronger evidence around authorization, auditability, data boundaries, and failure handling. A backend engineer at an AI-infrastructure company may need more emphasis on control-plane APIs, observability, reliability, deployment tradeoffs, and safe AI-assisted debugging.

The JD may be similar, but the assessment emphasis should not necessarily be identical.

Once the employer reviews and approves a blueprint, however, every candidate for that role should receive the same scored assessment. This creates a stable evidence surface for comparison. An employer can create a new version of the role assessment, but SignalLoop should not silently generate an easier or harder scored task for each applicant.

Candidate resumes can still add value. They can inform:

  • skill gaps to probe during an interview,
  • claims that require validation,
  • candidate-specific follow-up questions,
  • report caveats and interviewer notes.

They should not initially determine:

  • which scored questions the candidate receives,
  • the scoring rubric,
  • the time allocation,
  • the required coverage for the role.

This distinction is important. Personalization is not automatically fairness. A candidate-specific test may look intelligent while making employer comparisons less defensible.

What exists today

SignalLoop currently implements guided role matching. An employer can provide the role, JD, seniority, team or domain context, expected AI usage, and an optional resume. The system maps the inputs to a versioned skill taxonomy, recommends the closest supported assessment pack, and explains:

  • directly tested skills,
  • partially tested skills,
  • unsupported skills,
  • the rationale for the match,
  • suggested interview follow-ups.

The current executable coverage is deliberately narrow: SignalLoop can recommend a Standard or Advanced FastAPI assessment. If a frontend or data-engineering JD is outside current executable coverage, the product says so instead of pretending that a backend score evaluated those skills.

The project therefore demonstrates the role-matching model today; it does not yet claim to be a complete dynamic question composer. The consolidated roadmap later in this article explains that next layer.

3. Evaluate: measure the engineering process, not only the output

Traditional coding assessments typically emphasize the final artifact:

  • Did the tests pass?
  • Is the code readable?
  • Was the task completed on time?
  • What was the final score?

Those signals remain important. AI-assisted engineering, however, introduces additional questions.

  • Did the candidate understand the problem before changing code?
  • Did they recognize ambiguity and risk?
  • Did they ask AI focused questions or delegate wholesale?
  • Did they verify AI-assisted changes?
  • Did they add meaningful tests?
  • Can they explain and defend the final result?

SignalLoop organizes this process evidence through FAVO: Frame, Ask, Verify, Own.

Frame

Did the candidate understand the problem, constraints, risks, and ambiguous requirements? Did their implementation reflect deliberate prioritization and product judgment?

Ask

How did the candidate use AI? Were the questions focused and grounded in an identified behavior, or did the candidate attempt to outsource the complete solution?

Verify

Did the candidate run tests, add tests, inspect edge cases, revisit assumptions, and validate changes after receiving AI assistance?

Own

Could the candidate explain what changed, why they made particular tradeoffs, what remains uncertain, and what they would improve next?

Candidates do not manually write a FAVO score. SignalLoop derives the interpretation from captured evidence such as:

  • code snapshots,
  • public and hidden test runs,
  • candidate-written tests,
  • AI conversations and policy redirects,
  • large code-paste signals,
  • final submission-review answers,
  • the consistency between the explanation and submitted code.

The Engineering Evidence Report combines this with the technical evaluation. FAVO is not intended to replace correctness, and it is not a personality or psychometric score.

The technical score tells the employer what worked. FAVO helps explain how the candidate got there and whether the process is trustworthy.

An initial deterministic version of this interpretation is implemented today. Some signals are necessarily simple proxies: prompt counts, test runs, candidate test files, feature behavior, hidden-test status, and submission-review completeness. Improving and validating those mappings is part of the consolidated roadmap below.

What exists today and what comes next

Rather than scatter the project status across the three ideas, here is the boundary in one place.

Constrain

Implemented today: A constrained assistant, candidate-visible context boundary, policy classification, progressive disclosure, anti-decomposition behavior, interaction logging, and evidence capture.

Next: Stronger adversarial testing, policy-evaluation datasets, progressive-disclosure calibration, and better measures for distinguishing productive coaching from disguised delegation.

Adapt

Implemented today: Guided role matching from role, JD, seniority, team or domain context, and optional resume information to the closest registered assessment pack. The result includes tested, partially tested, and unsupported skill coverage. The question-bank governance foundation also supports provenance, draft review, and approval.

Next: Compose role-level assessments from approved and calibrated questions:

Role/company/JD/cognitive requirements
              ↓
Required assessment slots
              ↓
Approved question bank
              ↓
Reviewable role-level blueprint
              ↓
Employer approval or same-slot swaps
              ↓
Reusable assessment for the candidate cohort

This requires connecting the question bank to employer blueprint creation, candidate delivery, and evidence-report scoring while preserving the same scored assessment for candidates in the same cohort.

Evaluate

Implemented today: Deterministic technical scoring, public and hidden test evidence, AI-interaction evidence, submission review, timeline capture, and an initial FAVO interpretation.

Next: Better evidence mappings, evaluator calibration, longitudinal validation, clearer evidence limits, and research into which signals are reliable, useful, fair, and resistant to superficial gaming.

Seeing the complete workflow

The short demo below shows the current end-to-end SignalLoop experience: super-admin visibility, employer assessment setup, the candidate workspace with constrained AI, and the resulting evidence report.

What SignalLoop deliberately does not claim

Being explicit about the boundaries is important.

SignalLoop does not treat AI use itself as misconduct. It evaluates how the candidate uses it.

It does not allow the embedded assistant to complete the assessment or access evaluator-only artifacts.

It does not currently give every candidate a different scored test. The role-level assessment is intended to remain comparable across the cohort.

It does not convert unsupported skills into implied evidence. If the current assessment cannot test an area, the report should identify the gap.

It does not reduce a hiring decision to one number. The report is evidence for employer review, not an autonomous hiring decision.

And it does not yet implement the entire adaptive question-composition vision described here.

From proof of concept to production-grade system

The distinction between a working reference implementation and a production system matters, especially in hiring.

SignalLoop began as a POC to explore whether constrained AI collaboration and process evidence could produce a better assessment signal. The current system demonstrates the complete workflow: employers can create invites, candidates can work in a browser with a constrained assistant, tests and interactions are captured, and evidence reports can be generated.

That proves the product loop. It does not complete the production journey.

A production-grade version would still need work in areas such as:

  • isolated and hardened execution infrastructure for untrusted candidate code,
  • private, calibrated assessment content rather than public demo packs,
  • production authentication, secrets management, monitoring, backups, and incident response,
  • explicit data-retention, consent, privacy, and compliance policies,
  • accessibility, abuse testing, rate-limit hardening, and operational support,
  • evaluator calibration and studies of reliability, validity, bias, and adverse impact,
  • question effectiveness analytics and ongoing assessment-version governance,
  • human review processes appropriate for consequential hiring decisions.

Some technical scaffolding and product foundations for these areas exist in the repository. Others remain design and validation work. We would not recommend using the current project as an autonomous hiring system, and we do not believe any evidence report should replace accountable human judgment.

How you can help

The project is a working reference implementation and a foundation for further exploration. The roadmap above is intentionally open. We are particularly interested in privacy, fairness, question effectiveness, assessment versioning, policy evaluation, and methods for evaluating AI-assisted engineering without rewarding superficial activity.

SignalLoop is available on GitHub. It is joint work by Sreenivas Makam and Ritesh Dhoot, and the next phase of the project is intentionally open to a wider community of contributors and critics.

If you are working on technical hiring, assessment design, AI safety, developer tools, or evidence-based evaluation, we would value your feedback. Contributions, critique, experiments, question-bank ideas, evaluator studies, and help with any of these future directions would be genuinely appreciated.

The central idea is simple:

Do not ask whether candidates used AI. Design the assessment so that their use of AI produces meaningful evidence.

Constrain the collaboration. Adapt the assessment. Evaluate the process.

Try It Yourself

Explore the hosted SignalLoop application to experience the assessment flow firsthand. SignalLoop is fully open source—you can inspect the implementation, run it locally, contribute, or adapt it to your own hiring process in the GitHub repository.