Tag Archives: openrouter

Model routing is three decisions, made on your own data: 1. Model pool (GPT-5 nano and GPT-5), 2. Classification (SIMPLE to nano, COMPLEX to GPT-5), 3. Scope (per conversation or per question).

Model Routing, Part 1: Choosing a Classifier

AI, , , , , ,

Objective

How do you choose a model router for your own application? I tested six classifiers on a marketplace support agent to compare cost against answer quality. This small study demonstrates the decision process, not a general ranking of products.

Both GPT-5 nano and GPT-5 answered 18 support cases: 16 single-turn and two two-turn cases, or 20 answer turns per model. The LLM classifier and JEV each selected fully correct answers for 88.9% of cases, versus 94.4% for GPT-5 alone and 50% for nano, at an estimated 55–58% of GPT-5’s cost.

Part 2 examines how long a routing decision should last across a conversation.

The Use Case

I built a simulated marketplace with buyers, sellers, orders, shipments, payouts, returns, refunds and versioned policies. Its internal support agent answers questions using read-only SQL and policy search, citing its evidence. It runs on the OpenAI Agents SDK, through LiteLLM, with Langfuse tracing; payments and business data are simulated.

The workload mixes quick lookups—“What products are in order-2?”—with investigations such as “Why hasn’t this seller been paid?” The latter can require payment records, delivery events, policy and a time calculation.

Routing Is Three Decisions

Model routing is not one setting. It is three decisions:

Three boxes: 1. Model pool (a fixed, approved set of models; ours are GPT-5 nano and GPT-5), 2. Classification (label each request, then map each label to a model; ours map SIMPLE to nano and COMPLEX to GPT-5), 3. Scope (per conversation, per question, per model call, named stage). Below, one box lists what a gateway such as LiteLLM can manage: allowed models, built-in classifiers, per-conversation selection through session affinity, per-question selection through user_turn and per-model-call selection through every_request. Another box lists what the application must supply: session identity, structured messages and named stages. A note says that in the conversation-granularity experiment the author's own runner did the classification, selection and pinning, and LiteLLM forwarded each request to the selected model.
Routing is three decisions. A gateway can manage most of them; the application supplies session identity and any named stages.
  1. Model pool: which approved models the router may choose between.
  2. Classification: how each request is labeled, and which model each label maps to.
  3. Scope: how long a routing decision lasts, such as a whole conversation, a single question or a single model call. That is Part 2.

Choosing and Fixing the Model Pool

Shortlist approved models on benchmarks, domain fit, cost and latency, then run evals on your own golden set. Candidates must support the agent’s tool calls and output formats. Cost per correct answer matters more than token price alone.

My evaluated pair was GPT-5 nano, the lower-cost candidate, and GPT-5, the stronger performer. Both use the same provider behind one gateway. I did not run public benchmarks myself; the choice rested on workload evals, cost and latency.

Keep routing within the evaluated, approved pool and pin model versions for reproducibility. Even OpenRouter Auto was restricted to these two candidates.

How I Checked the Quality

A router can only be judged as well as the grades behind it, so grading came first.

I built a golden set: cases with known business data behind them and expected answers written in advance. A case counts as fully correct only if every requested fact is right, in every turn of the case.

Grading combined human review and LLM review:

  • Calibration, earlier: in an earlier model comparison, I graded 36 answers by hand and had Claude Opus 5.5 grade them as an LLM judge. After clarifying the grading rules, the judge agreed with my grade on 31 / 36 (86%). That calibration applies to that judge on those answers.
  • This comparison: the nano grades are existing accepted grades. The GPT-5 answers were reviewed by an LLM reviewer, GPT-6 Astra, against the expected answers, and I adjudicated the borderline cases. The Claude calibration does not validate that reviewer; the adjudication is what anchors those grades.

Baselines: What Routing Has to Beat

First, the two models on their own over the 18 cases. The cases cover order lookups, policy questions (current and historical), payout eligibility, refund histories, incidents, reconciliation, an ambiguous name, and a request the agent must refuse.

ModelCases correct / partial / incorrectFully correctEstimated cost, all 18 cases
GPT-5 nano9 / 7 / 250.0%$0.023
GPT-517 / 1 / 094.4%$0.456

GPT-5 was far more accurate and cost about 20× more. That gap is what makes routing worth considering. The question routing has to answer is simple: how much cost can we save, and how much accuracy do we give up?

Classification: Labels Are a Design Choice

A classifier assigns a label to each request. A separate mapping turns that label into a model. Both are your choices.

  • Labels don’t have to be about difficulty. They can be task type (lookup, investigation, action request), business domain (payments, returns, fulfillment), risk, language, or a mix.
  • “Simple” doesn’t have to mean “cheapest”. A short but risky request such as “refund this order now” could be labeled simple and still be mapped to the strong model.
  • A pool can have more than two tiers.

For this study, I used the simplest useful setup for a two-model pool: SIMPLE → GPT-5 nano and COMPLEX → GPT-5, for the classifiers I configured. LiteLLM’s built-in tiers also include MEDIUM and REASONING; I mapped everything above SIMPLE to GPT-5. OpenRouter’s Auto Router is the exception: it does not use my labels at all (see below).

The Six Classifiers and How I Configured Them

Every rule, example and prompt below was written from the support-user journeys in my specification: the kinds of questions support staff actually ask. They were frozen before testing. The one exception is a separately versioned refinement of the semantic examples, which I made after seeing the first results. It is reported on its own and is not part of the results table.

How each one ran. For the first five, my own runner produced or collected the classification and chose the model. OpenRouter Auto made its own choice. None of this used a gateway’s automatic routing on live traffic.

  • Keyword rules, heuristic scorer and semantic similarity: LiteLLM’s native classification code, exercised separately. It was not routing live traffic through my gateway.
  • LLM classifier: GPT-5 nano called through my LiteLLM gateway.
  • JEV: called directly through OpenRouter’s Decisions API.
  • OpenRouter Auto: called through OpenRouter.

Keyword Rules

LiteLLM’s keyword rules map words to tiers without calling any model. My lists:

keyword_tier_rules:
  - tier: SIMPLE
    keywords: [list, show, quantity, stock, balance, price]
  - tier: COMPLEX
    keywords: [why, investigate, reconcile, history, policy, eligible, eligibility,
               refund, failed, incident, cause, "can you", "can the", "can we"]

Simple words signal a direct lookup. Complex words signal explanation, investigation, policy, eligibility, money movement, or a request to act. If both lists match, the higher tier wins. If nothing matches, the question falls back to the heuristic scorer below.

Heuristic Scorer

This is LiteLLM’s default classifier, and I used its defaults unchanged on purpose, to see how a generic heuristic behaves out of the box. It scores seven text signals that were built with software questions in mind:

  • code words (weight 0.30)
  • reasoning phrases like “step by step” (0.25)
  • technical terms (0.25)
  • length (0.10)
  • simple phrases like “what is” (0.05)
  • “first… then” patterns (0.03)
  • several question marks (0.02)

LiteLLM lets you add your own technical keywords, which is how you would adapt it to a domain.

Semantic Similarity

LiteLLM can also match questions by meaning. I wrote 14 example sentences, embedded with text-embedding-3-small. A question takes the label of its closest example if the similarity is at least 0.5; otherwise it falls back to the heuristic. A few of the examples:

  • SIMPLE (6 examples): “Show the products purchased in this order and their quantities.” · “What is the listed price of this product?” · “List the sellers and their current outstanding amounts.”
  • COMPLEX (8 examples): “Explain why the buyer paid but the merchant has not received the money.” · “Determine whether delivery and the waiting period permit a payout now.” · “Can you approve a refund and transfer the money on my behalf?”

LLM Classifier

Here a small model, GPT-5 nano, reads a routing prompt and returns SIMPLE or COMPLEX with a one-line reason. The prompt has five parts:

  1. Purpose: classify the evidence and reasoning a correct answer needs, “not sentence length or how easy the answer sounds”.
  2. The application and its tools: read-only SQL, policy search, event history.
  3. “What makes business questions deceptively difficult”: eight domain traps. Examples: “Buyer payment, seller payout and buyer refund are different money movements”; “Current payout/refund rows can hide failed attempts preceding a success”; “Recorded state, eligibility and successful execution differ.”
  4. Decision rules: SIMPLE only for direct, bounded lookups. COMPLEX for causal explanation, event reconstruction, policy applied to live data, time-boundary calculations, reconciliation or ambiguity. “A short question can be COMPLEX. A long factual list can be SIMPLE.”
  5. Generic data definitions: the application’s entities, relationships and metrics, without any test data.

A typical pair of decisions:

  • “For order-2, list the product, quantity and seller” → SIMPLE: “Direct bounded retrieval… No complex inference or policy.”
  • “Explain why its seller has not received the payout” → COMPLEX: “requires tracing allocations, payments, shipments, returns/refunds, and payout status, plus policy-based eligibility.”

JEV

JEV is TypeSafe’s decision model, which I wrote about in Right Model, Right Job. I gave it the same routing rubric, adapted to JEV’s typed API. The rubric text went in as the instructions of a typed SIMPLE/COMPLEX choice question, with the first line changed to “Choose SIMPLE or COMPLEX” instead of asking for a reason. That keeps the two close to a like-for-like comparison.

JEV returns a typed choice and a probability for each option:

{"tier": {"type": "choice", "choice": "COMPLEX",
          "probabilities": {"SIMPLE": 0, "COMPLEX": 1}}}

That was its answer for “Why has Cedar not been paid for order-1?”

Both the LLM classifier and JEV ran in standalone mode rather than through LiteLLM’s built-in auto-routing. LiteLLM supports both an LLM classifier and JEV as built-in options, so the same setup can be configured inside the gateway.

OpenRouter Auto

OpenRouter’s Auto Router works differently from the other five. There is no prompt or rule set to write, and it doesn’t use my SIMPLE/COMPLEX labels. OpenRouter applies its own task classification, then ranks candidate models using its market data.

  • I restricted it to only my two models (allowed_models: openai/gpt-5-nano, openai/gpt-5).
  • I supplied the cost tier myself, trying low and medium.
  • Both settings selected nano for every case.

That is a finding about this restricted two-model configuration. It is not evidence that OpenRouter’s router performs poorly in general.

The Results

Here is how each approach did on the same 18 cases. Costs are replay estimates for all 18 cases at full list price, and they include the classifier’s own cost. The keyword and semantic rows include their heuristic fallback.

ApproachCases correct / partial / incorrectFully correctnano / GPT-5 picksEstimated costSaving vs always GPT-5
Always GPT-5 nano9 / 7 / 250.0%18 / 0$0.02394.9%
Always GPT-517 / 1 / 094.4%0 / 18$0.456—
Keyword rules15 / 2 / 183.3%8 / 10$0.24047.4%
Heuristic (default)9 / 7 / 250.0%17 / 1$0.03891.7%
Semantic similarity13 / 4 / 172.2%13 / 5$0.13869.8%
LLM classifier16 / 1 / 188.9%9 / 9$0.24945.3%
JEV16 / 1 / 188.9%8 / 10$0.26641.7%
OpenRouter Auto (low and medium)9 / 7 / 250.0%18 / 0$0.02394.9%
Scatter plot titled "18-case saved-answer replay: case-level correctness vs estimated cost". Always GPT-5 is top right at 94.4% and $0.456. The LLM classifier and JEV sit at 88.9% around $0.25 to $0.27, keyword rules with heuristic fallback at 83.3% and $0.240, semantic similarity with heuristic fallback at 72.2% and $0.138. The default heuristic, always nano and OpenRouter Auto all sit at 50% near $0.02 to $0.04.
On these 18 cases, classifiers written for the domain came closest to GPT-5’s case-level correctness at about half its estimated cost.

On this tested set:

  • The domain-aware classifiers did better than the generic ones. The LLM classifier and JEV share one domain-aware routing rubric. They kept 16 of GPT-5’s 17 fully correct cases at 55–58% of its estimated cost. The default heuristic and the restricted OpenRouter configuration selected nano almost every time and scored the same as nano alone.
  • Two short keyword lists got surprisingly close: 83.3% against 88.9%, at similar cost and with no model call.
  • Separate the classifier’s cost from the route’s cost. JEV’s 18 classification calls cost less than the LLM classifier’s (about $0.0038 versus $0.0054), and each decision was faster (about 0.7 s versus 2.7 s). Its route still cost more overall, because JEV selected GPT-5 for 10 cases against 9. The extra one was a case nano had already answered correctly.
  • Semantic matching depends on its examples. 10 of 18 cases matched no example closely enough and fell back to the heuristic. In the separately versioned refinement, I added more examples: fallbacks dropped to 6, but correctness did not improve and cost went up.
  • Routing did not outperform fixed GPT-5 here. That isn’t a law. Routing can beat the best fixed model when different models succeed on different cases. In this pair, GPT-5 was also right on every case where nano was right, so there was nothing for routing to gain over GPT-5 alone.

Right vs Wrong: Three Examples

Averages hide the interesting part, so here are three cases in detail.

1. A short question isn’t a simple question. “Why has Cedar not been paid for order-1?” is only eight words, and the default heuristic scored it as simple and sent it to nano. The real answer needs the payout records: the seller’s payout had failed because of a provider timeout. Nano’s answer did not mention the timeout and was graded partial. GPT-5 identified the recorded provider timeout and was correct. The keyword rules (“why”), the LLM classifier and JEV all sent this question to GPT-5. Generic signals measure how a question looks; domain-aware ones try to measure what answering it takes.

2. Two identical questions, routed differently. Two questions differed only in the order ID: “Explain the refund history for ord-6f3604a5fe.” and “Explain the refund history for ord-aaa1768930.” In the refined semantic version, the first scored 0.529 and the second 0.493 against the closest example, on opposite sides of the 0.5 threshold.

  • The first went to GPT-5, although nano’s answer to it was already fully correct. That was wasted cost.
  • The second fell back to the heuristic and went to nano, whose answer was partial while GPT-5’s was correct. That was lost accuracy.

It was wrong in both directions, because of an order ID. Similarity thresholds can be sensitive to details that don’t change the task.

3. A keyword list is only as good as its coverage. “After returning one Canvas Weekender, what remains payable to Cedar?” is a money calculation: take the seller’s share of the sale, then adjust for the refunded item and the fee. Its wording (“returning”, “payable”) wasn’t in either keyword list, so it fell back to the heuristic and went to nano. Nano’s answer was partial; GPT-5 computed the correct amount, ₹2,249.10. Rules are cheap and transparent, but this gap led to a routing mistake. A fallback can sometimes still choose correctly; here it didn’t.

Lessons Learned

Domain-aware configurations did better on this set. Two keyword lists reached 83.3%, versus 88.9% for the LLM classifier and JEV. That makes simple rules a useful baseline, with their coverage gaps tested explicitly.

The LLM classifier and JEV agreed on 17 of 18 choices. JEV was faster and cheaper per decision, but agreement does not establish which part of the setup caused the benefit. Separate classifier cost from selected-model cost: a cheap classifier can still produce an expensive route by choosing GPT-5 more often.

A Checklist for Choosing a Classifier

  1. Shortlist models on benchmarks, domain fit, cost and latency, from approved providers that support your agent’s needs.
  2. Run evals on your golden set to narrow the shortlist. Then fix the pool and pin the versions.
  3. Weigh cost savings against accuracy. If the cheaper model meets your accuracy bar, use it. If only the stronger one does and its cost is acceptable, use it. Route only when routing saves meaningful cost while keeping accuracy within your bar.
  4. Design the classification. Choose labels that matter for your traffic and map them to models. Start with keyword rules, then try semantic or LLM-based classifiers built from real user journeys, not from test questions. Compare them on the same golden set, and look at both classifier cost and route cost.

Part 2 adds the remaining steps: choosing the scope on real conversations, and validating before you deploy.

Limits and What’s Next

This was a small study, and I would not treat it as a benchmark:

  • Sample size: 18 cases (20 answer turns per model), one run each.
  • Grading: the earlier calibrated judge, existing accepted grades, and an LLM reviewer with my adjudication of borderline cases.
  • Tuning: the rules, examples and prompts were written by someone who knew the test cases.
  • Costs: replay estimates at full list price.

The point is the process, not the percentages. Next, I want to:

  • test on fresh held-out cases, with repeated runs
  • add a middle-tier model
  • try routers I haven’t tested yet: Not Diamond, which learns from your graded data; Aurelio’s semantic-router; RouteLLM; Arch-Router; and the cloud platforms’ own model routers

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude Code, which also helped me research the routing landscape and plan the experiments. The application and experiments were built and run with GPT-6 Astra, which also reviewed answers against the expected results. Claude Opus 5.5 served as the calibrated LLM judge in the earlier model comparison.

References

This Series

Earlier Posts

Routing Tools

For Future Work

Right Model, Right Job: A Daily AI Digest with Jev and Claude at 1/10th the Cost

AI, , ,

Objective

The goal of this post is to show, with real numbers from a working system, how to pick the right model for each step of an AI workflow:

  • Where a decision model like Jev fits, and where an LLM is still the right tool.
  • How much that split saves compared to using an LLM for everything, on cost and on speed.
  • How to check that the cheaper model is good enough, by comparing its decisions against a strong LLM on the same inputs.
  • How to run it on a schedule in the cloud, using either Claude Code routines or GitHub Actions, and what each one is like to operate.

The example is a daily AI blog digest I built for myself. I used Jev, TypeSafe’s decision model, to judge every post, and Claude only to summarize the few that made the cut. The whole run costs about four cents a day. The pattern applies to any workflow that judges a lot of items and writes about a few.

The code is on GitHub: smakam/ai-blog-digest.

The Use Case

I follow around two dozen AI blogs through Feedly: frontier labs, research groups, a few people whose writing I trust, and some general tech and startup news. On a typical day that is 60 new posts. Maybe five of them are worth my time. The problem is not reading; it is deciding what to read.

So I built a small daily job that does the deciding for me. The requirements were short:

  1. Input: my Feedly subscriptions, exported as OPML. Only posts from the last day or so.
  2. Classify each post on two questions: is it about AI, and is it worth reading?
  3. Summarize only the best posts, 4–5 lines each: what it is, why it matters, the key takeaway.
  4. Deliver one Telegram message a day at 7 AM: top picks with summaries, a few more as plain links.
  5. Never send the same post twice, and never fail silently.
  6. Run in the cloud, without my laptop on.

One detail shaped the classifier: the worthiness judgment is about content only. Who wrote the post is not an input (there is a hook for an author boost later, but it is off).

The Question I Wanted to Answer

Can a purpose-built decision model replace an LLM for the high-volume, low-glamour part of an AI workflow, and what do you actually give up?

A digest is a good test case because it has a very lopsided shape:

  • Classification runs on every post. About 60 a day, each one a full article that has to be read in full.
  • Generation runs on almost nothing. Five summaries a day.

If you use one LLM for both, you pay LLM prices for the 55 posts you throw away. That is where nearly all the cost goes.

System One and System Two: Picking the Right Model for Each Job

TypeSafe describes Jev as a System One model. It does not generate text. You give it some state (here, the post’s title and text) and a set of typed questions, and it returns typed answers with probabilities:

  • a noul (yes/no) returns the probability of yes;
  • a score places the state on an ordered scale you describe in words and returns the expected level plus a probability per level;
  • a choice picks one option from a set.

I think of the LLM as the System Two side of this system (that is my framing, not TypeSafe’s): slower, more expensive, and the right tool when you need language out, not a decision.

JobVolumeWhat it needsModel
Is this post about AI? How worthwhile is it?Every post (~60/day)A calibrated decision, fast and cheapJev (System One)
Summarize the best postsTop 5/dayFluent, specific writingClaude Haiku 4.5 (LLM)

I used Haiku 4.5 for summaries because it is the cheapest Claude model and the summaries are short. The summarizer is a single model ID in the config, so Claude Sonnet or Opus can be dropped in when you want richer summaries. I ran Sonnet 5 for part of the testing; the cost difference is in the table further down.

Both are reached through OpenRouter, so one API key covers the whole pipeline. Jev uses OpenRouter’s Decisions API rather than chat completions. I call it through TypeSafe’s official Python SDK with the base URL pointed at OpenRouter, so moving to TypeSafe’s native API later is a base-URL and key change.

How It Works

How the daily digest works
  1. Fetch. Parse the OPML, fetch every feed, and keep posts from the last 36 hours that haven’t been processed before. (36 rather than 24, so a late or skipped run doesn’t lose posts; the seen-post record prevents repeats.)
  2. Get the text. Use the full text from the feed if it’s there, otherwise fetch the article and extract the main text, otherwise fall back to the title and feed summary. On a typical day, 62 of 64 posts had full text.
  3. Classify with Jev. One call per post, two questions. Text is capped at about 20K tokens because Jev’s context is 32K.
  4. Apply the policy in code. This is deliberately not in the model (details below).
  5. Summarize the top 5 High posts with Claude Haiku 4.5.
  6. Send one Telegram message, save the seen-post record and a per-post log, and send a short notice if anything failed.

The Classification Criteria

This is the actual question and rubric Jev gets. The whole “prompt engineering” for the classifier fits on one screen.

Is it about AI? (noul)

Is this blog post primarily about artificial intelligence: machine learning, LLMs, generative AI, AI agents, AI infrastructure, AI research, or AI products? Embodied AI, robotics, self-driving hardware, and humanoids count as NO.

With the two outcomes described as: yes means the post’s main subject is AI/ML software, models, research, tooling, or products; no means it is not mainly about AI, or it is about robotics or embodied AI.

So embodied AI, robotics, self-driving hardware, humanoids, and anything else where the AI lives in a physical machine are classified as no. Any other topic is also classified as no. I specifically called out embodied AI and Robotics as these were in my feedly.

How worthwhile is it? (score, lowest to highest)

How worthwhile is this post for an experienced AI engineer who wants technical insight and significant launches, not news noise? Judge the content only.

LevelDescription
0Not worth reading: funding or acquisition news, listicles, prompt-tip posts, generic opinion or hype pieces, pricing changes, marketing fluff.
1Low value: minor feature updates, shallow news recaps, or thin commentary with no technical substance.
2Moderate: a tutorial on known techniques, a modest product update, or informed commentary with some concrete specifics.
3High value: architecture detail, agent infrastructure, LLMOps practice, or a significant product launch (a new model, a new agent or developer framework, a major platform capability).
4Exceptional: genuinely new technical insight, such as original research results, deep architecture write-ups, or first-hand engineering lessons.

Jev returns a probability-weighted level, so a post can land at 2.8 rather than exactly 3. I divide by 4 to get a 0–1 worthiness score.

The call itself is short:

from typesafe_sdk import Noul, Score, TypeSafeClient
client = TypeSafeClient(api_key=OPENROUTER_API_KEY,
                        base_url="https://openrouter.ai/api", model="jev-1.13")
result = client.system_one(
    state={"title": post.title, "text": post.text[:80_000]},
    questions={"is_ai": IS_AI, "worthiness": WORTHINESS},
)
is_ai = result.nouls["is_ai"].noul                  # e.g. 0.97
worthiness = result.scores["worthiness"].score / 4  # e.g. 0.75

No JSON parsing, no “respond only with…”, no retries because the model added a sentence before the JSON. The answer is typed.

What the Script Does on Top of Jev

Jev answers the questions. The decisions stay in code, where they are easy to see and change:

  • Drop non-AI posts: is_ai below 0.5.
  • Bucket the rest by worthiness, with configurable thresholds:
BucketDefault thresholdWhat happens
High≥ 0.7Top 5 get a summary and link; the rest are listed as links
Medium0.4 to 0.7Title and link
Low< 0.4Not delivered (but logged)
  • Cap summaries at five. That keeps the digest to one Telegram message and caps LLM cost. High posts beyond the fifth are still delivered, as links.
  • Never send twice. Seen-post IDs are saved after each successful delivery. Posts Jev failed on are not marked seen, so the next run retries them.
  • Never fail silently. A broken feed, a Jev error, a summary error, or a failed delivery sends a short notice to Telegram.
  • Log every post with title, URL, source, both scores, bucket, where the text came from, latency, and cost. That log is what made the cost and quality comparisons below possible.

What It Costs

All numbers below come from real runs on my 24 feeds, priced by OpenRouter’s usage.cost field, not estimated from token counts.

Classification: Jev vs Claude Sonnet 5

I gave Jev and Claude Sonnet 5 exactly the same 59 posts, the same two questions, and the same rubric, and ran both through the same policy.

JevClaude Sonnet 5
Price (per million tokens)$0.042 input, output free$2 input, $10 output
Cost for 59 posts$0.006$0.41
Cost per 1,000 posts~$0.10~$6.95
Median latency per post0.34 s2.6 s

That is about 70× cheaper and 8× faster for the classification step.

Sonnet is a mid-tier model; a larger model would widen the gap. Cheaper LLMs narrow it but do not close it: from list prices, I estimate Gemini 3.5 Flash-Lite at around $0.06 for the same run (I did not measure this one), still roughly 10× Jev. And with a general-purpose LLM you also own the output format, the parsing, and the calibration.

The Whole Pipeline

SetupClassifySummarize (5 posts)Per runPer month
Jev + Claude Haiku 4.5 (what I run)$0.006$0.032$0.038~$1.15
Jev + Claude Sonnet 5$0.006$0.088$0.094~$2.80
Claude Sonnet 5 for both$0.41$0.088~$0.50~$15

So the split gives roughly 13× savings end to end, and 70× on the step that runs on every post.

At my volume these are small numbers; nobody will notice $15 a month. The point is the shape. Classification scales with how much you read; generation scales with how much you keep. If this were 10,000 posts a day for a team or a product, classification is the line that grows, and it is the line Jev takes off the bill.

How I Checked the Quality

Cheap is only interesting if the answers are good. I didn’t have hand-labeled data on day one, so I used Sonnet as a reference point. It is not ground truth, but it is a strong model reading the same text with the same rubric.

On the same 59 posts:

AgreementResult
Is it AI?58 / 59
Delivered or not (High/Medium vs dropped)55 / 59
Same bucket (High / Medium / Low / Not AI)50 / 59

Bucket by bucket (rows are Jev, columns are Sonnet):

HighMediumLowNot AI
High7110
Medium3330
Low0080
Not AI00132

The nine disagreements split into two kinds.

Four were near-misses. Posts that Jev scored between 0.53 and 0.69 and Sonnet put at High, or a funding story both of them dropped under different labels. Part of this is mechanical: Sonnet almost always picked a whole level (0, 0.25, 0.5, 0.75, 1), while Jev returns a probability-weighted position anywhere in between. Anything near a threshold will flip.

Five were real differences, and in every one Sonnet was the stricter judge. Jev put two Gemini product launches at High where Sonnet said Medium or Low, and put a Claude Code release note and two startup-news pieces at Medium where Sonnet said Low. You could argue either side on the launches (my own rubric counts “major platform capabilities” as High). On release notes and business news, Sonnet was closer to what I asked for.

The practical reading: Jev is slightly more permissive, and it never dropped anything Sonnet rated High. The worst case is a couple of extra links in the Medium list, not a missed post. If I want Jev closer to Sonnet, raising the Medium threshold from 0.4 to about 0.5 would catch some of it. I am going to decide that after a week of my own manual review, not from one day.

This was one day and 59 posts. I would not treat it as a benchmark. It was enough to convince me the cheap model is not trading away the part I care about.

Scheduling: Claude Code Routines vs GitHub Actions

The requirement was to run in the cloud with my laptop closed. I set up two options and ran the identical script on both, so any difference is the runtime, not the digest.

GitHub ActionsClaude Code routine
What runs the jobGitHub runs python -m digest directlyA Claude session (Haiku 4.5) reads a short instruction and runs the same command
Schedulecron, 07:00 ISTcron, 07:15 IST
SecretsRepository secretsCloud environment variables
Where state is savedCommits to mainCommits to its own claude/digest-routine branch
Cost to runFree within Actions minutesUses some of your Claude plan
DebuggingJob logSession transcript
Runtime (script only)~26 s~28–41 s, plus about a minute of session start-up

What Worked Well

Both delivered the same digest. Each run’s data cost was identical ($0.038) because both call the same models. GitHub Actions needed nothing beyond a workflow file and three secrets. The routine was easy to create and, once configured, followed its instructions exactly: it didn’t try to fix anything or peek at secrets, and it reported failures clearly.

What You Give Up with the Routine

With GitHub Actions, the schedule starts the script directly. With a routine, the schedule starts a Claude session, and Claude reads my instructions and then runs the script. That extra step is useful when a job needs judgment, but here the job is a fixed command, so the agent adds a moving part (and some Claude usage) without adding anything the digest needs. The routine’s cloud environment also has constraints that GitHub Actions doesn’t:

  • GitHub access is scoped to the configured repo. One of my feeds was a GitHub releases feed; from the routine it returned 403 with “sessions are bound to their configured repositories.”
  • Some sites block its outbound IP. One feed’s host served a CAPTCHA page to the routine instead of the feed.
  • State needs a home. Each run starts on a fresh machine, so anything not pushed back to git is gone. By default routines can only push to claude/ branches, which turned out to be a nice constraint: the routine keeps its state on its own branch and never touches main.
  • A missing secret can’t notify you. My first routine run failed because the keys were in a different environment, and with no Telegram token it couldn’t report that to Telegram. It was visible only in the session transcript.

My Recommendation

For a fixed script on a schedule, GitHub Actions is the simpler and more predictable choice. Reach for a routine when the job actually benefits from an agent: when it needs to read something and decide what to do, not just run a command. I am running both for a week to measure reliability properly, but I expect Actions to be the one I keep.

Lessons Learned Along the Way

A few general notes that weren’t about models at all:

  • Feedly exports aren’t all public feeds. Nine of the 31 feeds in my export failed on the first fetch outside Feedly. Three were feedproxy.feedly.com URLs that only work inside Feedly, five had moved, and one was just slow. Check every feed before you trust an OPML file.
  • Cap and check LLM output length. Every summary request sets a hard output limit (max_tokens) so a runaway response can’t blow up cost or the Telegram message. My first summaries asked for “4–5 lines” but came back around 8 lines and hit the limit (400 tokens at the time), so they were cut off mid-word. Two changes fixed it: a tighter prompt (“4–5 short sentences, 90 words at most”, with the limit lowered to 250 tokens to match), and trimming any reply that still hits the limit back to its last complete sentence. The trimming needed its own fix: a summary cut inside “2.66x” ended in “2.” and looked complete.
  • Decision models are not perfectly deterministic at the boundary. The same post, with identical text, scored 0.6975 in one run and 0.705 in another, either side of the 0.7 threshold. Small, but visible if you compare runs.
  • Strip what an unattended agent doesn’t need. When I created the routine through the API, it attached every connector on my account by default, including Gmail and Drive. A job that reads arbitrary web pages has no business holding my email. I removed them all.
  • Log everything you’ll want to compare. A JSONL line per post (scores, bucket, content source, latency, cost) turned every question in this post into a one-line query.

Summary

The lesson is simple: use the right model for the right job.

Most AI workflows have a high-volume judging step and a low-volume generating step. An LLM can do both, but you pay generation-model prices to make decisions, and with a general-purpose LLM you also get free-form output to parse. A System One model like Jev is built for the judging: typed answers, probabilities you can threshold in code, about 70× cheaper and 8× faster than Sonnet in my test, and in agreement with Sonnet on what to deliver for 55 of 59 posts. The LLM then does what it is uniquely good at, on the handful of posts that earned it.

For my digest that meant about four cents a day instead of fifty. At a larger scale, it is the difference between a feature you can afford to run on everything and one you ration.

Acknowledgements

This post was written with the help of Claude Opus 5.5 in Claude Code, which also helped me build the digest and run the cost and quality comparisons described here.

References

Code

Jev and TypeSafe

Scheduling