Merit Order
GitHub
v0.0.1Pre-production, open source under AGPL-3.0

The cheapest model that's good enough, for every request.

Merit Order is a self-hosted LLM gateway. Your tools keep their own API. For each request it picks the cheapest model and backend that meets the work's quality bar, and records what it cost.

Claude Code · Codex CLI · pi · OpenAI and Anthropic SDKs

SIM 2,000 tasks routing simulator, every model call mocked
level
from the task's wording; tools only say it's coding
chance
each model's, from published benchmarks
pick
lowest expected cost to finish the task
  • auto, expected total cost 97.7% finished$0.40
  • auto, threshold 97.5% finished$1.01
  • Claude Sonnet 5.5, always 97.4% finished$0.73
  • Claude Opus 5.5, always 97.7% finished$1.43
Model spend per finished task. Simulated, not measured on live traffic.default 45% less than Sonnet

The idea

Line up the options by cost. Skip what isn't good enough. Take the first.

Power grids dispatch generators in merit order: cheapest first, expensive peakers only when needed. Merit Order does the same with models. Your own GPUs, when steady load pays for them, are the cheap baseload. Provider APIs are always there for everything else.

The merit order for coding level 4 · input price per million tokens, each model's cheapest route

  1. DeepSeek V4.1 Flash coding 2.4, below level 4$0.02
  2. Claude Haiku 5.5 coding 3.0, below level 4$0.10
  3. GPT-6 Luna coding 2.3, below level 4$0.10
  4. Kimi K3.1 Pro coding 2.5, below level 4$0.24
  5. GPT-6.1 Terra coding 4.2$1.00
  6. Gemini 4 Carbon coding 4.4$2.00
  7. Claude Sonnet 5.5 coding 3.9, below level 4$2.00
  8. Claude Opus 5.5 coding 4.1$4.00
  9. Claude Fable 5.2 coding 5.0$8.00
List prices of 12 November 2026. Taking the first model at or above the level is the threshold rule. The default goes further: it weighs each model by its chance of finishing the task and what a failure costs, so a cheaper model below the level, or a dearer one above it, can win. On the simulator's coding-only work, the default spent 56% of its model money on Gemini 4 Carbon, 39% of its model money on GPT-6.1 Terra, 4% of its model money on Claude Haiku 5.5. A GPU you already run would sit at the front of this line, priced at 2% of its model's API price up to 60% load.

How it saves money

Six mechanisms, each with a worked example.

Routing's numbers come from the routing simulator; the others are list-price arithmetic or illustrations on the prices of 12 November 2026, and each says which. Every mechanism in detail.

Routing by expected total cost

routing simulator

Every model has a chance of finishing each level of work, from published benchmarks. auto picks the model with the lowest expected cost to finish the task: the try, plus the chance it fails times a person's time and the next attempt. Priced for the rest of the session, including rebuilding the prompt cache elsewhere.

2,000 simulated tasks · 97.7% against 97.4% finished

$0.73
$0.40

Sonnet 5.5 for everything → auto, per finished task

Prompt caching

list-price arithmetic

The gateway adds Claude's cache markers, sets OpenAI's prompt_cache_key, and never changes the early part of the prompt. On Claude it asks for a 1-hour cache from a session's second turn, for $0.0030 more per turn.

100,000-token conversation · person steps away 20 minutes

$0.25
$0.010

write again → read from cache

Sticky sessions

list-price arithmetic

A session's cache lives on one provider. It moves only if that saves at least 10% plus $0.01 over the rest of the session, and within a task it keeps its model outright.

80,000-token Claude Opus 5.5 session, past its first turn

$0.608
$0.044

rebuild on a cold backend → a turn where it is

Tool-output compression

illustration · opt-in

Deterministic rules strip terminal codes, collapse repeated lines, and keep the head and tail of anything over 24,000 characters. Same bytes every turn, so the cache stays valid. About $0.10 less on Sonnet 5.5 the turn it arrives.

A 100,000-character test log

100k ch
24k ch

roughly 25,000 fewer tokens

Batch prices

illustration

OpenAI's Batch API and Anthropic's Message Batches are routed like live requests, among providers with a batch API and running GPUs with room. Ten providers' batch APIs are supported.

10,000 summaries · 5,000 tokens in, 500 out · Sonnet 5.5

$150
$75

live → batch price

Your GPUs at break-even

shipped configuration

A replica starts when the work it could serve would cost more than its hourly price plus 20% at API prices for 10 minutes, and drains when demand falls below break-even. While it runs, it is priced at 2% of the API price up to 60% load.

Qwen3 Coder Next on four H100s, $16 an hour

$16/h
$19.20/h

GPU price → API-priced demand that starts it

Recorded per request, by cause:routingprompt cachingself-hostingbatch discountcompression (rules)compression (model)response cache

The seven causes are computed in a fixed order, each against the result of the previous one, so they add up to the total. Prompt caching the harness set up itself isn't counted as a Merit Order saving.

How it routes

A level for the work. A score for each model.

Send model: "auto" and Merit Order chooses. Name a model and you always get it; Merit Order only picks the cheapest place to run it. Follow one request through.

12345tools → coding, at 3hard wording +1"write tests" → 2

Levels from the task's wording

Tools say the work is coding; the task's wording sets the level, with no model call. Claude Code and Codex send dozens of tools on every request, so a tool count can't tell a typo from a redesign. Length never raises a level. The word lists are settings, and every request records why it got its level.

per finished task, counting failed attemptsthreshold, the old default$3.63expected total cost$3.08

Expected total cost, the default

A failed try is still paid for, and a person has to notice it and ask again. So auto compares try + (1 − chance) × (person's time + next attempt) for every model that may serve the task. Cost per success and the old threshold remain as settings.

outcomesshadowrouterecords only≥ 80% suretests, buildslearned router · opt-in

Learns from your own traffic

Outcome signals, such as a resent request, a raised floor, failing tests or a fix loop, train a router that corrects the rules' level. It proves itself in shadow mode first. Opt-in exploration tries about 4% of new tasks one level up or down.

Model scores come from Epoch AI's benchmark data by default, refreshed daily, against a fixed bar. Run meritorder eval to score models on your own tasks instead.

What you'd pay vs. what you pay

Ten developers on Claude Code and Codex, simulated.

A month of model spend for 10 developers, 8 tasks a day each, 21 working days: 1,680 coding tasks, scaled from the routing simulator's 2,000-task coding-only run. Every model call is mocked; this is not yet measured on live traffic. The full results, or scale them to your team.

Claude Opus 5.5 for everything98.3% of tasks finished
$2,947$6,866 total
Claude Sonnet 5.5 for everything97.9% of tasks finished; also a fixed-model proxy set to Sonnet: the same tokens
$1,518$5,752 total
auto, threshold (the default until decision 62)98.5% of tasks finished
$1,694$5,562 total
auto, cost per success98.2% of tasks finished; opt-in
$759$4,955 total
auto, expected total cost (the default)98.1% of tasks finished
$773$4,830 total
auto, expected total cost, with one escalation98.3% of tasks finished; opt-in (routing.cascade)
$980$4,923 total
Oracle: the cheapest model that would have succeeded99.0% of tasks finished; a lower bound: no router knows in advance
$273$3,410 total

The mark is the simulated month; the band is its 90% interval (a paired bootstrap over the tasks). The total counts failed attempts too: it adds $15 of someone's time for each one (noticing it, asking again, waiting, cleaning up).

What the numbers say

Against Claude Opus 5.5 for everything: $773 against $2,947 of model spend, 74% less, with 98.1% of tasks finished against 98.3%.

Against Claude Sonnet 5.5 for everything (or a fixed-model proxy set to it, which sends the same tokens): $773 against $1,518, 49% less, at 98.1% against 97.9%. In total, counting failed attempts: $4,830 against $5,752.

Where it doesn't win. On coding-only work, cost per success spends less on models ($759) and comes out about even once failed attempts are counted: $3.00 against $2.93 per finished task. The opt-in escalation inside a task finished slightly more (98.3%) but cost more ($980). The oracle, which knows in advance which cheapest model would succeed, needs $273: no router gets there, and the gap is how much better routing could still do.

A hosted auto-router isn't simulated: its decision model isn't public, so we can't say what it would pick, and its platform fee would come on top.

Assumptions behind the simulated month
AssumptionValue
Developers, tasks a day, working days10, 8, 21
Clientswho sends the coding tasksClaude Code 75%, a small agent such as Codex 15%, chat with code 10%
Task difficultySWE-bench Verified's time-to-fix mixeasy 38%, medium 53%, hard 8%, very-hard 1%
Outcomesno model is calleddrawn from each model's published benchmark ability against each task's difficulty
Attempts per taskup to 3
Later turns after a 5 to 60 minute pause10%
Price per failed attemptsomeone's time; adjustable in the calculator$15
Prices and scoresthe main repo's price list of 12 November 2026

From scripts/routing_sim.py in the main repository, run on 16 November 2026. Not counted: compression, batch, the response cache and GPUs, and the cost of running the gateway. Claude Code on non-Claude models works through translation, but Anthropic doesn't support it, so expect rough edges.

Direct, one model

Nothing to run. Claude Code on Claude is what Anthropic supports. Predictable quality, if the model is strong enough for your hardest work.

Fixed-model proxy

One place for keys, budgets and logs. LiteLLM covers over 100 providers and is used in production by many organizations.

Hosted auto-router

Nothing to deploy, very large scale, hundreds of models. OpenRouter's Auto Router weighs the cost of losing a prompt cache and picks reasoning effort per request.

Merit Order

Runs in your network on your provider accounts. Budgets checked before a request is sent. Savings split by cause and checked against bills. Your own GPUs at break-even.

Works with

Point your tools at it. Nothing else changes.

Merit Order speaks OpenAI Chat Completions, OpenAI Responses (HTTP and WebSocket) and Anthropic Messages. When a tool and the chosen model speak different formats, it translates requests, replies and streams both ways.

ProvidersAnthropicAmazon BedrockMicrosoft FoundryVertex AIGemini APIAzure OpenAIOpenAIxAIOpenRouterDeepSeekMistralMoonshotZ.aiMiniMaxvLLM, SGLangyour GPUs

Control and trust

Your network, your keys, your records.

Merit Order is software you run, not a service you send prompts to.

Self-hosted

Docker Compose or a Helm chart. Your keys, your network, your Postgres and Redis. Signed multi-arch images when a version is tagged.

AGPL-3.0, every line

Read, change and run all of it. If you change it and offer it to others over a network, you share your changes under the same license.

No content stored by default

The ledger and session table hold metadata: hashes, timestamps, token counts, costs. The response cache, stored Responses and batch files are opt-in or explicit, and sealed.

Every decision recorded

Each request keeps its routing reason, for example coding: 32 tools; simple task wording (write (unit )?tests): level 2, and the chances and costs it compared.

Pre-production: version 0.0.1

Most of Merit Order is tested only against mocks of each provider's API. Checked live so far: OpenRouter, Anthropic and OpenAI, Claude Code sessions through the gateway, the Docker Compose stack, and GPUs on a local kind cluster with a stand-in vLLM.

Not yet checked live: the other providers, real GPUs, a real Codex CLI session, Entra and Google sign-in, batch APIs and bills. Set an organization budget before you send it real traffic. What's verified.

Quickstart

No keys needed to try it.

Without provider keys, requests go to built-in mock backends, and the reply says [mock-small] You said: hello.

  1. Needs Python 3.12+ and uv, or Docker.
  2. Set MERITORDER_DEV_KEY in .env, plus any provider keys you have.
  3. Run meritorder check to see which backends and models are live.

Read the full quickstart

git clone https://github.com/Merit-Order/meritorder && cd meritorder
uv sync
cp .env.example .env      # set MERITORDER_DEV_KEY
uv run meritorder check   # which backends are live
uv run meritorder serve   # http://localhost:8080

# in a second terminal
curl localhost:8080/v1/chat/completions \
  -H "authorization: Bearer $MERITORDER_DEV_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "hello"}]}'