LLM router: what it does and when you need one

What a router does for your requests

An LLM router is a service or library that receives a model request and decides which model or provider should handle it. Your application sends one request in a common format, usually the OpenAI chat format, and the router sends it on to the destination it considers best at that moment.

The decision can be fixed ("code questions go to model A, everything else to model B"), dynamic ("whichever is cheapest right now") or reactive ("try A, and if it errors, try B"). Some routers are open-source libraries you run yourself. Others are hosted marketplaces that give you one key for many models. The goal is the same in both: stop hard-coding a single model into every call.

The three routing strategies

Rule-based routing

You write the logic. A request tagged summarize goes to a cheap, fast model. A request tagged analysis goes to a stronger one. It is predictable and easy to test, but you maintain the rules.

Cost-based routing

The router estimates the job's difficulty or length and chooses the lowest-price model that is likely to handle it. A classifier or a heuristic decides. The saving can be large when most requests are easy, and the risk is that a hard request lands on a weak model and returns a poor answer without any error to warn you.

Fallback routing

The router tries a primary model and switches on failure: timeouts, 5xx errors, rate limits. This is about uptime rather than price. It only helps if the secondary model is close enough in behavior that your prompts still work.

What routing costs you

Routing is not free, even when the software is.

• Inconsistent behavior. Two models answer the same prompt differently. Output style, refusal habits, formatting and tool-call reliability can shift between requests, which makes bugs hard to reproduce.
• Prompt tuning multiplies. A prompt tuned for one model may behave differently on another. Each route needs its own tests.
• More moving parts. The router is a service that can fail, be misconfigured or add latency.
• Data handling. Each extra provider is another party that sees your prompts. Read the policy of every destination.
• Harder cost forecasting. If the model choice changes per request, so does your bill.

A router earns its keep only if the savings or resilience outweigh these. For a deeper look at claims people make about one popular hosted router, see our page on router myths and facts.

A useful rule of thumb: add a router only after you can name the specific request types it will separate and the measured difference between the models for each. Without that evidence, you are building machinery around a guess.

How to tell whether routing saved money

Teams adopt routers for savings, then never check. A fair test takes a week of work and answers the question with data.

1. Log tokens, model used and a quality signal for every request (a thumbs-up rate, a human spot check, or an automated grader).
2. Compute the cost per request by model, using each provider's current published price.
3. Compare against a baseline: the same traffic sent to your single best model.
4. Include the cost of retries and re-prompts caused by weak answers.

A simple example with made-up numbers: if 70% of requests are easy and the cheaper model costs a third as much per request, the traffic cost falls to 0.7 x (1/3) + 0.3 x 1 = 0.533 of baseline, a 47% saving. If the weaker model forces a re-prompt on 20% of the easy requests, those retries cost 0.7 x 0.2 x (1/3) = 0.047 more, so the saving shrinks to about 42%. Real numbers vary, but the method works for any price list.

Per-request costs on this endpoint are easy to compute because pricing is flat: $0.15 per 1M input tokens and $0.50 per 1M output tokens. Read the pricing page for top-up details.

When you do need a router

Look for at least one of these conditions:

1. You use models with clearly different strengths, such as one for code and one for casual chat, and your traffic mixes both.
2. Your volume is high enough that sending easy requests to a cheaper model changes the budget.
3. You have an uptime requirement that a single provider cannot meet alone.
4. You are still evaluating models and want to compare them on live traffic without changing application code.
5. Different customers or regions must be served by different providers.

If none of these fits, you are paying the cost of routing without collecting the benefit.

When a single-model endpoint is enough

Plenty of products have one job and one model that does it well. A fiction or roleplay app, a companion chat, a drafting tool or an internal assistant often wants consistent voice above everything else. A fixed model gives that consistency. A single hosted endpoint also removes the router from the failure path.

This service is that kind of endpoint. It serves one open-weight model, run by an independent provider and tuned to answer without content refusals for lawful adult use. It is not the official API of any model vendor and is not a router across vendors. The model ID is uncensored and the interface follows the OpenAI format. Sexual content involving minors is never allowed.

Copyfrom openai import OpenAI

client = OpenAI(
    base_url="https://api.llmapirouter.com/v1",
    api_key="YOUR_KEY",
)

r = client.chat.completions.create(
    model="uncensored",
    messages=[{"role": "user", "content": "Draft a two-line opening for a mystery story."}],
)
print(r.choices[0].message.content)

A fallback router in twenty lines

To see how little magic a basic router contains, here is a fallback loop. It tries a list of endpoints in order and moves on when one fails or is overloaded. Only one entry here, because this service is a single-model endpoint, but the shape is the same when you add others.

Copyimport os, time
from openai import OpenAI, APIError, RateLimitError

ROUTES = [
    {"base_url": "https://api.llmapirouter.com/v1",
     "api_key": os.environ["LLM_API_KEY"],
     "model": "uncensored"},
    # add other OpenAI-compatible endpoints here
]

def ask(messages):
    last = None
    for route in ROUTES:
        client = OpenAI(base_url=route["base_url"], api_key=route["api_key"])
        try:
            return client.chat.completions.create(model=route["model"], messages=messages)
        except RateLimitError as e:
            last = e
            time.sleep(int(e.response.headers.get("Retry-After", 1)))
        except APIError as e:
            last = e
    raise last

Everything a commercial router adds (health checks, latency tracking, weighted choice, cost tables) is a refinement of this loop. If your needs fit this skeleton, you may not need extra software.

Using it alongside a router

The two ideas are not exclusive. If you run your own router, such as an open-source proxy library that accepts custom OpenAI-compatible upstreams, you can add this endpoint as one destination. A common pattern is a routing rule that sends requests needing no content refusals for lawful adult use to this endpoint and everything else to your default model. Give the router the base URL https://api.llmapirouter.com/v1, your key and the model ID uncensored.

Plan around the limits. A full key allows 8 concurrent requests and 300 requests per minute, and a streamed request occupies its slot until the stream ends. On HTTP 429, the error code rate_limited means requests per minute and concurrency means too many at once. A 429 with Retry-After: 5 signals temporary overload, which is exactly the kind of error a fallback rule should treat as retryable.

Pricing is $0.15 per 1M input tokens and $0.50 per 1M output tokens, from prepaid credit with no subscription. Failed or refused requests are not charged. See the pricing page for top-ups, the docs for parameters, and the API key page to claim the $0.50 trial credit and test your routing rule before you commit any money.

More guides

• Free LLM API router vs paid prepaid API: real costs
• Open router myths vs facts