Routing without hidden policy.

llm-gateway is a single-node request router and governance boundary, not a model classifier or quota scheduler.

Model selectors

Exact target

provider/model selects one configured provider and native model. Provider retry can repeat it; cross-target failover does not occur.

Named endpoint

An endpoint is an ordered list of exact targets. Eligible coding requests can advance through the list before output starts.

Bare alias

Provider-native IDs are normalized only when one eligible target remains. Collisions are omitted rather than resolved by order.

endpoints:
  coding:
    failover:
      - { provider: hosted, model: <model-a> }
      - { provider: local, model: <model-b> }

Retries and failover

Retries happen inside one provider target. Endpoint failover moves between targets. Chat and compatible Responses/Messages paths can advance after supported throttling, server errors, or timeouts, and past a billing refusal or a refusal of the request's shape, such as a context it exceeds or tools or images its model does not take. A billing refusal also counts against the provider's circuit; neither is repeated.

Embeddings, transcription, speech, image, and video choose the first eligible member. Embeddings intentionally avoid cross-model failover because vector spaces differ.

Failover budget and affinity

Chat Completions, Responses, and Messages run each routed request under one budget, 2 minutes by default, covering target choice and the wait for an answer to begin, provider retries included. It never cuts an answer that has begun: an opened stream runs until it ends or the client leaves, and a non-streaming attempt under way is bounded by its provider's timeout instead. Override it per request in milliseconds with X-LLMGW-Fallback-Timeout-Ms or fallback_timeout_ms. X-LLMGW-Affinity-Key or affinity_key rotates an endpoint's eligible members to a start chosen by a SHA-256 hash of the key, so the same key starts at the same member.

API adaptation

force_api_support is experimental and off by default. The persisted model catalog can direct Chat to a Responses-only target. Translation is catalog-driven and rejects fields it cannot preserve; it is not a universal intermediate representation.

Governance path

1. Authenticate the static or project key.
2. Resolve principal, project, role, key status, and expiry.
3. Resolve exact target or endpoint.
4. Intersect project/key model and provider allowlists.
5. Resolve private/system/shared credential under provider-specific rules.
6. Validate the request against the resolved targets.
7. Check every quota limit and count the request in one transaction; a request refused before this step consumes no quota.
8. Dispatch, record usage and telemetry, settle token/cost/credit counters, and evaluate alerts.

When policy or credentials remove every target, resolution fails closed.

Architecture

Claude / Codex / Copilot / SDK / HTTP
                │
     OpenAI or Anthropic facade
                │
 authentication + project/key policy
                │
 exact target / endpoint / safe alias
                │
 provider retry + ordered failover
                │
 transport + credential adapter
                │
 hosted API / subscription / local model

The embedded Preact console calls authenticated admin/user APIs. SQLite owns IAM, keys, usage, quotas, audit, alerts, and outbox. Catalog, failover telemetry, optional request logs, and the legacy savings ledger remain local sidecars.

Single-node state

SQLite WAL and one deterministic control-plane connection simplify quota and lifecycle transactions. There is no clustered state, distributed circuit breaker, or HA scheduler. Use backup and restore for continuity.