llmgw
Docs/API

Public data plane

Core OpenAI and Anthropic surfaces.

Familiar HTTP shapes with explicit model selection, policy, usage, failover, and strict loss checks.

This is core protocol compatibility, not a claim of every vendor endpoint or field. Management APIs are authenticated but not yet a versioned public OpenAPI contract.

Authentication

/health is unauthenticated. Other data-plane routes accept x-api-key or Authorization: Bearer. When both exist, x-api-key wins. Give clients gateway-issued project keys; the static administrator keys are also accepted but bypass key and project allowlists and quotas. Unauthenticated local mode never grants admin access.

Endpoint inventory

MethodPathContract
GET/healthProcess liveness and build identity; no provider readiness.
GET/v1/modelsConfigured, credential/policy-eligible models, endpoints, and safe aliases.
POST/v1/chat/completionsChat, tools, streaming, vision filtering, and eligible ordered failover.
POST/v1/responsesNative Responses or strict stateless Chat fallback.
POST/v1/messagesNative or strict adapted Anthropic Messages.
POST/v1/messages/count_tokensNative count or marked deterministic estimate.
POST/v1/embeddingsOne OpenAI-compatible embedding target.
POST/v1/audio/transcriptionsMultipart single-target STT proxy.
POST/v1/audio/speechSingle-target speech, including native Edge TTS.
POST/v1/images/generationsInline base64 image output through a capable provider.
POST/v1/videos/generationsStart or poll a long-running video operation.

Bare aliases

/models, /chat/completions, /responses, /messages, /messages/count_tokens, /embeddings, /audio/transcriptions, /audio/speech, /images/generations, and /videos/generations map to their /v1 handlers. /completions maps to Chat Completions; there is no /v1/completions.

Chat Completions

request
curl http://127.0.0.1:8787/v1/chat/completions \
  -H 'Authorization: Bearer <GATEWAY_PROJECT_KEY>' \
  -H 'Content-Type: application/json' \
  -d '{"model":"<MODEL_SELECTOR>","messages":[{"role":"user","content":"hello"}]}'

Core chat supports tools, tool calls, streaming, usage normalization, vision filtering, and cancellation. Experimental Chat-to-Responses adaptation is opt-in and catalog-driven. Chat, Responses, and Messages also accept a per-request failover budget and an affinity key; see failover.

Responses

Native targets preserve Responses envelopes and stream events. Strict Chat fallback is limited to compatible stateless requests. Provider state, encrypted reasoning state, built-in tools, and non-empty include require a native Responses target that supports the requested feature.

No background jobsbackground: true is always rejected, including for native Responses targets. Native transport is not a guarantee of model or account capability.

Anthropic Messages and counting

Native non-streaming targets receive Messages directly after model/preamble resolution. Adapted paths reject unsupported semantic fields before dispatch.

Token counting uses a native count where possible. Other targets receive an estimate marked X-LLMGW-Token-Count: estimate. Counting has a 32 MiB request limit and does not consume inference quota or write usage.

Embeddings and media

  • Embeddings use one target to avoid mixing vector spaces.
  • Transcription accepts multipart file and model. Uploads count against the request body limit, 64 MiB unless LLMGW_MAX_REQUEST_BODY_BYTES changes it.
  • Speech proxies OpenAI-shaped providers or uses native Edge TTS.
  • Images return inline b64_json; native generation is currently Google-capable-provider specific.
  • Video starts with HTTP 202 and an operation ID; poll with the same selector and operation.

Errors, streaming, cancellation

A request body over the limit is refused with 413 in the standard error envelope. Meaningful upstream status is preserved while credential-shaped diagnostics are sanitized. Endpoint streaming can move to another target only before output starts. Covered coding paths stop upstream work, retry, and failover after client cancellation.

Asynchronous jobsAfter a video operation ID exists, canceling the initiating HTTP request does not guarantee provider-side job cancellation.