LLM Proxy

A self-hosted gateway for LLM APIs. Clients send OpenAI, Anthropic, or Cohere-compatible requests; the proxy forwards each one to a configured provider and translates the response back.

LLM Proxy sits between your application and the model providers you pay for. Point an OpenAI or Anthropic SDK at the proxy, or a Cohere SDK at the rerank endpoint, and the proxy sends the request to whichever upstream serves the requested model. Supported providers include OpenAI, Anthropic, Gemini, OpenRouter, Ollama, vLLM, SGLang, and any OpenAI-compatible endpoint. Fallback, per-key budgets, request logs, and the admin console are built in. Configuration lives in the database and is applied without a restart.

Contents

• How it works
• Why LLM Proxy
• What it supports
• Quick start
• Use it from code
• Configuration
• Operations
• Documentation
• More screenshots
• Development
• Contributing
• Security
• License

How it works

Client                     LLM Proxy                                  Upstream
OpenAI / Anthropic /  ──▶  parse to InternalRequest                   provider API
Cohere rerank wire         route: mapping, allowlists, budgets,   ──▶  (built-in
                           retries, fallback, circuit breaker           adapters)
                      ◀──  translate back to the client's protocol

The proxy parses each client request into an internal request model. A protocol serializer handles the client side, and a provider serializer builds the upstream body. A request can arrive as OpenAI Chat Completions and leave as Anthropic Messages, or be forwarded unchanged when both sides use the same protocol.

Why LLM Proxy

• Clients keep their existing SDK and change only the base URL. Model names are defined in the proxy, so one name can point to any upstream model.
• Failed requests retry on the same provider and then fall back to the next mapping. A circuit breaker skips providers that keep failing.