每个大型语言模型都配备一个网关。任何客户端协议进出，任何提供商都出——配备智能路由、成本控制和完整的管理控制台。

LLM 代理是一个自托管的 LLM API 网关。将你现有的 OpenAI 或 Anthropic SDK 指向它，即可即时访问所有配置好的供应商——配备自动故障切换、成本导向路由、按密钥预算、请求日志以及管理仪表盘，无需重启即可管理所有内容。

文献资料

完整文档：

为什么选择LLM代理？

• 写一次，运行任意供应商 — 你的应用使用一种协议（OpenAI 或 Anthropic）;代理会双向转换到每个上游。切换提供者时，可以通过更改模型名称，而不是你的代码。

LLM Proxy

A self-hosted gateway for LLM APIs. Clients send OpenAI, Anthropic, or Cohere-compatible requests; the proxy forwards each one to a configured provider and translates the response back.

LLM Proxy sits between your application and the model providers you pay for. Point an OpenAI or Anthropic SDK at the proxy, or a Cohere SDK at the rerank endpoint, and the proxy sends the request to whichever upstream serves the requested model. Supported providers include OpenAI, Anthropic, Gemini, OpenRouter, Ollama, vLLM, SGLang, and any OpenAI-compatible endpoint. Fallback, per-key budgets, request logs, and the admin console are built in. Configuration lives in the database and is applied without a restart.

Contents

• How it works
• Why LLM Proxy
• What it supports
• Quick start
• Use it from code
• Configuration
• Operations
• Documentation
• More screenshots
• Development
• Contributing
• Security
• License

How it works

Client                     LLM Proxy                                  Upstream
OpenAI / Anthropic /  ──▶  parse to InternalRequest                   provider API
Cohere rerank wire         route: mapping, allowlists, budgets,   ──▶  (built-in
                           retries, fallback, circuit breaker           adapters)
                      ◀──  translate back to the client's protocol

The proxy parses each client request into an internal request model. A protocol serializer handles the client side, and a provider serializer builds the upstream body. A request can arrive as OpenAI Chat Completions and leave as Anthropic Messages, or be forwarded unchanged when both sides use the same protocol.

Why LLM Proxy

• Clients keep their existing SDK and change only the base URL. Model names are defined in the proxy, so one name can point to any upstream model.
• Failed requests retry on the same provider and then fall back to the next mapping. A circuit breaker skips providers that keep failing.