
gateway
Open-source gateway routing apps to 250+ LLMs with retries, fallbacks and caching
- Pricing
- Open source
- Platforms
- API, CLI, self-hosted
- GitHub stars
- 13.4K
- Last commit
- 6 months ago
- Language
- TypeScript
- License
- MIT
Similar tools
About gateway
gateway is an open-source AI tool for developer tools. It is available on API, CLI, self-hosted and Web.
The README describes this as a lightweight router that sits between your application and many model providers, so one OpenAI-style API can reach hundreds of language, vision, audio and image models. Requests can be retried, failed over to another provider, load balanced across keys, or served from a cache when the same question comes in again.
It runs locally with a single npx command, exposes a local endpoint and a console for viewing request logs, and can be deployed through Docker, Node.js, Cloudflare or Replit. Routing and guardrail behaviour is controlled by config objects passed with each client, so teams can change reliability rules without touching application code.
The project suits developers who call several model providers and want failover, cost tracking and observability without writing that plumbing themselves. Some features, such as semantic caching and prompt management, are noted in the README as available only in the hosted or enterprise versions.
What it does
- Provider fallbacks
- Switches to another provider or model when a request fails with a chosen error type.
- Automatic retries
- Retries failed requests up to five times using exponential backoff.
- Load balancing
- Distributes requests across several API keys or providers using weights.
- Simple caching
- Caches repeat LLM responses to lower cost and latency.
- Configurable guardrails
- Checks inputs and outputs against rules, with pre-built and custom checks.
- Request logging console
- Shows local request logs in a built-in console at the gateway address.
Good for
- Keep an AI feature running when one model provider has an outage
- Compare or switch between many LLMs through one OpenAI-compatible API
- Cap retries and timeouts on slow or unreliable model endpoints
- Track request volume, latency and cost across several AI providers
- Add content guardrails to model replies before they reach users
Pricing
Open source, from $49 a month
Open-source core under MIT; hosted plans include a free Developer tier and paid Production plan from $49/month.
Prices change. Check portkey.ai before you buy.
gateway FAQ
What license is the gateway released under?
The repository is licensed under MIT.
How do I start it locally?
The README gives the command npx @portkey-ai/gateway, which runs the gateway on localhost port 8787 with a console at /public/.
Which features are only in the hosted or enterprise versions?
The README marks semantic caching and prompt template management as available in the hosted and enterprise versions.
Which agent frameworks does it work with?
The README lists Autogen, CrewAI, LangChain, LlamaIndex, Phidata and Control Flow, plus custom agents.
Is gateway free?
Yes. gateway is open source, so you can use and self-host it for free.
What are the best alternatives to gateway?
The closest alternatives to gateway are ShanHaiGateway, OmniRoute, bifrost, langfuse and Eden AI.
Where this comes from
- Description: written from the project's README and the project's website.

