Skip to content
Tool

gateway

Open-source gateway routing apps to 250+ LLMs with retries, fallbacks and caching

Pricing
Open source
Platforms
API, CLI, self-hosted
GitHub stars
13.4K
Last commit
6 months ago
Language
TypeScript
License
MIT

Similar tools

About gateway

gateway is an open-source AI tool for developer tools. It is available on API, CLI, self-hosted and Web.

The README describes this as a lightweight router that sits between your application and many model providers, so one OpenAI-style API can reach hundreds of language, vision, audio and image models. Requests can be retried, failed over to another provider, load balanced across keys, or served from a cache when the same question comes in again.

It runs locally with a single npx command, exposes a local endpoint and a console for viewing request logs, and can be deployed through Docker, Node.js, Cloudflare or Replit. Routing and guardrail behaviour is controlled by config objects passed with each client, so teams can change reliability rules without touching application code.

The project suits developers who call several model providers and want failover, cost tracking and observability without writing that plumbing themselves. Some features, such as semantic caching and prompt management, are noted in the README as available only in the hosted or enterprise versions.

What it does

Provider fallbacks
Switches to another provider or model when a request fails with a chosen error type.
Automatic retries
Retries failed requests up to five times using exponential backoff.
Load balancing
Distributes requests across several API keys or providers using weights.
Simple caching
Caches repeat LLM responses to lower cost and latency.
Configurable guardrails
Checks inputs and outputs against rules, with pre-built and custom checks.
Request logging console
Shows local request logs in a built-in console at the gateway address.

Good for

  • Keep an AI feature running when one model provider has an outage
  • Compare or switch between many LLMs through one OpenAI-compatible API
  • Cap retries and timeouts on slow or unreliable model endpoints
  • Track request volume, latency and cost across several AI providers
  • Add content guardrails to model replies before they reach users

Pricing

Open source, from $49 a month

Open-source core under MIT; hosted plans include a free Developer tier and paid Production plan from $49/month.

Prices change. Check portkey.ai before you buy.

gateway FAQ

What license is the gateway released under?

The repository is licensed under MIT.

How do I start it locally?

The README gives the command npx @portkey-ai/gateway, which runs the gateway on localhost port 8787 with a console at /public/.

Which features are only in the hosted or enterprise versions?

The README marks semantic caching and prompt template management as available in the hosted and enterprise versions.

Which agent frameworks does it work with?

The README lists Autogen, CrewAI, LangChain, LlamaIndex, Phidata and Control Flow, plus custom agents.

Is gateway free?

Yes. gateway is open source, so you can use and self-host it for free.

What are the best alternatives to gateway?

The closest alternatives to gateway are ShanHaiGateway, OmniRoute, bifrost, langfuse and Eden AI.

Where this comes from

  • Description: written from the project's README and the project's website.