Skip to content
Tool

ShanHaiGateway

Self-hosted OpenAI-compatible gateway that load-balances multiple model upstreams

Pricing
Open source
Platforms
API, CLI, self-hosted
Listing updated
today

Similar tools

About ShanHaiGateway

ShanHaiGateway is an open-source AI tool for developer tools. It is available on API, CLI, self-hosted, Linux, macOS and Windows.

The README describes ShanHaiGateway as a Go service that sits in front of several upstream model providers and exposes them as one OpenAI-compatible endpoint. Requests are distributed by weighted round-robin, failing backends are cooled down with exponential backoff, and a session affinity layer pins multi-turn jobs to the same backend so provider-side prefix caches keep hitting.

The project ships as a single binary or a Docker image with SQLite storage, a built-in stats dashboard at /api/stats, and hot-reloading of backends.json. A separate Python sidecar watches per-window quota usage and removes backends that hit their limits.

It is aimed at developers and operators who run their own gateway and want control over routing, retries and cost across several model accounts. The README is written in Chinese with an English language switch linked from the top.

What it does

Weighted round-robin routing
Distributes requests across backends by configurable weight, concurrency and timeout, with a separate fallback chain.
Error-class based cooldown
Applies exponential backoff for quota and permission errors and queues requests at capacity instead of returning 503.
Job-level session affinity
Keeps requests from the same job or session on one backend to reuse provider-side prefix caches.
Hot-reloaded configuration
Reloads backends.json when its modification time changes, so weight changes and new backends apply without a restart.
Quota-window sidecar
A standard-library Python process that polls quota windows and temporarily removes exhausted backends.
Docker deployment
Multi-stage build with a pure-Go SQLite driver, non-root runtime and a health endpoint.

Good for

  • Spreading traffic across several model provider accounts behind one base URL
  • Keeping multi-turn agent sessions on one upstream to improve prompt cache hit rates
  • Automatically sidelining a provider whose quota or model has been withdrawn
  • Tracking request outcomes and failure types per backend on a local dashboard

Pricing

Open source

MIT-licensed self-hosted tool; upstream provider costs are separate.

ShanHaiGateway FAQ

Is the gateway compatible with the OpenAI API?

The README describes it as exposing an OpenAI-compatible endpoint, with POST /v1/chat/completions as the main route.

What does it need to run?

The README lists a Docker Compose setup, or building a binary with Go 1.27 or newer. Real backend credentials go in a config file that is kept out of git.

Does it store request content?

Captured request and response data is written to SQLite after a scrubbing step that masks keys, internal paths, phone numbers and auth headers. The README notes that stored captures are still sensitive data.

Is ShanHaiGateway free?

Yes. ShanHaiGateway is open source, so you can use and self-host it for free.

What are the best alternatives to ShanHaiGateway?

The closest alternatives to ShanHaiGateway are gateway, bifrost, langfuse, Llmtee and OmniRoute.

Where this comes from

  • Description: written from the project's README.