# ShanHaiGateway

> ShanHaiGateway is an open-source AI tool for developer tools. Self-hosted OpenAI-compatible gateway that load-balances multiple model upstreams. It is available on API, CLI, self-hosted, Linux, macOS and Windows.

- **Kind:** Tool
- **Pricing:** Open source
- **Platforms:** API, CLI, self-hosted, Linux, macOS, Windows
- **Repository:** https://github.com/ffseika0304/shanhai-gateway
- **Categories:** [developer tools](https://ai-powered.com/categories/developer-tools)
- **Listing updated:** 2026-11-18

## About

The README describes ShanHaiGateway as a Go service that sits in front of several upstream model providers and exposes them as one OpenAI-compatible endpoint. Requests are distributed by weighted round-robin, failing backends are cooled down with exponential backoff, and a session affinity layer pins multi-turn jobs to the same backend so provider-side prefix caches keep hitting.

The project ships as a single binary or a Docker image with SQLite storage, a built-in stats dashboard at /api/stats, and hot-reloading of backends.json. A separate Python sidecar watches per-window quota usage and removes backends that hit their limits.

It is aimed at developers and operators who run their own gateway and want control over routing, retries and cost across several model accounts. The README is written in Chinese with an English language switch linked from the top.

## Features

- **Weighted round-robin routing:** Distributes requests across backends by configurable weight, concurrency and timeout, with a separate fallback chain.
- **Error-class based cooldown:** Applies exponential backoff for quota and permission errors and queues requests at capacity instead of returning 503.
- **Job-level session affinity:** Keeps requests from the same job or session on one backend to reuse provider-side prefix caches.
- **Hot-reloaded configuration:** Reloads backends.json when its modification time changes, so weight changes and new backends apply without a restart.
- **Quota-window sidecar:** A standard-library Python process that polls quota windows and temporarily removes exhausted backends.
- **Docker deployment:** Multi-stage build with a pure-Go SQLite driver, non-root runtime and a health endpoint.

## Good for

- Spreading traffic across several model provider accounts behind one base URL
- Keeping multi-turn agent sessions on one upstream to improve prompt cache hit rates
- Automatically sidelining a provider whose quota or model has been withdrawn
- Tracking request outcomes and failure types per backend on a local dashboard

## FAQ

### Is the gateway compatible with the OpenAI API?

The README describes it as exposing an OpenAI-compatible endpoint, with POST /v1/chat/completions as the main route.

### What does it need to run?

The README lists a Docker Compose setup, or building a binary with Go 1.27 or newer. Real backend credentials go in a config file that is kept out of git.

### Does it store request content?

Captured request and response data is written to SQLite after a scrubbing step that masks keys, internal paths, phone numbers and auth headers. The README notes that stored captures are still sensitive data.

### Is ShanHaiGateway free?

Yes. ShanHaiGateway is open source, so you can use and self-host it for free.

### What are the best alternatives to ShanHaiGateway?

The closest alternatives to ShanHaiGateway are gateway, bifrost, langfuse, Llmtee and OmniRoute.

## Where this comes from

- Description: written from the project's README.

## Alternatives

- [gateway](https://ai-powered.com/tools/gateway-10): gateway is an open-source AI tool for developer tools.
- [bifrost](https://ai-powered.com/tools/bifrost-3): bifrost is an open-source AI tool for ai models and llm tooling.
- [langfuse](https://ai-powered.com/tools/langfuse-2): langfuse is an open-source AI tool for developer tools.
- [Llmtee](https://ai-powered.com/tools/llmtee): Llmtee is an open-source AI tool for developer tools.
- [OmniRoute](https://ai-powered.com/tools/omniroute): OmniRoute is a freemium AI tool for ai models and llm tooling.
- [Eden AI](https://ai-powered.com/tools/eden_ai): Eden AI is a paid AI tool for developer tools.
- [Werkbank](https://ai-powered.com/tools/werkbank): Werkbank is an open-source AI tool for developer tools.
- [archestra](https://ai-powered.com/tools/archestra): archestra is an open-source AI tool for security.

---
Source: https://ai-powered.com/tools/shanhaigateway
