# Inference AIops

> Inference AIops is an open-source MCP server for ai models and llm tooling. Governed AI-ops for vLLM, Ray Serve, SGLang and TGI: latency RCA, scaling, drain. It runs locally on your machine and is started by your AI client.

- **Kind:** MCP server
- **Pricing:** Open source
- **Platforms:** CLI
- **Repository:** https://github.com/aiops-tools/inference-aiops
- **MCP Registry name:** io.github.AIops-tools/inference-aiops
- **Categories:** [AI models and LLM tooling](https://ai-powered.com/categories/ai-and-llm), [cloud and infrastructure](https://ai-powered.com/categories/cloud-and-infrastructure)
- **Listing updated:** 2026-11-18

## About

Latency diagnosis is the centerpiece. `diagnose_latency_spike` correlates queue depth, KV-cache pressure and preemptions with prefix-cache locality into a ranked cause and the knob to turn, such as adding replicas, raising `max-num-seqs`, fixing routing or enlarging the KV cache. `diagnose_low_utilization` does the reverse for idle GPUs and over-provisioned replicas. It reads each engine's Prometheus `/metrics` directly, so no Prometheus server is needed.

vLLM with Ray Serve gets the full control plane: scaling, drain, autoscale, LoRA load and unload, model deploy and redeploy, sleep and wake. SGLang and TGI get engine-agnostic observability only, and scaling writes there return an explanatory error. Sleep Mode needs a vLLM server started with `VLLM_SERVER_DEV_MODE=1`.

The 39 tools split 23 reads and 16 writes. Eight high-risk writes, including scale-down, scale to zero, drain and undeploy, support `dry_run` and double confirmation, and reversible ones record an undo. It records every call but leaves permission to the environment. A bearer token is optional.

## Features

- **Latency spike RCA:** Ranks causes from queue depth, KV cache and prefix-cache locality.
- **Low-utilization analysis:** Finds idle GPUs and replicas that could scale down.
- **Prometheus-native:** Parses `/metrics` without a Prometheus deployment.
- **Ray Serve control:** Scale, drain, autoscale, redeploy and routing writes for vLLM.
- **Multi-engine reads:** Health, queue depth and latency for vLLM, SGLang and TGI.
- **Guarded high-risk writes:** Dry run and double confirm on scale-down, drain and undeploy.

## Good for

- Explain why time to first token jumped
- Decide whether to add replicas or enlarge the KV cache
- Find over-provisioned GPU replicas
- Drain a replica before maintenance
- Load or unload a LoRA adapter

## Install

### Claude Code (Terminal)

```sh
claude mcp add inference-aiops -- uvx inference-aiops
```

### Claude Desktop (claude_desktop_config.json)

```json
{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}
```

### Cursor (~/.cursor/mcp.json)

```json
{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}
```

### VS Code (.vscode/mcp.json)

```json
{
  "servers": {
    "inference-aiops": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}
```

### Windsurf (~/.codeium/windsurf/mcp_config.json)

```json
{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}
```

### Other clients (your client's MCP settings (most use this shape))

```json
{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}
```

## Tools

- `diagnose_latency_spike`: Ranks causes of rising latency with the knob to turn
- `diagnose_low_utilization`: Finds idle GPUs and over-provisioned replicas
- `request_metrics`: vLLM request metrics
- `queue_depth`: vLLM queue depth
- `kv_cache_stats`: vLLM KV cache stats
- `engine_health`: Engine health across vLLM, SGLang and TGI
- `engine_inventory`: Running model identity
- `diagnose_engine_latency`: Engine-agnostic latency RCA
- `serve_deployment_list`: Lists Ray Serve deployments
- `scale_replicas_up`: Scales replicas up
- `scale_replicas_down`: Scales replicas down
- `scale_to_zero`: Scales to zero
- `drain_replica`: Drains a replica
- `lora_load`: Loads a LoRA adapter
- `lora_unload`: Unloads a LoRA adapter
- `model_sleep`: Puts a model to sleep
- `model_wake`: Wakes a model
- `model_deploy`: Deploys a model
- `model_undeploy`: Undeploys a model
- `deployment_redeploy`: Redeploys a deployment
- `cost_per_token`: Estimates cost per token

## FAQ

### Which engines does it support?

vLLM with Ray Serve fully; SGLang and TGI for observability only.

### Do I need Prometheus?

No. It reads each engine's `/metrics` endpoint directly.

### Is a token required?

A bearer token is optional.

### What do Sleep Mode tools need?

vLLM started with `VLLM_SERVER_DEV_MODE=1`.

### How many tools?

39: 23 read and 16 write.

### Is Inference AIops free?

Yes. Inference AIops is open source, so you can use and self-host it for free.

### How do I add Inference AIops to Claude Code?

Run this command in your terminal: claude mcp add inference-aiops -- uvx inference-aiops

### What are the best alternatives to Inference AIops?

The closest alternatives to Inference AIops are headroom, omniget, next-ai-draw-io, mcp-for-blender and Claude Code Ultimate Guide.

## Where this comes from

- Description: written from the project's README.
- Install steps: from the official MCP Registry entry.

## Alternatives

- [headroom](https://ai-powered.com/mcp/headroom-3): headroom is an open-source MCP server for ai models and llm tooling.
- [omniget](https://ai-powered.com/mcp/omniget): omniget is an open-source MCP server for ai models and llm tooling.
- [next-ai-draw-io](https://ai-powered.com/mcp/next-ai-draw-io-2): next-ai-draw-io is an open-source MCP server for ai models and llm tooling.
- [mcp-for-blender](https://ai-powered.com/mcp/mcp-for-blender): mcp-for-blender is an open-source MCP server for design and media.
- [Claude Code Ultimate Guide](https://ai-powered.com/mcp/claude-code-ultimate-guide): Claude Code Ultimate Guide is an open-source MCP server for ai models and llm tooling.
- [unity-mcp](https://ai-powered.com/mcp/unity-mcp-2): unity-mcp is an open-source MCP server for ai models and llm tooling.
- [fastapi_mcp](https://ai-powered.com/mcp/fastapi-mcp): fastapi_mcp is an open-source MCP server for ai models and llm tooling.
- [tyk](https://ai-powered.com/mcp/tyk): tyk is an open-source MCP server for developer tools.

---
Source: https://ai-powered.com/mcp/inference-aiops
