Skip to content
MCP server

Inference AIops

Governed AI-ops for vLLM, Ray Serve, SGLang and TGI: latency RCA, scaling, drain

Runs locallyStatus and history
Install
Pricing
Open source
Platforms
CLI
Runs
Locally
Version
0.13.0
Listing updated
today

Similar MCP servers

About Inference AIops

Inference AIops is an open-source MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.

Latency diagnosis is the centerpiece. diagnose_latency_spike correlates queue depth, KV-cache pressure and preemptions with prefix-cache locality into a ranked cause and the knob to turn, such as adding replicas, raising max-num-seqs, fixing routing or enlarging the KV cache. diagnose_low_utilization does the reverse for idle GPUs and over-provisioned replicas. It reads each engine's Prometheus /metrics directly, so no Prometheus server is needed.

vLLM with Ray Serve gets the full control plane: scaling, drain, autoscale, LoRA load and unload, model deploy and redeploy, sleep and wake. SGLang and TGI get engine-agnostic observability only, and scaling writes there return an explanatory error. Sleep Mode needs a vLLM server started with VLLM_SERVER_DEV_MODE=1.

The 39 tools split 23 reads and 16 writes. Eight high-risk writes, including scale-down, scale to zero, drain and undeploy, support dry_run and double confirmation, and reversible ones record an undo. It records every call but leaves permission to the environment. A bearer token is optional.

Install

Run in a terminal

claude mcp add inference-aiops -- uvx inference-aiops

Add to claude_desktop_config.json

{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "inference-aiops": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "inference-aiops": {
      "command": "uvx",
      "args": [
        "inference-aiops"
      ]
    }
  }
}

Using several servers? Build one config file with the MCP config generator.

Tools this server provides (21)

Taken from the project documentation; the running server may differ.

diagnose_latency_spike
Ranks causes of rising latency with the knob to turn
diagnose_low_utilization
Finds idle GPUs and over-provisioned replicas
request_metrics
vLLM request metrics
queue_depth
vLLM queue depth
kv_cache_stats
vLLM KV cache stats
engine_health
Engine health across vLLM, SGLang and TGI
engine_inventory
Running model identity
diagnose_engine_latency
Engine-agnostic latency RCA
serve_deployment_list
Lists Ray Serve deployments
scale_replicas_up
Scales replicas up
scale_replicas_down
Scales replicas down
scale_to_zero
Scales to zero
drain_replica
Drains a replica
lora_load
Loads a LoRA adapter
lora_unload
Unloads a LoRA adapter
model_sleep
Puts a model to sleep
model_wake
Wakes a model
model_deploy
Deploys a model
model_undeploy
Undeploys a model
deployment_redeploy
Redeploys a deployment
cost_per_token
Estimates cost per token

What it does

Latency spike RCA
Ranks causes from queue depth, KV cache and prefix-cache locality.
Low-utilization analysis
Finds idle GPUs and replicas that could scale down.
Prometheus-native
Parses `/metrics` without a Prometheus deployment.
Ray Serve control
Scale, drain, autoscale, redeploy and routing writes for vLLM.
Multi-engine reads
Health, queue depth and latency for vLLM, SGLang and TGI.
Guarded high-risk writes
Dry run and double confirm on scale-down, drain and undeploy.

Good for

  • Explain why time to first token jumped
  • Decide whether to add replicas or enlarge the KV cache
  • Find over-provisioned GPU replicas
  • Drain a replica before maintenance
  • Load or unload a LoRA adapter

Pricing

Open source

MIT licensed

Inference AIops FAQ

Which engines does it support?

vLLM with Ray Serve fully; SGLang and TGI for observability only.

Do I need Prometheus?

No. It reads each engine's `/metrics` endpoint directly.

Is a token required?

A bearer token is optional.

What do Sleep Mode tools need?

vLLM started with `VLLM_SERVER_DEV_MODE=1`.

How many tools?

39: 23 read and 16 write.

Is Inference AIops free?

Yes. Inference AIops is open source, so you can use and self-host it for free.

How do I add Inference AIops to Claude Code?

Run this command in your terminal: claude mcp add inference-aiops -- uvx inference-aiops

What are the best alternatives to Inference AIops?

The closest alternatives to Inference AIops are headroom, omniget, next-ai-draw-io, mcp-for-blender and Claude Code Ultimate Guide.

Where this comes from

  • Description: written from the project's README.
  • Install steps: from the official MCP Registry entry.

How AIPoweredBot checks servers