Inference AIops
Governed AI-ops for vLLM, Ray Serve, SGLang and TGI: latency RCA, scaling, drain
- Pricing
- Open source
- Platforms
- CLI
- Runs
- Locally
- Version
- 0.13.0
- Listing updated
- today
Similar MCP servers
About Inference AIops
Inference AIops is an open-source MCP server for ai models and llm tooling. It runs locally on your machine and is started by your AI client.
Latency diagnosis is the centerpiece. diagnose_latency_spike correlates queue depth, KV-cache pressure and preemptions with prefix-cache locality into a ranked cause and the knob to turn, such as adding replicas, raising max-num-seqs, fixing routing or enlarging the KV cache. diagnose_low_utilization does the reverse for idle GPUs and over-provisioned replicas. It reads each engine's Prometheus /metrics directly, so no Prometheus server is needed.
vLLM with Ray Serve gets the full control plane: scaling, drain, autoscale, LoRA load and unload, model deploy and redeploy, sleep and wake. SGLang and TGI get engine-agnostic observability only, and scaling writes there return an explanatory error. Sleep Mode needs a vLLM server started with VLLM_SERVER_DEV_MODE=1.
The 39 tools split 23 reads and 16 writes. Eight high-risk writes, including scale-down, scale to zero, drain and undeploy, support dry_run and double confirmation, and reversible ones record an undo. It records every call but leaves permission to the environment. A bearer token is optional.
Install
Run in a terminal
claude mcp add inference-aiops -- uvx inference-aiopsAdd to claude_desktop_config.json
{
"mcpServers": {
"inference-aiops": {
"command": "uvx",
"args": [
"inference-aiops"
]
}
}
}Add to ~/.cursor/mcp.json
{
"mcpServers": {
"inference-aiops": {
"command": "uvx",
"args": [
"inference-aiops"
]
}
}
}Add to .vscode/mcp.json
{
"servers": {
"inference-aiops": {
"type": "stdio",
"command": "uvx",
"args": [
"inference-aiops"
]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"inference-aiops": {
"command": "uvx",
"args": [
"inference-aiops"
]
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"inference-aiops": {
"command": "uvx",
"args": [
"inference-aiops"
]
}
}
}Using several servers? Build one config file with the MCP config generator.
Tools this server provides (21)
Taken from the project documentation; the running server may differ.
- diagnose_latency_spike
- Ranks causes of rising latency with the knob to turn
- diagnose_low_utilization
- Finds idle GPUs and over-provisioned replicas
- request_metrics
- vLLM request metrics
- queue_depth
- vLLM queue depth
- kv_cache_stats
- vLLM KV cache stats
- engine_health
- Engine health across vLLM, SGLang and TGI
- engine_inventory
- Running model identity
- diagnose_engine_latency
- Engine-agnostic latency RCA
- serve_deployment_list
- Lists Ray Serve deployments
- scale_replicas_up
- Scales replicas up
- scale_replicas_down
- Scales replicas down
- scale_to_zero
- Scales to zero
- drain_replica
- Drains a replica
- lora_load
- Loads a LoRA adapter
- lora_unload
- Unloads a LoRA adapter
- model_sleep
- Puts a model to sleep
- model_wake
- Wakes a model
- model_deploy
- Deploys a model
- model_undeploy
- Undeploys a model
- deployment_redeploy
- Redeploys a deployment
- cost_per_token
- Estimates cost per token
What it does
- Latency spike RCA
- Ranks causes from queue depth, KV cache and prefix-cache locality.
- Low-utilization analysis
- Finds idle GPUs and replicas that could scale down.
- Prometheus-native
- Parses `/metrics` without a Prometheus deployment.
- Ray Serve control
- Scale, drain, autoscale, redeploy and routing writes for vLLM.
- Multi-engine reads
- Health, queue depth and latency for vLLM, SGLang and TGI.
- Guarded high-risk writes
- Dry run and double confirm on scale-down, drain and undeploy.
Good for
- Explain why time to first token jumped
- Decide whether to add replicas or enlarge the KV cache
- Find over-provisioned GPU replicas
- Drain a replica before maintenance
- Load or unload a LoRA adapter
Pricing
Open source
MIT licensed
Inference AIops FAQ
Which engines does it support?
vLLM with Ray Serve fully; SGLang and TGI for observability only.
Do I need Prometheus?
No. It reads each engine's `/metrics` endpoint directly.
Is a token required?
A bearer token is optional.
What do Sleep Mode tools need?
vLLM started with `VLLM_SERVER_DEV_MODE=1`.
How many tools?
39: 23 read and 16 write.
Is Inference AIops free?
Yes. Inference AIops is open source, so you can use and self-host it for free.
How do I add Inference AIops to Claude Code?
Run this command in your terminal: claude mcp add inference-aiops -- uvx inference-aiops
What are the best alternatives to Inference AIops?
The closest alternatives to Inference AIops are headroom, omniget, next-ai-draw-io, mcp-for-blender and Claude Code Ultimate Guide.
Where this comes from
- Description: written from the project's README.
- Install steps: from the official MCP Registry entry.



