Skip to content

Inference AIops status

We have not checked this server yet.

Answered (30 days)
—
Last seen working
—
Last commit
—
Licence
Not stated

Live checks

This server runs on your own machine, so we don't run it. The status below comes from its repository.

Tools (21)

Taken from the project documentation; the running server may differ.

diagnose_latency_spike
Ranks causes of rising latency with the knob to turn
diagnose_low_utilization
Finds idle GPUs and over-provisioned replicas
request_metrics
vLLM request metrics
queue_depth
vLLM queue depth
kv_cache_stats
vLLM KV cache stats
engine_health
Engine health across vLLM, SGLang and TGI
engine_inventory
Running model identity
diagnose_engine_latency
Engine-agnostic latency RCA
serve_deployment_list
Lists Ray Serve deployments
scale_replicas_up
Scales replicas up
scale_replicas_down
Scales replicas down
scale_to_zero
Scales to zero
drain_replica
Drains a replica
lora_load
Loads a LoRA adapter
lora_unload
Unloads a LoRA adapter
model_sleep
Puts a model to sleep
model_wake
Wakes a model
model_deploy
Deploys a model
model_undeploy
Undeploys a model
deployment_redeploy
Redeploys a deployment
cost_per_token
Estimates cost per token