Inference AIops status
We have not checked this server yet.
- Answered (30 days)
- —
- Last seen working
- —
- Last commit
- —
- Licence
- Not stated
Live checks
This server runs on your own machine, so we don't run it. The status below comes from its repository.
Tools (21)
Taken from the project documentation; the running server may differ.
- diagnose_latency_spike
- Ranks causes of rising latency with the knob to turn
- diagnose_low_utilization
- Finds idle GPUs and over-provisioned replicas
- request_metrics
- vLLM request metrics
- queue_depth
- vLLM queue depth
- kv_cache_stats
- vLLM KV cache stats
- engine_health
- Engine health across vLLM, SGLang and TGI
- engine_inventory
- Running model identity
- diagnose_engine_latency
- Engine-agnostic latency RCA
- serve_deployment_list
- Lists Ray Serve deployments
- scale_replicas_up
- Scales replicas up
- scale_replicas_down
- Scales replicas down
- scale_to_zero
- Scales to zero
- drain_replica
- Drains a replica
- lora_load
- Loads a LoRA adapter
- lora_unload
- Unloads a LoRA adapter
- model_sleep
- Puts a model to sleep
- model_wake
- Wakes a model
- model_deploy
- Deploys a model
- model_undeploy
- Undeploys a model
- deployment_redeploy
- Redeploys a deployment
- cost_per_token
- Estimates cost per token