
Diagnoses GKE TPU training throughput drops using ML Diagnostics and Cloud Monitoring
- Open source
- 24.4K GitHub stars
- Cloud and infrastructure


Diagnoses GKE TPU training throughput drops using ML Diagnostics and Cloud Monitoring

Monitors GKE TPU nodes, node pools and workloads using system metrics and PromQL

Diagnoses GKE TPU and GPU nodes stuck NotReady after kernel panics or hardware lockups

Diagnoses GKE TPU node pool stockouts caused by unhealthy pools holding reservation slots

Diagnoses GKE TPU VMs stuck in Compute Engine REPAIRING state and routes recovery

Guides agents through diagnosing GKE TPU multi-slice training hangs using MXLA analyzer

Diagnoses GKE JobSet restarts, preemptions and node failures for AI training jobs

Diagnoses GKE TPU slice creation stalls caused by unstable Compute Engine MIGs

Diagnoses vbar_control_agent OOMs and TPU v6e device failures on GKE clusters

Plans and validates GKE cluster upgrades, node pool strategies and maintenance runbooks

Guides an agent through diagnosing GKE nodes stuck in NotReady status

Diagnoses GKE TPU SliceCreationFailed errors caused by overlapping partitions or nodes
The closest alternatives to Gke AI troubleshooting handle disruption GPU tpu are Gke AI troubleshooting tpu performance degradation, Gke AI troubleshooting tpu metrics monitoring, Gke AI troubleshooting node unresponsive timeout, Gke AI troubleshooting tpu reservation stockout and Gke AI troubleshooting tpu vm repair stuck.
Yes. Gke AI troubleshooting tpu performance degradation, Gke AI troubleshooting tpu metrics monitoring, Gke AI troubleshooting node unresponsive timeout have a free or open-source option.