# Gke AI troubleshooting tpu performance degradation

> Gke AI troubleshooting tpu performance degradation is an open-source agent skill for cloud and infrastructure. Diagnoses GKE TPU training throughput drops using ML Diagnostics and Cloud Monitoring. It is available on Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose.

- **Kind:** Skill
- **Pricing:** Open source
- **Platforms:** Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- **Website:** https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation
- **Repository:** https://github.com/google/skills
- **GitHub stars:** 24,379
- **License:** Apache-2.0
- **Skill name:** gke-ai-troubleshooting-tpu-performance-degradation
- **Collection:** https://github.com/google/skills
- **Categories:** cloud and infrastructure
- **Listing updated:** 2026-11-18

## About

This skill guides an agent through investigating a drop in Cloud TPU duty cycle or slower step times on Google Kubernetes Engine, when training pods keep running rather than crashing. It starts by listing ML runs and PERFORMANCE_DEGRADATION monitored events in a time window around the incident, then reads the analyzer reports to see which analyzer fired.

The agent then corroborates the findings with read-only PromQL queries over one-minute Cloud Monitoring node metrics for TPU duty cycle, HBM utilization, and host memory and CPU allocation. Where an infrastructure analyzer names culprit Compute Engine instance IDs, a kubectl query maps them to GKE nodes and topology labels. Separate reference paths cover workload bottlenecks, infrastructure or network fabric throttling, events with no detected analyzer, and healthy telemetry.

The workflow is read-only: the skill tells the agent not to drain, delete or recreate nodes, and leaves any fixes for the user to apply. It expects the gcloud alpha mldiagnostics component, kubectl, and access to a Google Cloud project with the relevant APIs enabled. ML Diagnostics only supports JAX workloads on TPUs.

## When your agent uses it

> Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).

## What it can access

From a static check of the files on GitHub; nothing was run.

- [Instructions only, no scripts](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation)
- [License: Apache-2.0](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation)

## Features

- **Monitored event triage:** Lists ML runs and PERFORMANCE_DEGRADATION events in a time window and reads their analyzer reports.
- **System metrics correlation:** Provides PromQL queries for TPU duty cycle, HBM use, and host memory and CPU utilization.
- **Culprit node mapping:** Maps numeric Compute Engine instance IDs from analyzer output to GKE nodes and topology labels.
- **Category-based remediation:** Routes the investigation to workload, infrastructure, no-detection or healthy-telemetry reference paths.
- **Read-only safety guardrails:** Prevents node cordoning for workload saturation and forbids changes to managed instance groups.

## Good for

- Find out why TPU training throughput dropped without any pod crashing
- Check whether a PERFORMANCE_DEGRADATION monitored event fired during a training run
- Identify which GKE TPU nodes correspond to culprit instance IDs in an analyzer report
- Confirm whether HBM or host CPU saturation is slowing a multi-slice training job

## Install

### Claude Code

Copy the skill folder to `~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation` to use it in all your projects:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.claude/skills &&
rm -rf ~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"
```

For one project only, put the folder in `.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation` inside it instead.

### Codex

Copy the skill folder to `~/.agents/skills/gke-ai-troubleshooting-tpu-performance-degradation` to use it in every repository:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.agents/skills &&
rm -rf ~/.agents/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.agents/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"
```

For one repository only, put the folder in `.agents/skills/gke-ai-troubleshooting-tpu-performance-degradation` inside it instead.

### Cursor

Copy the skill folder to `~/.cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation` to use it in every project:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.cursor/skills &&
rm -rf ~/.cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"
```

For one project only, put the folder in `.cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation` inside it instead.

### Gemini CLI

Run in a terminal (add `--scope workspace` to install it for the current workspace only):

```sh
gemini skills install https://github.com/google/skills.git --path skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation
```

Or copy the folder to `~/.gemini/skills/gke-ai-troubleshooting-tpu-performance-degradation` yourself.

### VS Code

For GitHub Copilot in VS Code, copy the skill folder to `~/.copilot/skills/gke-ai-troubleshooting-tpu-performance-degradation`:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.copilot/skills &&
rm -rf ~/.copilot/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.copilot/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"
```

For one project only, put the folder in `.github/skills/gke-ai-troubleshooting-tpu-performance-degradation` inside it instead.

### Other agents

Get the folder:

```sh
git clone --depth 1 https://github.com/google/skills.git
```

Then copy `skills/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation` into your agent's skills folder as `gke-ai-troubleshooting-tpu-performance-degradation`. Agents that support SKILL.md read one folder per skill.

## FAQ

### What does this skill need installed?

The Google Cloud SDK with the alpha component for gcloud alpha mldiagnostics, plus kubectl. The project needs billing, and the container, monitoring and hypercomputecluster APIs enabled.

### Which workloads does it support?

Google Cloud ML Diagnostics only supports JAX on TPUs. Workload Monitoring supports the jobset and job GKE job types on GKE versions 1.36.0-gke.4681000 and later.

### Does the skill change my cluster?

No. It is designed to run read-only diagnostic commands and gives the user any fix to apply themselves.

### When should I use a different skill instead?

For complete XLA execution stalls with HANG_DETECTED logs, the skill points to the mxla-hang skill. For pod eviction or interruption restarts, it points to the jobset-interruption skill.

### How do I install Gke AI troubleshooting tpu performance degradation in Claude Code?

Copy the skill folder from GitHub to ~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation to use it in all your projects, or to .claude/skills/gke-ai-troubleshooting-tpu-performance-degradation in one project.

### Does Gke AI troubleshooting tpu performance degradation run scripts?

No. It is instructions only, with no scripts. This comes from a static check of the files on GitHub; nothing was run. Read the files before you install it.

### Which AI agents can use Gke AI troubleshooting tpu performance degradation?

Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose. Each reads skills in the SKILL.md format from its own folder; the install steps on this page list them.

### What are the best alternatives to Gke AI troubleshooting tpu performance degradation?

The closest alternatives to Gke AI troubleshooting tpu performance degradation are Gem devops guidelines, AWS cost optimize, AWS cdk python setup, Azure smart city iot solution builder and AWS resource health diagnose.

## Where this comes from

- Description: from the skill's own SKILL.md.

## Alternatives

- [Gem devops guidelines](https://ai-powered.com/skills/gem-devops-guidelines): Gem devops guidelines is an open-source agent skill for cloud and infrastructure.
- [AWS cost optimize](https://ai-powered.com/skills/aws-cost-optimize): AWS cost optimize is an open-source agent skill for cloud and infrastructure.
- [AWS cdk python setup](https://ai-powered.com/skills/aws-cdk-python-setup): AWS cdk python setup is an open-source agent skill for cloud and infrastructure.
- [Azure smart city iot solution builder](https://ai-powered.com/skills/azure-smart-city-iot-solution-builder): Azure smart city iot solution builder is an open-source agent skill for cloud and infrastructure.
- [AWS resource health diagnose](https://ai-powered.com/skills/aws-resource-health-diagnose): AWS resource health diagnose is an open-source agent skill for cloud and infrastructure.
- [Azure static web apps](https://ai-powered.com/skills/azure-static-web-apps): Azure static web apps is an open-source agent skill for cloud and infrastructure.
- [Azure well architected review](https://ai-powered.com/skills/azure-well-architected-review): Azure well architected review is an open-source agent skill for cloud and infrastructure.
- [Azure resource health diagnose](https://ai-powered.com/skills/azure-resource-health-diagnose): Azure resource health diagnose is an open-source agent skill for cloud and infrastructure.

---
Source: https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-performance-degradation
