Skip to content
Skill

Gke AI troubleshooting tpu performance degradation

Diagnoses GKE TPU training throughput drops using ML Diagnostics and Cloud Monitoring

Install
Scripts
None
Platforms
Claude Code, Codex, Cursor
GitHub stars
24.4K
Last commit
1 day ago
License
Apache-2.0
Listing updated
today

Similar skills

About Gke AI troubleshooting tpu performance degradation

Gke AI troubleshooting tpu performance degradation is an open-source agent skill for cloud and infrastructure. It is available on Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose.

This skill guides an agent through investigating a drop in Cloud TPU duty cycle or slower step times on Google Kubernetes Engine, when training pods keep running rather than crashing. It starts by listing ML runs and PERFORMANCE_DEGRADATION monitored events in a time window around the incident, then reads the analyzer reports to see which analyzer fired.

The agent then corroborates the findings with read-only PromQL queries over one-minute Cloud Monitoring node metrics for TPU duty cycle, HBM utilization, and host memory and CPU allocation. Where an infrastructure analyzer names culprit Compute Engine instance IDs, a kubectl query maps them to GKE nodes and topology labels. Separate reference paths cover workload bottlenecks, infrastructure or network fabric throttling, events with no detected analyzer, and healthy telemetry.

The workflow is read-only: the skill tells the agent not to drain, delete or recreate nodes, and leaves any fixes for the user to apply. It expects the gcloud alpha mldiagnostics component, kubectl, and access to a Google Cloud project with the relevant APIs enabled. ML Diagnostics only supports JAX workloads on TPUs.

When your agent uses it

Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).

From the skill's SKILL.md. Your agent reads this description to decide when to load the skill.

What it can access

From a static check of its 6 files (22 KB) on GitHub on 17 November 2026. Nothing was run. Read the files before you install a skill: it can tell your agent to run anything its tools allow.

Check another skill before you install it.

Install

Copy the skill folder to ~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation to use it in all your projects:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.claude/skills &&
rm -rf ~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"

For one project only, put the folder in .claude/skills/gke-ai-troubleshooting-tpu-performance-degradation inside it instead.

Claude Code docs on skills

Copy the skill folder to ~/.agents/skills/gke-ai-troubleshooting-tpu-performance-degradation to use it in every repository:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.agents/skills &&
rm -rf ~/.agents/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.agents/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"

For one repository only, put the folder in .agents/skills/gke-ai-troubleshooting-tpu-performance-degradation inside it instead.

Codex docs on skills

Copy the skill folder to ~/.cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation to use it in every project:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.cursor/skills &&
rm -rf ~/.cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"

For one project only, put the folder in .cursor/skills/gke-ai-troubleshooting-tpu-performance-degradation inside it instead.

Cursor docs on skills

Run in a terminal (add --scope workspace to install it for the current workspace only):

gemini skills install https://github.com/google/skills.git --path skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation

Or copy the folder to ~/.gemini/skills/gke-ai-troubleshooting-tpu-performance-degradation yourself.

Gemini CLI docs on skills

For GitHub Copilot in VS Code, copy the skill folder to ~/.copilot/skills/gke-ai-troubleshooting-tpu-performance-degradation:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.copilot/skills &&
rm -rf ~/.copilot/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation" ~/.copilot/skills/gke-ai-troubleshooting-tpu-performance-degradation &&
rm -rf "$tmp"

For one project only, put the folder in .github/skills/gke-ai-troubleshooting-tpu-performance-degradation inside it instead.

VS Code docs on skills

Get the folder:

git clone --depth 1 https://github.com/google/skills.git

Then copy skills/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradation into your agent's skills folder as gke-ai-troubleshooting-tpu-performance-degradation. Agents that support SKILL.md read one folder per skill.

The SKILL.md format

What it does

Monitored event triage
Lists ML runs and PERFORMANCE_DEGRADATION events in a time window and reads their analyzer reports.
System metrics correlation
Provides PromQL queries for TPU duty cycle, HBM use, and host memory and CPU utilization.
Culprit node mapping
Maps numeric Compute Engine instance IDs from analyzer output to GKE nodes and topology labels.
Category-based remediation
Routes the investigation to workload, infrastructure, no-detection or healthy-telemetry reference paths.
Read-only safety guardrails
Prevents node cordoning for workload saturation and forbids changes to managed instance groups.

Good for

  • Find out why TPU training throughput dropped without any pod crashing
  • Check whether a PERFORMANCE_DEGRADATION monitored event fired during a training run
  • Identify which GKE TPU nodes correspond to culprit instance IDs in an analyzer report
  • Confirm whether HBM or host CPU saturation is slowing a multi-slice training job

Gke AI troubleshooting tpu performance degradation FAQ

What does this skill need installed?

The Google Cloud SDK with the alpha component for gcloud alpha mldiagnostics, plus kubectl. The project needs billing, and the container, monitoring and hypercomputecluster APIs enabled.

Which workloads does it support?

Google Cloud ML Diagnostics only supports JAX on TPUs. Workload Monitoring supports the jobset and job GKE job types on GKE versions 1.36.0-gke.4681000 and later.

Does the skill change my cluster?

No. It is designed to run read-only diagnostic commands and gives the user any fix to apply themselves.

When should I use a different skill instead?

For complete XLA execution stalls with HANG_DETECTED logs, the skill points to the mxla-hang skill. For pod eviction or interruption restarts, it points to the jobset-interruption skill.

How do I install Gke AI troubleshooting tpu performance degradation in Claude Code?

Copy the skill folder from GitHub to ~/.claude/skills/gke-ai-troubleshooting-tpu-performance-degradation to use it in all your projects, or to .claude/skills/gke-ai-troubleshooting-tpu-performance-degradation in one project.

Does Gke AI troubleshooting tpu performance degradation run scripts?

No. It is instructions only, with no scripts. This comes from a static check of the files on GitHub; nothing was run. Read the files before you install it.

Which AI agents can use Gke AI troubleshooting tpu performance degradation?

Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose. Each reads skills in the SKILL.md format from its own folder; the install steps on this page list them.

What are the best alternatives to Gke AI troubleshooting tpu performance degradation?

The closest alternatives to Gke AI troubleshooting tpu performance degradation are Gem devops guidelines, AWS cost optimize, AWS cdk python setup, Azure smart city iot solution builder and AWS resource health diagnose.

Where this comes from

  • Description: from the skill's own SKILL.md.

More from google/skills

  1. Route platform choices to Google's own skills by fetching and filtering its skill catalog

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  2. Guardrails for gcloud commands: validate syntax with help, scope projects, dry-run first

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  3. Authenticate to Google Cloud: ADC, impersonation, service accounts and IAM roles

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  4. First-time Google Cloud setup: sign in, pick a project, link billing, with consent gates

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  5. Look up Google developer docs through the Developer Knowledge MCP server or REST fallback

    • Open source
    • 24.4K GitHub stars
    • Developer tools

Alternatives to Gke AI troubleshooting tpu performance degradation

  1. A DevOps checklist for deploys, Docker, Kubernetes, CI/CD, rollback and mobile releases

    • Open source
    • 41.3K GitHub stars
    • Cloud and infrastructure
  2. Finds AWS savings in IaC and live resources and files GitHub issues for each fix

    • Open source
    • 41.3K GitHub stars
    • Cloud and infrastructure
  3. Walks through setting up an AWS CDK Python project from prerequisites to first deploy

    • Open source
    • 41.3K GitHub stars
    • Cloud and infrastructure
  4. Diagnoses one AWS resource's health from CloudWatch data and drafts a remediation plan

    • Open source
    • 41.3K GitHub stars
    • Cloud and infrastructure
  5. Runs the SWA CLI to set up, test locally and deploy Azure Static Web Apps

    • Open source
    • 41.3K GitHub stars
    • Cloud and infrastructure