Skip to content
Skill

Gke AI troubleshooting tpu mxla hang

Guides agents through diagnosing GKE TPU multi-slice training hangs using MXLA analyzer

Install
Scripts
None
Platforms
Claude Code, Codex, Cursor
GitHub stars
24.4K
Last commit
2 days ago
License
Apache-2.0
Listing updated
1 day ago

Similar skills

About Gke AI troubleshooting tpu mxla hang

Gke AI troubleshooting tpu mxla hang is an open-source agent skill for cloud and infrastructure. It is available on Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose.

This skill helps an agent investigate multi-slice Cloud TPU training jobs on Google Kubernetes Engine that freeze without advancing steps. It walks through a fixed sequence: searching Cloud Logging for Megascale HANG_DETECTED messages, listing ML Diagnostics hang monitored events, and reading the Megascale XLA Hang Analyzer report for its detection state and suggested causes.

The agent then correlates those findings with one-minute multi-slice latency and node duty-cycle metrics via PromQL queries, and maps numeric culprit instance IDs to GKE node names and topology labels with a read-only kubectl command. Based on the suspected root cause, the skill routes to separate reference guides for compiler divergence, input stalls, hardware or network faults, undetected or unknown causes, and healthy telemetry.

It is built around read-only investigation. Remediation steps are left for the user to apply, and the skill warns against cordoning or replacing nodes for software-side hangs. It expects a gcloud CLI with the alpha component, kubectl, and IAM read access to the relevant GKE, logging, and monitoring resources, and it applies to JAX workloads only.

When your agent uses it

Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).

From the skill's SKILL.md. Your agent reads this description to decide when to load the skill.

What it can access

From a static check of its 7 files (23 KB) on GitHub on 17 November 2026. Nothing was run. Read the files before you install a skill: it can tell your agent to run anything its tools allow.

Check another skill before you install it.

Install

Copy the skill folder to ~/.claude/skills/gke-ai-troubleshooting-tpu-mxla-hang to use it in all your projects:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.claude/skills &&
rm -rf ~/.claude/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-mxla-hang" ~/.claude/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
rm -rf "$tmp"

For one project only, put the folder in .claude/skills/gke-ai-troubleshooting-tpu-mxla-hang inside it instead.

Claude Code docs on skills

Copy the skill folder to ~/.agents/skills/gke-ai-troubleshooting-tpu-mxla-hang to use it in every repository:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.agents/skills &&
rm -rf ~/.agents/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-mxla-hang" ~/.agents/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
rm -rf "$tmp"

For one repository only, put the folder in .agents/skills/gke-ai-troubleshooting-tpu-mxla-hang inside it instead.

Codex docs on skills

Copy the skill folder to ~/.cursor/skills/gke-ai-troubleshooting-tpu-mxla-hang to use it in every project:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.cursor/skills &&
rm -rf ~/.cursor/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-mxla-hang" ~/.cursor/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
rm -rf "$tmp"

For one project only, put the folder in .cursor/skills/gke-ai-troubleshooting-tpu-mxla-hang inside it instead.

Cursor docs on skills

Run in a terminal (add --scope workspace to install it for the current workspace only):

gemini skills install https://github.com/google/skills.git --path skills/cloud/gke-ai-troubleshooting-tpu-mxla-hang

Or copy the folder to ~/.gemini/skills/gke-ai-troubleshooting-tpu-mxla-hang yourself.

Gemini CLI docs on skills

For GitHub Copilot in VS Code, copy the skill folder to ~/.copilot/skills/gke-ai-troubleshooting-tpu-mxla-hang:

tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.copilot/skills &&
rm -rf ~/.copilot/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-mxla-hang" ~/.copilot/skills/gke-ai-troubleshooting-tpu-mxla-hang &&
rm -rf "$tmp"

For one project only, put the folder in .github/skills/gke-ai-troubleshooting-tpu-mxla-hang inside it instead.

VS Code docs on skills

Get the folder:

git clone --depth 1 https://github.com/google/skills.git

Then copy skills/skills/cloud/gke-ai-troubleshooting-tpu-mxla-hang into your agent's skills folder as gke-ai-troubleshooting-tpu-mxla-hang. Agents that support SKILL.md read one folder per skill.

The SKILL.md format

What it does

Hang log detection
Searches Cloud Logging for Megascale HANG_DETECTED entries to confirm a hang and find the first stalled pods.
Monitored event analysis
Lists and describes ML Diagnostics hang events to read the MXLA analyzer's detection state and recommended actions.
Multi-slice latency correlation
Runs PromQL queries over one-minute collective, DCN and host-to-device latency and TPU duty-cycle metrics.
Node topology mapping
Maps culprit Compute Engine instance IDs to GKE node names and topology block labels with a read-only kubectl query.
Root-cause routing
Directs the investigation to separate guides for compiler, input, hardware, unknown and healthy-telemetry cases.
Read-only safety guardrails
Prohibits cluster-changing commands and forbids node cordoning for compiler or input-stall hangs.

Good for

  • Find out why a multi-slice TPU training job stopped progressing without errors
  • Check whether a HANG_DETECTED log matches an MXLA hang monitored event
  • Identify which GKE nodes correspond to culprit instance IDs from a hang analyzer report
  • Compare multi-slice collective latency across pods during a suspected stall
  • Determine whether a training freeze comes from compiler divergence or a data input stall

Gke AI troubleshooting tpu mxla hang FAQ

Which frameworks does this skill support?

ML Diagnostics only supports JAX on TPUs, so workloads using PyTorch or other custom resource types won't show up in the gcloud alpha mldiagnostics commands this skill relies on.

What tools and permissions are needed?

It requires the Google Cloud SDK with the alpha component, kubectl, and viewer roles for ML Diagnostics, Monitoring, Logging and GKE. Remediation steps need Kubernetes Engine Cluster Admin.

Does the skill change my cluster?

No. It is designed to run read-only diagnostic commands, and any fixes are left for the user to apply themselves.

When should I use a different skill instead?

For gradual step-time slowdowns without hangs, use the TPU performance degradation skill. For pod preemption or eviction restarts, use the JobSet interruption skill.

How do I install Gke AI troubleshooting tpu mxla hang in Claude Code?

Copy the skill folder from GitHub to ~/.claude/skills/gke-ai-troubleshooting-tpu-mxla-hang to use it in all your projects, or to .claude/skills/gke-ai-troubleshooting-tpu-mxla-hang in one project.

Does Gke AI troubleshooting tpu mxla hang run scripts?

No. It is instructions only, with no scripts. This comes from a static check of the files on GitHub; nothing was run. Read the files before you install it.

Which AI agents can use Gke AI troubleshooting tpu mxla hang?

Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose. Each reads skills in the SKILL.md format from its own folder; the install steps on this page list them.

What are the best alternatives to Gke AI troubleshooting tpu mxla hang?

The closest alternatives to Gke AI troubleshooting tpu mxla hang are Gke AI troubleshooting tpu performance degradation, Aks GPU inference, Gke AI troubleshooting tpu slice mig stuck updating, Aks troubleshooting and Gke AI troubleshooting handle disruption GPU tpu.

Where this comes from

  • Description: from the skill's own SKILL.md.

More from google/skills

  1. Route platform choices to Google's own skills by fetching and filtering its skill catalog

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  2. Guardrails for gcloud commands: validate syntax with help, scope projects, dry-run first

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  3. Authenticate to Google Cloud: ADC, impersonation, service accounts and IAM roles

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  4. First-time Google Cloud setup: sign in, pick a project, link billing, with consent gates

    • Open source
    • 24.4K GitHub stars
    • Cloud and infrastructure
  5. Look up Google developer docs through the Developer Knowledge MCP server or REST fallback

    • Open source
    • 24.4K GitHub stars
    • Developer tools