# Gke AI troubleshooting tpu metrics monitoring

> Gke AI troubleshooting tpu metrics monitoring is an open-source agent skill for cloud and infrastructure. Monitors GKE TPU nodes, node pools and workloads using system metrics and PromQL. It is available on Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose.

- **Kind:** Skill
- **Pricing:** Open source
- **Platforms:** Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- **Website:** https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring
- **Repository:** https://github.com/google/skills
- **GitHub stars:** 24,379
- **License:** Apache-2.0
- **Skill name:** gke-ai-troubleshooting-tpu-metrics-monitoring
- **Collection:** https://github.com/google/skills
- **Categories:** [cloud and infrastructure](https://ai-powered.com/categories/cloud-and-infrastructure)
- **Listing updated:** 2026-11-18

## About

This skill guides an agent through watching Google Kubernetes Engine clusters that run TPU accelerators. It checks whether a workload is set up to export runtime metrics, then reads TensorCore duty cycle, TPU memory and utilization figures from Cloud Monitoring for both containers and nodes.

The workflow goes beyond accelerator usage. It queries node readiness conditions, multi-host TPU node pool status and availability, and interruption counts broken down by type and reason, including host maintenance events. It also includes PromQL to derive mean time to recovery and mean time between interruptions over a seven-day window, which helps determine whether a slowdown or stop comes from the underlying infrastructure.

The guide expects a GCP project, a GKE cluster name and location, and optionally a node or node pool name, which the agent can look up with its own cloud tools. Queries are written for Cloud Monitoring and the steps assume GKE system metrics are enabled. The skill ships with bash scripts and a usage attribution reference file.

## When your agent uses it

> Monitors and troubleshoots GKE TPU workloads, nodes, and node pools using GKE system metrics and PromQL. Use when monitoring TensorCore duty cycle, TPU memory, node readiness, multi-host TPU node pool availability, host maintenance or preemption interruptions, and calculating MTTR or MTBI metrics for GKE TPUs. Don't use for general non-TPU GKE workload monitoring or non-metric TPU debugging.

## What it can access

From a static check of the files on GitHub; nothing was run.

- [Runs shell scripts](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring)
- [No network calls found in its scripts](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring)
- [License: Apache-2.0](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring)

## Features

- **Runtime metrics setup check:** Verifies pod ports, JAX version, GKE version and system metrics settings needed for TPU metric export.
- **TPU utilization metrics:** Reads duty cycle, memory usage and tensorcore utilization for TPU containers and nodes.
- **Node readiness queries:** Lists PromQL checks for whether TPU nodes are Ready and surfaces non-Ready conditions.
- **Multi-host node pool availability:** Tracks node pool status and whether all hosts in a multi-host TPU pool are available.
- **Interruption analysis:** Breaks down maintenance, preemption and termination events by type and reason.
- **Recovery and interruption metrics:** Calculates mean time to recovery and mean time between interruptions over seven days.

## Good for

- Check whether a stalled TPU training job lost a node to maintenance
- See how busy the TensorCores are on a GKE TPU cluster
- Confirm that a multi-host TPU node pool has all hosts available
- Measure recovery time and interruption frequency for TPU nodes
- Find which node pools had recent preemption or host maintenance events

## Install

### Claude Code

Copy the skill folder to `~/.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` to use it in all your projects:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.claude/skills &&
rm -rf ~/.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring" ~/.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
rm -rf "$tmp"
```

For one project only, put the folder in `.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` inside it instead.

### Codex

Copy the skill folder to `~/.agents/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` to use it in every repository:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.agents/skills &&
rm -rf ~/.agents/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring" ~/.agents/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
rm -rf "$tmp"
```

For one repository only, put the folder in `.agents/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` inside it instead.

### Cursor

Copy the skill folder to `~/.cursor/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` to use it in every project:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.cursor/skills &&
rm -rf ~/.cursor/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring" ~/.cursor/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
rm -rf "$tmp"
```

For one project only, put the folder in `.cursor/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` inside it instead.

### Gemini CLI

Run in a terminal (add `--scope workspace` to install it for the current workspace only):

```sh
gemini skills install https://github.com/google/skills.git --path skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring
```

Or copy the folder to `~/.gemini/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` yourself.

### VS Code

For GitHub Copilot in VS Code, copy the skill folder to `~/.copilot/skills/gke-ai-troubleshooting-tpu-metrics-monitoring`:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.copilot/skills &&
rm -rf ~/.copilot/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring" ~/.copilot/skills/gke-ai-troubleshooting-tpu-metrics-monitoring &&
rm -rf "$tmp"
```

For one project only, put the folder in `.github/skills/gke-ai-troubleshooting-tpu-metrics-monitoring` inside it instead.

### Other agents

Get the folder:

```sh
git clone --depth 1 https://github.com/google/skills.git
```

Then copy `skills/skills/cloud/gke-ai-troubleshooting-tpu-metrics-monitoring` into your agent's skills folder as `gke-ai-troubleshooting-tpu-metrics-monitoring`. Agents that support SKILL.md read one folder per skill.

## FAQ

### Which environment does this skill need?

It needs a GCP project, a GKE cluster name and location, and optionally node or node pool names. The queries run against Cloud Monitoring with GKE system metrics enabled.

### Does the skill change any cluster resources?

Its steps are read-only metric queries and configuration checks. It does not describe making changes to the cluster.

### Which GKE versions are required?

The guide lists minimums: GKE 1.27.4-gke.900 or later for runtime metrics, 1.28.1-gke.1066000 or later for host metrics, and 1.32.1-gke.1357001 or later for node status conditions.

### When should it not be used?

According to its description, it should not be used for general non-TPU GKE workload monitoring or for TPU debugging that does not involve metrics.

### How do I install Gke AI troubleshooting tpu metrics monitoring in Claude Code?

Copy the skill folder from GitHub to ~/.claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring to use it in all your projects, or to .claude/skills/gke-ai-troubleshooting-tpu-metrics-monitoring in one project.

### Does Gke AI troubleshooting tpu metrics monitoring run scripts?

Yes. It runs shell scripts. No network calls were found in its scripts. This comes from a static check of the files on GitHub; nothing was run. Read the files before you install it.

### Which AI agents can use Gke AI troubleshooting tpu metrics monitoring?

Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose. Each reads skills in the SKILL.md format from its own folder; the install steps on this page list them.

### What are the best alternatives to Gke AI troubleshooting tpu metrics monitoring?

The closest alternatives to Gke AI troubleshooting tpu metrics monitoring are Gke AI troubleshooting tpu performance degradation, Gke AI troubleshooting handle disruption GPU tpu, Gke AI troubleshooting tpu reservation stockout, Gke AI troubleshooting node unresponsive timeout and Aks GPU inference.

## Where this comes from

- Description: from the skill's own SKILL.md.

## Alternatives

- [Gke AI troubleshooting tpu performance degradation](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-performance-degradation): Gke AI troubleshooting tpu performance degradation is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting handle disruption GPU tpu](https://ai-powered.com/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu): Gke AI troubleshooting handle disruption GPU tpu is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu reservation stockout](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-reservation-stockout): Gke AI troubleshooting tpu reservation stockout is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting node unresponsive timeout](https://ai-powered.com/skills/gke-ai-troubleshooting-node-unresponsive-timeout): Gke AI troubleshooting node unresponsive timeout is an open-source agent skill for cloud and infrastructure.
- [Aks GPU inference](https://ai-powered.com/skills/aks-gpu-inference): Aks GPU inference is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu vbar oom](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-vbar-oom): Gke AI troubleshooting tpu vbar oom is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu vm repair stuck](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-vm-repair-stuck): Gke AI troubleshooting tpu vm repair stuck is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu mxla hang](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-mxla-hang): Gke AI troubleshooting tpu mxla hang is an open-source agent skill for cloud and infrastructure.

---
Source: https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-metrics-monitoring
