# Gke AI troubleshooting handle disruption GPU tpu

> Gke AI troubleshooting handle disruption GPU tpu is an open-source agent skill for cloud and infrastructure. Diagnoses and mitigates GKE GPU and TPU node disruptions from host maintenance events. It is available on Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose.

- **Kind:** Skill
- **Pricing:** Open source
- **Platforms:** Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- **Website:** https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu
- **Repository:** https://github.com/google/skills
- **GitHub stars:** 24,379
- **License:** Apache-2.0
- **Skill name:** gke-ai-troubleshooting-handle-disruption-gpu-tpu
- **Collection:** https://github.com/google/skills
- **Categories:** cloud and infrastructure
- **Listing updated:** 2026-11-18

## About

This skill guides an agent through finding out whether a GKE GPU or TPU node disruption was caused by Compute Engine host maintenance. It walks through checking nodes for scheduled maintenance labels, querying Cloud Monitoring node interruption metrics with PromQL, and searching Cloud Logging and node taints for active maintenance signals. The output is a short summary of confirmed or scheduled maintenance rather than raw log dumps.

When maintenance is confirmed or expected, the skill recommends three protections together: graceful termination with a longer terminationGracePeriodSeconds so workloads can save state, opportunistic maintenance for idle accelerator nodes, and PodDisruptionBudgets to keep minimum replicas available. It also warns against tolerating the impending-node-termination taint. If the signals show no maintenance, it steers the investigation toward application-level causes such as OOM kills or CUDA errors.

The skill expects a project ID, location, cluster name and incident timestamp before it begins a live investigation, and it relies on kubectl, gcloud and Cloud Monitoring or Logging access. It contains no bundled scripts.

## When your agent uses it

> Diagnoses, predicts, and mitigates node disruptions during Compute Engine host maintenance and hardware or software maintenance events for GPU and TPU workloads on GKE. Use when diagnosing node disruptions, predicting host maintenance events on GPU/TPU nodepools, inspecting node interruption PromQL metrics, auditing node taints, or configuring workload protection strategies (graceful termination, opportunistic maintenance, PodDisruptionBudgets). Don't use for general GKE cluster creation, network policy configuration, or non-disruption workload deployment.

## What it can access

From a static check of the files on GitHub; nothing was run.

- [Instructions only, no scripts](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu)
- [License: Apache-2.0](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu)

## Features

- **Scheduled maintenance check:** Lists nodes carrying the scheduled maintenance time label to spot upcoming host maintenance.
- **PromQL interruption metrics:** Provides PromQL queries over node and node pool interruption counts for HW/SW maintenance.
- **Log and taint inspection:** Checks Cloud Logging and node taints for active maintenance and cordoned nodes.
- **Workload protection guidance:** Recommends graceful termination, opportunistic maintenance and PodDisruptionBudgets together.
- **Negative result handling:** Directs investigation to application-level causes when host maintenance is ruled out.

## Good for

- Find out why a GPU training job on GKE was restarted overnight
- Check whether upcoming host maintenance is scheduled on TPU nodes
- Build a Cloud Monitoring query for node interruptions by node pool
- Configure graceful termination for a checkpointing ML workload on GKE
- Set up a PodDisruptionBudget for an inference deployment on GPU nodes

## Install

### Claude Code

Copy the skill folder to `~/.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` to use it in all your projects:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.claude/skills &&
rm -rf ~/.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu" ~/.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
rm -rf "$tmp"
```

For one project only, put the folder in `.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` inside it instead.

### Codex

Copy the skill folder to `~/.agents/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` to use it in every repository:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.agents/skills &&
rm -rf ~/.agents/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu" ~/.agents/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
rm -rf "$tmp"
```

For one repository only, put the folder in `.agents/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` inside it instead.

### Cursor

Copy the skill folder to `~/.cursor/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` to use it in every project:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.cursor/skills &&
rm -rf ~/.cursor/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu" ~/.cursor/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
rm -rf "$tmp"
```

For one project only, put the folder in `.cursor/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` inside it instead.

### Gemini CLI

Run in a terminal (add `--scope workspace` to install it for the current workspace only):

```sh
gemini skills install https://github.com/google/skills.git --path skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu
```

Or copy the folder to `~/.gemini/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` yourself.

### VS Code

For GitHub Copilot in VS Code, copy the skill folder to `~/.copilot/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu`:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.copilot/skills &&
rm -rf ~/.copilot/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu" ~/.copilot/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu &&
rm -rf "$tmp"
```

For one project only, put the folder in `.github/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu` inside it instead.

### Other agents

Get the folder:

```sh
git clone --depth 1 https://github.com/google/skills.git
```

Then copy `skills/skills/cloud/gke-ai-troubleshooting-handle-disruption-gpu-tpu` into your agent's skills folder as `gke-ai-troubleshooting-handle-disruption-gpu-tpu`. Agents that support SKILL.md read one folder per skill.

## FAQ

### What does the skill need before it starts an investigation?

For a live disruption it requires project ID, location, cluster name and timestamp, and optionally node name, workload name, namespace and node pool name.

### Does it change anything in my cluster?

Its steps are read-oriented checks such as kubectl get, PromQL queries and log filters. It recommends configuration changes but the content does not describe the skill applying them.

### Which tools does it use?

It refers to kubectl, gcloud with a usage attribution prefix, Cloud Logging queries and Cloud Monitoring PromQL.

### Does it cover general GKE cluster setup?

No. Its description says it does not cover general cluster creation, network policy configuration or non-disruption deployments.

### How do I install Gke AI troubleshooting handle disruption GPU tpu in Claude Code?

Copy the skill folder from GitHub to ~/.claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu to use it in all your projects, or to .claude/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu in one project.

### Does Gke AI troubleshooting handle disruption GPU tpu run scripts?

No. It is instructions only, with no scripts. This comes from a static check of the files on GitHub; nothing was run. Read the files before you install it.

### Which AI agents can use Gke AI troubleshooting handle disruption GPU tpu?

Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose. Each reads skills in the SKILL.md format from its own folder; the install steps on this page list them.

### What are the best alternatives to Gke AI troubleshooting handle disruption GPU tpu?

The closest alternatives to Gke AI troubleshooting handle disruption GPU tpu are Gem devops guidelines, AWS cost optimize, AWS cdk python setup, Azure smart city iot solution builder and AWS resource health diagnose.

## Where this comes from

- Description: from the skill's own SKILL.md.

## Alternatives

- [Gem devops guidelines](https://ai-powered.com/skills/gem-devops-guidelines): Gem devops guidelines is an open-source agent skill for cloud and infrastructure.
- [AWS cost optimize](https://ai-powered.com/skills/aws-cost-optimize): AWS cost optimize is an open-source agent skill for cloud and infrastructure.
- [AWS cdk python setup](https://ai-powered.com/skills/aws-cdk-python-setup): AWS cdk python setup is an open-source agent skill for cloud and infrastructure.
- [Azure smart city iot solution builder](https://ai-powered.com/skills/azure-smart-city-iot-solution-builder): Azure smart city iot solution builder is an open-source agent skill for cloud and infrastructure.
- [AWS resource health diagnose](https://ai-powered.com/skills/aws-resource-health-diagnose): AWS resource health diagnose is an open-source agent skill for cloud and infrastructure.
- [Azure static web apps](https://ai-powered.com/skills/azure-static-web-apps): Azure static web apps is an open-source agent skill for cloud and infrastructure.
- [Azure well architected review](https://ai-powered.com/skills/azure-well-architected-review): Azure well architected review is an open-source agent skill for cloud and infrastructure.
- [Azure resource health diagnose](https://ai-powered.com/skills/azure-resource-health-diagnose): Azure resource health diagnose is an open-source agent skill for cloud and infrastructure.

---
Source: https://ai-powered.com/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu
