# Gke AI troubleshooting tpu vbar oom

> Gke AI troubleshooting tpu vbar oom is an open-source agent skill for cloud and infrastructure. Diagnoses vbar_control_agent OOMs and TPU v6e device failures on GKE clusters. It is available on Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose.

- **Kind:** Skill
- **Pricing:** Open source
- **Platforms:** Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- **Website:** https://github.com/google/skills/tree/main/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom
- **Repository:** https://github.com/google/skills
- **GitHub stars:** 24,379
- **License:** Apache-2.0
- **Skill name:** gke-ai-troubleshooting-tpu-vbar-oom
- **Collection:** https://github.com/google/skills
- **Categories:** [cloud and infrastructure](https://ai-powered.com/categories/cloud-and-infrastructure)
- **Listing updated:** 2026-11-18

## About

This skill guides an agent through troubleshooting crashes and memory errors on TPU v6e nodes in Google Kubernetes Engine. It focuses on vbar_control_agent segfaults, memory cgroup OOM messages in serial console logs, and checksum corruption reported by the tpu-device-plugin, which the skill links to race conditions during TPU device resets and frequent metrics polling.

The workflow starts by establishing a time window around the incident and then searches Cloud Logging with predefined filter templates. It checks serial console output for OOM kills tied to vbar_control_agent, looks for metric fetch failures in tpu-device-plugin logs, and inspects pods for custom TPU monitoring code that frequently queries host metrics. When custom collection is found, the agent is directed to recommend disabling it, and it can advise waiting for a future GKE version with a more resilient agent.

The skill expects gcloud access to the project and cluster, with Cloud Logging enabled, plus kubectl for listing pod images and environment details. Bash is used for the commands. It is scoped to TPU v6e issues and is not meant for general container OOM debugging.

## When your agent uses it

> Diagnoses and prevents vbar_control_agent segfaults, out-of-memory (OOM) errors, and TPU device initialization failures on TPU v6e nodes in GKE caused by race conditions during TPU device resets or high-frequency metrics polling. Use when troubleshooting vbar_control_agent crashes, memory cgroup OOMs in serial console logs, tpu-device-plugin metrics checksum corruption errors, or custom TPU metrics collection conflicts on GKE TPU v6e nodes. Don't use for general non-TPU container OOM troubleshooting or standard GKE node lifecycle operations.

## What it can access

From a static check of the files on GitHub; nothing was run.

- [Runs shell scripts](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom)
- [No network calls found in its scripts](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom)
- [License: Apache-2.0](https://github.com/google/skills/tree/HEAD/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom)

## Features

- **Serial console OOM detection:** Filter templates locate memory cgroup OOM kills of vbar_control_agent in node serial logs.
- **Device plugin failure checks:** Queries tpu-device-plugin error logs for metric checksum mismatches indicating vBAR corruption.
- **Custom metrics collector audit:** Looks for workloads using libtpu monitoring that poll host metrics too often.
- **Time window calculation:** Derives a 30-minute search window on either side of the reported incident time.
- **Mitigation guidance:** Recommends disabling custom metrics collection and upgrading GKE once a fix ships.

## Good for

- Investigate why vbar_control_agent crashed on a TPU v6e node
- Find out whether OOM kills appear in serial console logs for a node
- Check GKE logs for tpu-device-plugin checksum corruption errors
- Determine if a custom TPU metrics script is causing node instability

## Install

### Claude Code

Copy the skill folder to `~/.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom` to use it in all your projects:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.claude/skills &&
rm -rf ~/.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom" ~/.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
rm -rf "$tmp"
```

For one project only, put the folder in `.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom` inside it instead.

### Codex

Copy the skill folder to `~/.agents/skills/gke-ai-troubleshooting-tpu-vbar-oom` to use it in every repository:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.agents/skills &&
rm -rf ~/.agents/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom" ~/.agents/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
rm -rf "$tmp"
```

For one repository only, put the folder in `.agents/skills/gke-ai-troubleshooting-tpu-vbar-oom` inside it instead.

### Cursor

Copy the skill folder to `~/.cursor/skills/gke-ai-troubleshooting-tpu-vbar-oom` to use it in every project:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.cursor/skills &&
rm -rf ~/.cursor/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom" ~/.cursor/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
rm -rf "$tmp"
```

For one project only, put the folder in `.cursor/skills/gke-ai-troubleshooting-tpu-vbar-oom` inside it instead.

### Gemini CLI

Run in a terminal (add `--scope workspace` to install it for the current workspace only):

```sh
gemini skills install https://github.com/google/skills.git --path skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom
```

Or copy the folder to `~/.gemini/skills/gke-ai-troubleshooting-tpu-vbar-oom` yourself.

### VS Code

For GitHub Copilot in VS Code, copy the skill folder to `~/.copilot/skills/gke-ai-troubleshooting-tpu-vbar-oom`:

```sh
tmp=$(mktemp -d) &&
git clone --depth 1 https://github.com/google/skills.git "$tmp" &&
mkdir -p ~/.copilot/skills &&
rm -rf ~/.copilot/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
cp -R "$tmp/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom" ~/.copilot/skills/gke-ai-troubleshooting-tpu-vbar-oom &&
rm -rf "$tmp"
```

For one project only, put the folder in `.github/skills/gke-ai-troubleshooting-tpu-vbar-oom` inside it instead.

### Other agents

Get the folder:

```sh
git clone --depth 1 https://github.com/google/skills.git
```

Then copy `skills/skills/cloud/gke-ai-troubleshooting-tpu-vbar-oom` into your agent's skills folder as `gke-ai-troubleshooting-tpu-vbar-oom`. Agents that support SKILL.md read one folder per skill.

## FAQ

### Which TPU hardware does this skill cover?

It targets TPU v6e nodes in GKE, as stated in its description.

### What does it need to run?

Cloud Logging enabled for the project and access to the project and cluster through gcloud or an equivalent tool.

### When should it not be used?

The description says not to use it for general non-TPU container OOM troubleshooting or standard GKE node lifecycle operations.

### How do I install Gke AI troubleshooting tpu vbar oom in Claude Code?

Copy the skill folder from GitHub to ~/.claude/skills/gke-ai-troubleshooting-tpu-vbar-oom to use it in all your projects, or to .claude/skills/gke-ai-troubleshooting-tpu-vbar-oom in one project.

### Does Gke AI troubleshooting tpu vbar oom run scripts?

Yes. It runs shell scripts. No network calls were found in its scripts. This comes from a static check of the files on GitHub; nothing was run. Read the files before you install it.

### Which AI agents can use Gke AI troubleshooting tpu vbar oom?

Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf and Goose. Each reads skills in the SKILL.md format from its own folder; the install steps on this page list them.

### What are the best alternatives to Gke AI troubleshooting tpu vbar oom?

The closest alternatives to Gke AI troubleshooting tpu vbar oom are Aks GPU inference, Gke AI troubleshooting tpu performance degradation, Gke AI troubleshooting tpu metrics monitoring, Gke AI troubleshooting node unresponsive timeout and Gke AI troubleshooting tpu reservation stockout.

## Where this comes from

- Description: from the skill's own SKILL.md.

## Alternatives

- [Aks GPU inference](https://ai-powered.com/skills/aks-gpu-inference): Aks GPU inference is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu performance degradation](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-performance-degradation): Gke AI troubleshooting tpu performance degradation is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu metrics monitoring](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-metrics-monitoring): Gke AI troubleshooting tpu metrics monitoring is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting node unresponsive timeout](https://ai-powered.com/skills/gke-ai-troubleshooting-node-unresponsive-timeout): Gke AI troubleshooting node unresponsive timeout is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu reservation stockout](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-reservation-stockout): Gke AI troubleshooting tpu reservation stockout is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu vm repair stuck](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-vm-repair-stuck): Gke AI troubleshooting tpu vm repair stuck is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting tpu mxla hang](https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-mxla-hang): Gke AI troubleshooting tpu mxla hang is an open-source agent skill for cloud and infrastructure.
- [Gke AI troubleshooting handle disruption GPU tpu](https://ai-powered.com/skills/gke-ai-troubleshooting-handle-disruption-gpu-tpu): Gke AI troubleshooting handle disruption GPU tpu is an open-source agent skill for cloud and infrastructure.

---
Source: https://ai-powered.com/skills/gke-ai-troubleshooting-tpu-vbar-oom
