Skip to content

Nemo mbridge perf cuda graphs vs Nemo mbridge resiliency

Nemo mbridge perf cuda graphs is an open-source agent skill for developer tools. Nemo mbridge resiliency is an open-source agent skill for developer tools.

  • Nemo mbridge perf cuda graphs is best for enable cuda graphs for a dense model training run in megatron bridge.
  • Nemo mbridge resiliency is best for enable automatic restart after hangs or node failures in a slurm training job.
What it is
Nemo mbridge perf cuda graphs is an open-source agent skill for developer tools.
Nemo mbridge resiliency is an open-source agent skill for developer tools.
Best for
Enable CUDA graphs for a dense model training run in Megatron Bridge
Enable automatic restart after hangs or node failures in a Slurm training job
Kind
Skill
Skill
Pricing
Open source
Open source
Platforms
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
GitHub stars
4K
4K
Last commit
Nov 2026
Nov 2026
License
Apache-2.0
Apache-2.0
Open source
Good for
Enable CUDA graphs for a dense model training run in Megatron Bridge; Pick a graph scope for a dropless MoE pretraining configuration; Diagnose a crash caused by a CUDA graph config change
Enable automatic restart after hangs or node failures in a Slurm training job; Find slow GPUs in a distributed run using straggler detection; Set up graceful checkpointing before a Slurm job is preempted

What Nemo mbridge perf cuda graphs does

  • Two graph implementations

    Explains the local full-iteration wrapper and the Transformer Engine per-module scoped graphs, and when each applies.

  • Scope selection guidance

    Recommends starting scopes for dense, dropless MoE and VLM models and how graphs interact with selective recompute.

  • Configuration constraints

    Lists required settings and conflicts that the framework enforces, such as RNG tracker and NaN-check requirements.

  • Pitfalls and memory notes

    Covers memory overhead, variable-length sequence limits, cleanup of graph objects and workload-dependent speedups.

  • Bring-up and validation steps

    Gives an ordered procedure for enabling graphs and comparing replay timing against eager runs, plus test commands.

What Nemo mbridge resiliency does

  • Fault tolerance setup

    Covers enabling the fault tolerance package through a NeMo Run plugin or direct config with ft_launcher.

  • Straggler detection

    Configures NVRx straggler detection thresholds, reporting intervals and whether to stop training on detection.

  • Preemption handling

    Sets up signal handling so jobs can checkpoint before Slurm time limits or preemption.

  • In-process restart

    Explains restart granularity, hang timeouts and the environment variables required for in-process recovery.

  • Re-run state machine

    Describes the experimental rerun modes for validating results and catching NaN or spiky loss.

  • Checkpoint saving options

    Covers async checkpoint saves and local non-persistent checkpoints using NVRx.

Nemo mbridge perf cuda graphs vs Nemo mbridge resiliency: questions

Is Nemo mbridge perf cuda graphs or Nemo mbridge resiliency cheaper?

Both are open source. Compare current plans on each official site.

What is the difference between Nemo mbridge perf cuda graphs and Nemo mbridge resiliency?

Nemo mbridge perf cuda graphs is an open-source agent skill for developer tools. Nemo mbridge resiliency is an open-source agent skill for developer tools.

Keep comparing