Nemo mbridge perf cuda graphs vs Nemo mbridge resiliency
Nemo mbridge perf cuda graphs is an open-source agent skill for developer tools. Nemo mbridge resiliency is an open-source agent skill for developer tools.
- Nemo mbridge perf cuda graphs is best for enable cuda graphs for a dense model training run in megatron bridge.
- Nemo mbridge resiliency is best for enable automatic restart after hangs or node failures in a slurm training job.
- What it is
- Nemo mbridge perf cuda graphs is an open-source agent skill for developer tools.Nemo mbridge resiliency is an open-source agent skill for developer tools.
- Best for
- Enable CUDA graphs for a dense model training run in Megatron BridgeEnable automatic restart after hangs or node failures in a Slurm training job
- Kind
- SkillSkill
- Pricing
- Open sourceOpen source
- Platforms
- Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, GooseClaude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- GitHub stars
- 4K4K
- Last commit
- Nov 2026Nov 2026
- License
- Apache-2.0Apache-2.0
- Open source
- Good for
- Enable CUDA graphs for a dense model training run in Megatron Bridge; Pick a graph scope for a dropless MoE pretraining configuration; Diagnose a crash caused by a CUDA graph config changeEnable automatic restart after hangs or node failures in a Slurm training job; Find slow GPUs in a distributed run using straggler detection; Set up graceful checkpointing before a Slurm job is preempted
What Nemo mbridge perf cuda graphs does
Two graph implementations
Explains the local full-iteration wrapper and the Transformer Engine per-module scoped graphs, and when each applies.
Scope selection guidance
Recommends starting scopes for dense, dropless MoE and VLM models and how graphs interact with selective recompute.
Configuration constraints
Lists required settings and conflicts that the framework enforces, such as RNG tracker and NaN-check requirements.
Pitfalls and memory notes
Covers memory overhead, variable-length sequence limits, cleanup of graph objects and workload-dependent speedups.
Bring-up and validation steps
Gives an ordered procedure for enabling graphs and comparing replay timing against eager runs, plus test commands.
What Nemo mbridge resiliency does
Fault tolerance setup
Covers enabling the fault tolerance package through a NeMo Run plugin or direct config with ft_launcher.
Straggler detection
Configures NVRx straggler detection thresholds, reporting intervals and whether to stop training on detection.
Preemption handling
Sets up signal handling so jobs can checkpoint before Slurm time limits or preemption.
In-process restart
Explains restart granularity, hang timeouts and the environment variables required for in-process recovery.
Re-run state machine
Describes the experimental rerun modes for validating results and catching NaN or spiky loss.
Checkpoint saving options
Covers async checkpoint saves and local non-persistent checkpoints using NVRx.
Nemo mbridge perf cuda graphs vs Nemo mbridge resiliency: questions
Is Nemo mbridge perf cuda graphs or Nemo mbridge resiliency cheaper?
Both are open source. Compare current plans on each official site.
What is the difference between Nemo mbridge perf cuda graphs and Nemo mbridge resiliency?
Nemo mbridge perf cuda graphs is an open-source agent skill for developer tools. Nemo mbridge resiliency is an open-source agent skill for developer tools.
