Jetson LLM benchmark vs Jetson LLM serve
Jetson LLM benchmark is an open-source agent skill for ai models and llm tooling. Jetson LLM serve is an open-source agent skill for ai models and llm tooling.
- Jetson LLM benchmark is best for measure time to first token and throughput for a model served by vllm on a jetson.
- Jetson LLM serve is best for serve a hugging face llm on my jetson orin as an openai-compatible endpoint.
- What it is
- Jetson LLM benchmark is an open-source agent skill for ai models and llm tooling.Jetson LLM serve is an open-source agent skill for ai models and llm tooling.
- Best for
- Measure time to first token and throughput for a model served by vLLM on a JetsonServe a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpoint
- Kind
- SkillSkill
- Pricing
- Open sourceOpen source
- Platforms
- Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, GooseClaude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- GitHub stars
- 4K4K
- Last commit
- Nov 2026Nov 2026
- License
- Apache-2.0Apache-2.0
- Open source
- Good for
- Measure time to first token and throughput for a model served by vLLM on a Jetson; Compare a model's speed before and after changing power mode or runtime flags; Benchmark a GGUF model with llama.cpp on a Jetson Orin or ThorServe a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpoint; Start a vision-language model server on Jetson Thor for my app; Launch an SGLang server on Thor for tool-use workflows
What Jetson LLM benchmark does
Runtime-specific benchmark wrappers
Separate scripts measure vLLM, Ollama and llama.cpp so each runtime uses its correct client path.
Structured JSON output
Each run returns one JSON object with latency, throughput and device metadata for comparison.
Concurrency sweep for vLLM
The vLLM wrapper measures several concurrency levels and reports a run entry for each.
Device and power warnings
Results flag non-maximum power modes, background GPU load and thermal throttling during a run.
Jetson-specific interpretation
Guidance covers memory bandwidth limits, cache effects after upgrades and comparing quantizations.
Error handling guidance
Documented exit codes and preflight checks explain how to fix unreachable servers or missing models.
What Jetson LLM serve does
Runtime selection by Jetson family
Chooses between upstream vLLM, the NVIDIA-AI-IOT container, and NVIDIA SGLang based on the device generation and JetPack version.
Docker launch commands
Provides ready-to-adapt docker run and vllm serve commands with host networking and port 8000.
Pre-flight device checks
Verifies Jetson identity, power mode, MIG status, GPU device users, existing containers and Docker NVIDIA runtime before launch.
Memory fallback defaults
Offers per-device context length and GPU memory settings, with a rule to lower memory utilization after startup OOM.
Quantization format guidance
Recommends checkpoint formats such as NVFP4 or W4A16 per Jetson family, with GGUF options for llama.cpp and Ollama.
Safe token handling
Explains when HF_TOKEN is needed and how to limit exposure of the credential in shared container setups.
Jetson LLM benchmark vs Jetson LLM serve: questions
Is Jetson LLM benchmark or Jetson LLM serve cheaper?
Both are open source. Compare current plans on each official site.
What is the difference between Jetson LLM benchmark and Jetson LLM serve?
Jetson LLM benchmark is an open-source agent skill for ai models and llm tooling. Jetson LLM serve is an open-source agent skill for ai models and llm tooling.
