Jetson LLM serve vs Jetson speculative decoding
Jetson LLM serve is an open-source agent skill for ai models and llm tooling. Jetson speculative decoding is an open-source agent skill for ai models and llm tooling.
- Jetson LLM serve is best for serve a hugging face llm on my jetson orin as an openai-compatible endpoint.
- Jetson speculative decoding is best for speed up single-user token generation on a jetson agx orin running vllm.
- What it is
- Jetson LLM serve is an open-source agent skill for ai models and llm tooling.Jetson speculative decoding is an open-source agent skill for ai models and llm tooling.
- Best for
- Serve a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpointSpeed up single-user token generation on a Jetson AGX Orin running vLLM
- Kind
- SkillSkill
- Pricing
- Open sourceOpen source
- Platforms
- Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, GooseClaude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
- GitHub stars
- 4K4K
- Last commit
- Nov 2026Nov 2026
- License
- Apache-2.0Apache-2.0
- Open source
- Good for
- Serve a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpoint; Start a vision-language model server on Jetson Thor for my app; Launch an SGLang server on Thor for tool-use workflowsSpeed up single-user token generation on a Jetson AGX Orin running vLLM; Enable EAGLE-3 speculative decoding for a model on a Jetson Thor server; Benchmark whether a draft model improves decode latency before keeping it
What Jetson LLM serve does
Runtime selection by Jetson family
Chooses between upstream vLLM, the NVIDIA-AI-IOT container, and NVIDIA SGLang based on the device generation and JetPack version.
Docker launch commands
Provides ready-to-adapt docker run and vllm serve commands with host networking and port 8000.
Pre-flight device checks
Verifies Jetson identity, power mode, MIG status, GPU device users, existing containers and Docker NVIDIA runtime before launch.
Memory fallback defaults
Offers per-device context length and GPU memory settings, with a rule to lower memory utilization after startup OOM.
Quantization format guidance
Recommends checkpoint formats such as NVFP4 or W4A16 per Jetson family, with GGUF options for llama.cpp and Ollama.
Safe token handling
Explains when HF_TOKEN is needed and how to limit exposure of the credential in shared container setups.
What Jetson speculative decoding does
EAGLE-3 configuration
Shows how to add an EAGLE-3 speculative-config to a vllm serve command when a matching head is published.
Draft-model fallback
Provides a draft-model speculative-config using a small model from the same family.
Jetson token tuning
Gives starting num_speculative_tokens values for Thor and AGX Orin and an acceptance-rate threshold for raising them.
Memory headroom adjustment
Advises lowering gpu-memory-utilization by about 0.05 to fit the draft model alongside the target.
Before-and-after benchmarking
Defines concurrency-1 benchmark steps and improvement thresholds to decide whether speculation stays enabled.
Runtime and error guidance
Lists vLLM image compatibility by Jetson platform and recovery steps for rejected configs or OOM startups.
Jetson LLM serve vs Jetson speculative decoding: questions
Is Jetson LLM serve or Jetson speculative decoding cheaper?
Both are open source. Compare current plans on each official site.
What is the difference between Jetson LLM serve and Jetson speculative decoding?
Jetson LLM serve is an open-source agent skill for ai models and llm tooling. Jetson speculative decoding is an open-source agent skill for ai models and llm tooling.
