Skip to content

Jetson LLM serve vs Jetson speculative decoding

Jetson LLM serve is an open-source agent skill for ai models and llm tooling. Jetson speculative decoding is an open-source agent skill for ai models and llm tooling.

  • Jetson LLM serve is best for serve a hugging face llm on my jetson orin as an openai-compatible endpoint.
  • Jetson speculative decoding is best for speed up single-user token generation on a jetson agx orin running vllm.
What it is
Jetson LLM serve is an open-source agent skill for ai models and llm tooling.
Jetson speculative decoding is an open-source agent skill for ai models and llm tooling.
Best for
Serve a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpoint
Speed up single-user token generation on a Jetson AGX Orin running vLLM
Kind
Skill
Skill
Pricing
Open source
Open source
Platforms
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
GitHub stars
4K
4K
Last commit
Nov 2026
Nov 2026
License
Apache-2.0
Apache-2.0
Open source
Good for
Serve a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpoint; Start a vision-language model server on Jetson Thor for my app; Launch an SGLang server on Thor for tool-use workflows
Speed up single-user token generation on a Jetson AGX Orin running vLLM; Enable EAGLE-3 speculative decoding for a model on a Jetson Thor server; Benchmark whether a draft model improves decode latency before keeping it

What Jetson LLM serve does

  • Runtime selection by Jetson family

    Chooses between upstream vLLM, the NVIDIA-AI-IOT container, and NVIDIA SGLang based on the device generation and JetPack version.

  • Docker launch commands

    Provides ready-to-adapt docker run and vllm serve commands with host networking and port 8000.

  • Pre-flight device checks

    Verifies Jetson identity, power mode, MIG status, GPU device users, existing containers and Docker NVIDIA runtime before launch.

  • Memory fallback defaults

    Offers per-device context length and GPU memory settings, with a rule to lower memory utilization after startup OOM.

  • Quantization format guidance

    Recommends checkpoint formats such as NVFP4 or W4A16 per Jetson family, with GGUF options for llama.cpp and Ollama.

  • Safe token handling

    Explains when HF_TOKEN is needed and how to limit exposure of the credential in shared container setups.

What Jetson speculative decoding does

  • EAGLE-3 configuration

    Shows how to add an EAGLE-3 speculative-config to a vllm serve command when a matching head is published.

  • Draft-model fallback

    Provides a draft-model speculative-config using a small model from the same family.

  • Jetson token tuning

    Gives starting num_speculative_tokens values for Thor and AGX Orin and an acceptance-rate threshold for raising them.

  • Memory headroom adjustment

    Advises lowering gpu-memory-utilization by about 0.05 to fit the draft model alongside the target.

  • Before-and-after benchmarking

    Defines concurrency-1 benchmark steps and improvement thresholds to decide whether speculation stays enabled.

  • Runtime and error guidance

    Lists vLLM image compatibility by Jetson platform and recovery steps for rejected configs or OOM startups.

Jetson LLM serve vs Jetson speculative decoding: questions

Is Jetson LLM serve or Jetson speculative decoding cheaper?

Both are open source. Compare current plans on each official site.

What is the difference between Jetson LLM serve and Jetson speculative decoding?

Jetson LLM serve is an open-source agent skill for ai models and llm tooling. Jetson speculative decoding is an open-source agent skill for ai models and llm tooling.

Keep comparing