Skip to content

Jetson LLM benchmark vs Jetson LLM serve

Jetson LLM benchmark is an open-source agent skill for ai models and llm tooling. Jetson LLM serve is an open-source agent skill for ai models and llm tooling.

  • Jetson LLM benchmark is best for measure time to first token and throughput for a model served by vllm on a jetson.
  • Jetson LLM serve is best for serve a hugging face llm on my jetson orin as an openai-compatible endpoint.
What it is
Jetson LLM benchmark is an open-source agent skill for ai models and llm tooling.
Jetson LLM serve is an open-source agent skill for ai models and llm tooling.
Best for
Measure time to first token and throughput for a model served by vLLM on a Jetson
Serve a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpoint
Kind
Skill
Skill
Pricing
Open source
Open source
Platforms
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, Windsurf, Goose
GitHub stars
4K
4K
Last commit
Nov 2026
Nov 2026
License
Apache-2.0
Apache-2.0
Open source
Good for
Measure time to first token and throughput for a model served by vLLM on a Jetson; Compare a model's speed before and after changing power mode or runtime flags; Benchmark a GGUF model with llama.cpp on a Jetson Orin or Thor
Serve a Hugging Face LLM on my Jetson Orin as an OpenAI-compatible endpoint; Start a vision-language model server on Jetson Thor for my app; Launch an SGLang server on Thor for tool-use workflows

What Jetson LLM benchmark does

  • Runtime-specific benchmark wrappers

    Separate scripts measure vLLM, Ollama and llama.cpp so each runtime uses its correct client path.

  • Structured JSON output

    Each run returns one JSON object with latency, throughput and device metadata for comparison.

  • Concurrency sweep for vLLM

    The vLLM wrapper measures several concurrency levels and reports a run entry for each.

  • Device and power warnings

    Results flag non-maximum power modes, background GPU load and thermal throttling during a run.

  • Jetson-specific interpretation

    Guidance covers memory bandwidth limits, cache effects after upgrades and comparing quantizations.

  • Error handling guidance

    Documented exit codes and preflight checks explain how to fix unreachable servers or missing models.

What Jetson LLM serve does

  • Runtime selection by Jetson family

    Chooses between upstream vLLM, the NVIDIA-AI-IOT container, and NVIDIA SGLang based on the device generation and JetPack version.

  • Docker launch commands

    Provides ready-to-adapt docker run and vllm serve commands with host networking and port 8000.

  • Pre-flight device checks

    Verifies Jetson identity, power mode, MIG status, GPU device users, existing containers and Docker NVIDIA runtime before launch.

  • Memory fallback defaults

    Offers per-device context length and GPU memory settings, with a rule to lower memory utilization after startup OOM.

  • Quantization format guidance

    Recommends checkpoint formats such as NVFP4 or W4A16 per Jetson family, with GGUF options for llama.cpp and Ollama.

  • Safe token handling

    Explains when HF_TOKEN is needed and how to limit exposure of the credential in shared container setups.

Jetson LLM benchmark vs Jetson LLM serve: questions

Is Jetson LLM benchmark or Jetson LLM serve cheaper?

Both are open source. Compare current plans on each official site.

What is the difference between Jetson LLM benchmark and Jetson LLM serve?

Jetson LLM benchmark is an open-source agent skill for ai models and llm tooling. Jetson LLM serve is an open-source agent skill for ai models and llm tooling.

Keep comparing