Skip to content
Tool

TrnRT

Open OpenAI-compatible LLM inference engine for AWS Trainium2 with NKI megakernels

Pricing
Open source
Platforms
API, Linux, self-hosted
Listing updated
today

Similar tools

About TrnRT

TrnRT is an open-source AI tool for developer tools. It is available on API, Linux, self-hosted and CLI.

The README describes TrnRT as an inference server that exposes an OpenAI-compatible API from a single trn2.48xlarge instance. Decode runs as fused NKI kernels, roughly one per layer half, and a radix-tree prefix cache spans device and host memory so repeated context is reused across requests.

At launch the only supported model is GLM-5.3-Flash, a 320B mixture-of-experts model with 18B active parameters and a 1M-token context. Model compilation happens offline on a separate x86_64 CPU host, so the serving box loads precompiled graphs rather than compiling them. The README reports benchmark figures from the SemiAnalysis AgentX harness for batch sizes 4, 8 and 12.

The project suits teams that already run or plan to run AWS Trainium hardware and want a self-hosted endpoint for long, agentic coding sessions. The README notes that more open models are planned but not yet available.

What it does

Fused decode kernels
Decode uses NKI megakernels that fuse attention, the layer FFN and collectives into one kernel per layer half.
Radix prefix cache
KV pages are cached in a radix tree with a host-memory tier for evicted pages and state checkpoints.
OpenAI-compatible API
Chat and completions endpoints with streaming, tool calls and reasoning output, usable from standard OpenAI clients.
Session-affine routing
Two tensor-parallel engines per box sit behind a router that keeps a session on one engine's cache.
Offline graph compilation
Graphs are built on a CPU host ahead of time, so the serving instance loads them without compiling by default.

Good for

  • Self-host an OpenAI-compatible endpoint for GLM-5.3-Flash on AWS Trainium2
  • Serve long-context coding-agent sessions with cached prefixes
  • Benchmark LLM inference cost per million tokens on Trainium hardware
  • Run tool-calling workloads against a private model endpoint

Pricing

Open source

Apache-2.0 open source; users pay for AWS Trainium instance time.

TrnRT FAQ

What hardware does TrnRT require?

The README lists AWS Trainium2 on a trn2.48xlarge instance as the supported serving target, and an x86_64 host for compilation.

Which models are supported?

GLM-5.3-Flash is the only supported model in the README. More open models are listed as coming soon.

What license is it under?

The repository is licensed under the Apache License 2.0.

Does it expose an OpenAI-compatible API?

Yes. The README says it serves chat and completions, including streaming and tool calls, and shows use with the OpenAI Python client.

Is TrnRT free?

Yes. TrnRT is open source, so you can use and self-host it for free.

What are the best alternatives to TrnRT?

The closest alternatives to TrnRT are Prime Intellect, gateway, laplacesscissors, Eden AI and Llmtee.

Where this comes from

  • Description: written from the project's README.