# TrnRT

> TrnRT is an open-source AI tool for developer tools. Open OpenAI-compatible LLM inference engine for AWS Trainium2 with NKI megakernels. It is available on API, Linux, self-hosted and CLI.

- **Kind:** Tool
- **Pricing:** Open source
- **Platforms:** API, Linux, self-hosted, CLI
- **Repository:** https://github.com/runanywhereai/trnrt
- **Categories:** [developer tools](https://ai-powered.com/categories/developer-tools)
- **Listing updated:** 2026-11-18

## About

The README describes TrnRT as an inference server that exposes an OpenAI-compatible API from a single trn2.48xlarge instance. Decode runs as fused NKI kernels, roughly one per layer half, and a radix-tree prefix cache spans device and host memory so repeated context is reused across requests.

At launch the only supported model is GLM-5.3-Flash, a 320B mixture-of-experts model with 18B active parameters and a 1M-token context. Model compilation happens offline on a separate x86_64 CPU host, so the serving box loads precompiled graphs rather than compiling them. The README reports benchmark figures from the SemiAnalysis AgentX harness for batch sizes 4, 8 and 12.

The project suits teams that already run or plan to run AWS Trainium hardware and want a self-hosted endpoint for long, agentic coding sessions. The README notes that more open models are planned but not yet available.

## Features

- **Fused decode kernels:** Decode uses NKI megakernels that fuse attention, the layer FFN and collectives into one kernel per layer half.
- **Radix prefix cache:** KV pages are cached in a radix tree with a host-memory tier for evicted pages and state checkpoints.
- **OpenAI-compatible API:** Chat and completions endpoints with streaming, tool calls and reasoning output, usable from standard OpenAI clients.
- **Session-affine routing:** Two tensor-parallel engines per box sit behind a router that keeps a session on one engine's cache.
- **Offline graph compilation:** Graphs are built on a CPU host ahead of time, so the serving instance loads them without compiling by default.

## Good for

- Self-host an OpenAI-compatible endpoint for GLM-5.3-Flash on AWS Trainium2
- Serve long-context coding-agent sessions with cached prefixes
- Benchmark LLM inference cost per million tokens on Trainium hardware
- Run tool-calling workloads against a private model endpoint

## FAQ

### What hardware does TrnRT require?

The README lists AWS Trainium2 on a trn2.48xlarge instance as the supported serving target, and an x86_64 host for compilation.

### Which models are supported?

GLM-5.3-Flash is the only supported model in the README. More open models are listed as coming soon.

### What license is it under?

The repository is licensed under the Apache License 2.0.

### Does it expose an OpenAI-compatible API?

Yes. The README says it serves chat and completions, including streaming and tool calls, and shows use with the OpenAI Python client.

### Is TrnRT free?

Yes. TrnRT is open source, so you can use and self-host it for free.

### What are the best alternatives to TrnRT?

The closest alternatives to TrnRT are Prime Intellect, gateway, laplacesscissors, Eden AI and Llmtee.

## Where this comes from

- Description: written from the project's README.

## Alternatives

- [Prime Intellect](https://ai-powered.com/tools/prime-intellect): Prime Intellect is an AI tool for developer tools.
- [gateway](https://ai-powered.com/tools/gateway-10): gateway is an open-source AI tool for developer tools.
- [laplacesscissors](https://ai-powered.com/tools/laplacesscissors): laplacesscissors is an open-source AI tool for developer tools.
- [Eden AI](https://ai-powered.com/tools/eden_ai): Eden AI is a paid AI tool for developer tools.
- [Llmtee](https://ai-powered.com/tools/llmtee): Llmtee is an open-source AI tool for developer tools.
- [codesnippets](https://ai-powered.com/tools/codesnippets): codesnippets is a freemium AI tool for generative code.

---
Source: https://ai-powered.com/tools/trnrt
