NVIDIA ships TensorRT-LLM as one container image, and a single trtllm-serve command turns any supported Hugging Face model into an OpenAI-compatible API. The old workflow of building a TensorRT engine per model, GPU and batch size is gone. The PyTorch backend loads the checkpoint directly. You install TensorRT-LLM by pulling the image, and you serve by pointing it at a model ID.
This guide covers the TensorRT-LLM Docker install, serving a model with trtllm-serve, calling the OpenAI-compatible endpoints with curl and the Python SDK, a tuned YAML config that saves almost 4 GB of VRAM, and a trtllm-bench throughput run. It also lists the errors the current release throws at anyone following an older tutorial. Every command inside the container ran on a single GeForce RTX 5090 with the 595 driver, using TensorRT-LLM 1.3.0rc29 and Qwen3-8B-FP8, in October 2026.
What you need before installing TensorRT-LLM
The container brings its own CUDA, PyTorch and Python. The host needs three things: an NVIDIA GPU with a recent driver, Docker, and the NVIDIA Container Toolkit so Docker can hand the GPU to the container. On a fresh Ubuntu host, the NVIDIA driver and CUDA guide installs all three. It also runs nvidia-ctk runtime configure for Docker. Already have Docker CE from a Docker Engine install? Then add only the driver and toolkit, and skip the docker.io package so the two Docker builds do not collide. Still run sudo nvidia-ctk runtime configure --runtime=docker and restart Docker, or --gpus all fails.
| Item | Tested with | Notes |
|---|---|---|
| GPU | GeForce RTX 5090, 32 GB | Blackwell SM120. GeForce is not on the official NVIDIA support list. FP8 and NVFP4 kernels exist for SM120, and both worked. |
| Host driver | 595.71.05 | The image is built against CUDA 13.4, and nvidia-smi on this driver reports CUDA 13.4 support. Everything ran on it. |
| OS | Ubuntu 24.04 inside the image | The host OS does not matter much. Any distro with Docker and the NVIDIA Container Toolkit works, because everything else lives in the image. |
| Disk | 200 GB | The image is about 17 GB compressed. Add the model size on top (Qwen3-8B-FP8 is 9.5 GB). |
| System RAM | Plenty (a large shared host) | Plan on 32 GB or more. Weights are staged through host memory while loading. |
For model sizing, the weights need to fit with room left for KV cache. An 8B model in FP8 used 8.84 GiB for weights on our card, and the remaining memory went to KV cache. The GPU buying guide for LLMs has the VRAM math for bigger models.
Install TensorRT-LLM from the Docker image
Release images live on the NVIDIA registry at nvcr.io/nvidia/tensorrt-llm/release, tagged with the release version minus the leading v. NVIDIA publishes release candidates every week or two. The last stable tag is often months old, so detect the tag instead of copying one from a blog post. This picks the newest published release, release candidates included:
TRTLLM_TAG=$(curl -s "https://api.github.com/repos/NVIDIA/TensorRT-LLM/releases?per_page=1" | grep -m1 '"tag_name"' | cut -d '"' -f 4 | sed 's/^v//')
echo "$TRTLLM_TAG"GitHub can list a release before the matching image reaches the registry. If the pull fails with a missing tag, check the NGC tags page and use the newest tag listed there. On the day we tested, the command returned the newest release candidate:
1.3.0rc29If you would rather stay on the last stable release, query /releases/latest instead. That endpoint skips release candidates. It returned an April build from the 1.2 line, which still carries the TensorRT engine paths the 1.3 line removed. The commands in this guide were tested on the 1.3 release candidate:
TRTLLM_TAG=$(curl -s https://api.github.com/repos/NVIDIA/TensorRT-LLM/releases/latest | grep -m1 '"tag_name"' | cut -d '"' -f 4 | sed 's/^v//')Pull the image. It is large, so expect a few minutes on a fast link:
docker pull nvcr.io/nvidia/tensorrt-llm/release:${TRTLLM_TAG}Start the container
Create a working directory on the host for config files, and keep the Hugging Face cache on the host too. Without that mount, every new container downloads the model again:
mkdir -p ~/trtllm ~/.cache/huggingfaceNow start it. The --ipc=host and --ulimit flags come from the NVIDIA install guide, and the container docs name shared IPC as the fix for Bus error (core dumped) crashes in the workers. Port 8000 is where trtllm-serve listens.
docker run -it --name trtllm --ipc=host --gpus all \
--ulimit memlock=-1 --ulimit stack=67108864 \
-p 8000:8000 \
-v ~/trtllm:/workspace \
-v ~/.cache/huggingface:/root/.cache/huggingface \
nvcr.io/nvidia/tensorrt-llm/release:${TRTLLM_TAG}You land in a root shell inside the container. Confirm the GPU is visible and check what the image ships:
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader
python3 -c "import tensorrt_llm,torch;print('tensorrt_llm',tensorrt_llm.__version__,'torch',torch.__version__,torch.version.cuda)"The card, the host driver and the in-image CUDA build all line up:
NVIDIA GeForce RTX 5090, 595.71.05, 32607 MiB
[TensorRT-LLM] TensorRT LLM version: 1.3.0rc29
tensorrt_llm 1.3.0rc29 torch 2.14.0a0+4fdf77b940.nv26.08 13.4The image also ships nvidia-modelopt for quantization and flashinfer-python for attention kernels, so nothing else needs installing to serve a model.
Serve a model with trtllm-serve
One command. Pass the Hugging Face model ID and bind to all interfaces, because the default host is localhost and the published Docker port would get nothing:
trtllm-serve nvidia/Qwen3-8B-FP8 --host 0.0.0.0 --port 8000nvidia/Qwen3-8B-FP8 is a ModelOpt FP8 checkpoint NVIDIA publishes on Hugging Face, ungated, so no token is needed. Any model in the supported models list works the same way, including plain BF16 repos.
The first start is slow. Ours took 308 seconds from the command to a healthy endpoint. The download was only 14 seconds of that. About two minutes passed before the first log line (Python import and start-up on a cold container), and the log covers the rest:
[10/12/2026-09:16:16] [TRT-LLM] [I] [llmapi] Downloading model nvidia/Qwen3-8B-FP8 from HuggingFace
[10/12/2026-09:16:30] [TRT-LLM] [I] [llmapi] Finished downloading model nvidia/Qwen3-8B-FP8 from HuggingFace
[10/12/2026-09:17:24] [TRT-LLM] [I] [_torch] Memory used at after loading weights (inside torch) in memory usage profiling: 8.84 GiB
[10/12/2026-09:17:25] [TRT-LLM] [I] [_torch] max_seq_len=40960, max_num_requests=2048, max_num_tokens=8192, max_batch_size=2048
[10/12/2026-09:18:53] [TRT-LLM] [I] [_torch] [warmup][pid=8787][rank=0][purpose=memory_profiling][pass=1] sampling_module_prewarm: done in 88.4s
INFO: Application startup complete.That 88-second sampling_module_prewarm is a one-time compile. It drops to zero on the next start. Restarting trtllm-serve inside the same running container (with the tuned config below) came up in 54 seconds. That compile cache is not in the mounted Hugging Face directory, so it does not survive a new container. Reuse the container with docker start -ai trtllm instead of removing it. If you run TensorRT-LLM behind an orchestrator with readiness probes, give the first start at least six minutes before it gets killed and retried.
Leave the server running in that terminal and open a second one on the host.
Call the TensorRT-LLM OpenAI-compatible API
The server speaks the OpenAI REST dialect, so /v1/models, /v1/completions and /v1/chat/completions all work. List the model first:
curl -s localhost:8000/v1/modelsThe model ID is whatever you passed to trtllm-serve:
{"object":"list","data":[{"id":"nvidia/Qwen3-8B-FP8","object":"model","created":1791796777,"owned_by":"tensorrt_llm"}]}Send a chat request. Qwen3 thinks before it answers by default, and chat_template_kwargs passes straight through to the chat template to turn that off:
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"nvidia/Qwen3-8B-FP8","messages":[{"role":"user","content":"In one sentence, what does a Kubernetes readiness probe do?"}],"max_tokens":200,"chat_template_kwargs":{"enable_thinking":false}}'The answer comes back with OpenAI-shaped usage counters, plus a few TensorRT-LLM extras such as avg_decoded_tokens_per_iter:
{"id":"chatcmpl-7e543cad59914e7781c0c3e3645f73c6","object":"chat.completion","created":1791796778,"model":"nvidia/Qwen3-8B-FP8","choices":[{"index":0,"message":{"role":"assistant","content":"A Kubernetes readiness probe determines if a container is ready to serve traffic and is used to control when a container is considered ready to receive requests.","reasoning_content":"","reasoning":null,"tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":151645,"mm_embedding_handle":null,"disaggregated_params":null,"avg_decoded_tokens_per_iter":1.0}],"usage":{"prompt_tokens":24,"total_tokens":52,"completion_tokens":28,"prompt_tokens_details":{"cached_tokens":0}},"prompt_token_ids":null,"prompt_token_ids_b64":null}Existing OpenAI client code only needs a new base_url. The image already ships the openai SDK, so the quickest test runs inside the container. Create a small streaming client in the mounted working directory on the host:
vim ~/trtllm/client.pyPaste the script. The API key can be any string, because trtllm-serve does not check it:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
stream = client.chat.completions.create(
model="nvidia/Qwen3-8B-FP8",
messages=[{"role": "user", "content": "Give three reasons a pod can stay in Pending."}],
max_tokens=300,
stream=True,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
for chunk in stream:
delta = chunk.choices[0].delta.content if chunk.choices else None
if delta:
print(delta, end="", flush=True)
print()Open a shell in the running container:
docker exec -it trtllm bashRun the client there with time to see how quickly 300 tokens arrive:
time python3 /workspace/client.pyTokens stream in as they decode. The full 300-token answer took 2.7 seconds on the 5090 (output trimmed to the timing lines):
A pod in Kubernetes can remain in the **Pending** state for several reasons. Here are **three common reasons**:
real 0m2.737s
user 0m0.817s
sys 0m0.122sThe same endpoint works from another machine once port 8000 is open. That is how we tested it. The server has no authentication, so put it behind a reverse proxy or a private network before exposing it anywhere.
Tune trtllm-serve with a config file
The defaults are sized for a data-center card, not a desktop one. With no config, the server set max_seq_len to the full 40,960-token context. It allowed a batch of 2,048 requests and reserved 90% of free memory for KV cache. VRAM sat at 31,853 MiB of 32,607 MiB, leaving nothing for anything else on the card.
Settings go in a YAML file passed with --config. The older --extra_llm_api_options name still works as an alias, and any flag given on the command line overrides the file. Create the file in the mounted working directory on the host:
vim ~/trtllm/config.yamlThis is the exact file used for the tuned runs:
max_batch_size: 32
max_num_tokens: 8192
max_seq_len: 8192
kv_cache_config:
free_gpu_memory_fraction: 0.80
enable_block_reuse: true
cuda_graph_config:
enable_padding: true
max_batch_size: 32Each key is a direct trade. max_batch_size caps concurrent sequences. CUDA graphs are captured only up to that size, so a small value also shortens warmup. max_seq_len limits prompt plus output per request. free_gpu_memory_fraction is the share of memory left after weights that goes to KV cache. enable_block_reuse is prefix caching. It is already on by default and listed here so the file shows it, because it pays off when many requests share a system prompt.
Stop the running server with Ctrl+C and start it again with the file:
trtllm-serve nvidia/Qwen3-8B-FP8 --host 0.0.0.0 --port 8000 --config /workspace/config.yamlIt came back in 54 seconds, using 27,987 MiB instead of 31,853 MiB. Throughput did not move. At 8 and 32 requests the tuned server matched the default within 1%, and at a single request it was 4% faster (169 against 163 tokens per second). Thirty-two parallel requests never needed more than a fraction of either KV pool. On a desktop card, those 3.8 GiB are what a display, a second model or an embedding server can use.
Benchmark throughput with trtllm-bench
trtllm-bench ships in the same image and drives the engine directly, without HTTP in the way. Synthetic datasets come from its own prepare-dataset subcommand. Open a second shell in the running container:
docker exec -it trtllm bashStop the server in the first shell before benchmarking, because both want the whole GPU. Then generate 256 synthetic requests of 512 input and 256 output tokens:
trtllm-bench --model nvidia/Qwen3-8B-FP8 prepare-dataset --output /workspace/ds_512_256.txt \
token-norm-dist --num-requests 256 --input-mean 512 --input-stdev 0 --output-mean 256 --output-stdev 0Run the throughput test at 32 concurrent requests with the same config file the server uses, so the numbers describe the server you will actually run:
trtllm-bench --model nvidia/Qwen3-8B-FP8 throughput --dataset /workspace/ds_512_256.txt \
--config /workspace/config.yaml --concurrency 32 --streaming --report_json /workspace/tbench-fp8.jsonThe report prints a long settings dump, then the numbers that matter:
model_loader.total_model_loading_seconds: 2.4077
Request Throughput (req/sec): 12.9483
Total Output Throughput (tokens/sec): 3314.7649
Total Token Throughput (tokens/sec): 9944.2947
Average request latency (ms): 2468.2586
Average time-to-first-token [TTFT] (ms): 350.0681
Per User Output Speed (tps/user): 120.6902
[TPOT] P50 : 8.5885
[TTFT] P50 : 287.9128
[TTFT] P99 : 512.0309That is over 3,300 output tokens per second on one consumer card. Each user still sees about 120 tokens per second. TTFT here includes queueing, because all 32 requests arrive at once and share the prefill.
Errors and warnings from the current release
Most TensorRT-LLM tutorials online were written for the engine-build workflow. The 1.3 release candidates removed it, so copied commands now fail. These are the messages we hit, verbatim.
Error: Invalid value for ‘–backend’: ‘tensorrt’ is not ‘pytorch’
Older guides pass --backend tensorrt to serve a prebuilt engine. Release candidate 28 removed the TensorRT engine paths from trtllm-serve, trtllm-bench and trtllm-eval, so the only accepted backend is PyTorch:
Usage: trtllm-serve serve [OPTIONS] MODEL
Try 'trtllm-serve serve --help' for help.
Error: Invalid value for '--backend': 'tensorrt' is not 'pytorch'.The fix is to drop the flag and point the server at the Hugging Face checkpoint instead of an engine directory. If you depend on prebuilt engines, stay on the 1.2 line until you migrate.
Warning: Field name “schema” in “ResponseFormat” shadows an attribute in parent “OpenAIBaseModel”
Every trtllm-serve start prints this Pydantic warning, and both trtllm-serve and trtllm-bench print a torchao line about skipping cpp extensions. Both are harmless. Every run in this guide printed them and served correctly, so don’t go chasing a torch downgrade over them.
Warning: You are sending unauthenticated requests to the HF Hub
Ungated models download without a token, but anonymous requests get lower rate limits. Gated repos such as the Meta Llama checkpoints need a token as well. Accept the license on Hugging Face, then pass a read token into the container with -e HF_TOKEN=hf_xxx on the docker run line.
TensorRT-LLM numbers on an RTX 5090
These come from a streaming OpenAI client sending a 472-token prompt and asking for exactly 256 tokens (ignore_eos on), against the tuned server with Qwen3-8B-FP8. It is the same client and prompt we used in the Ollama, vLLM and llama.cpp comparison, so the numbers line up with those tables.
| Concurrent requests | Aggregate tokens/s | Per-request decode tokens/s | Median TTFT |
|---|---|---|---|
| 1 | 169 | 170 | 14 ms |
| 4 | 635 | 160 | 20 ms |
| 8 | 1,087 | 138 | 25 ms |
| 16 | 2,238 | 143 | 29 ms |
| 32 | 4,112 | 133 | 44 ms |
At 32 requests the card delivers 24 times the single-request rate, and each request stays above 130 tokens per second. Swapping the FP8 checkpoint for nvidia/Qwen3-8B-NVFP4 (the Blackwell 4-bit float format) pushed the same test to 5,340 tokens per second at 32 requests and 221 at one, for 1.5 points of GSM8K strict-match accuracy (the same benchmark we used to compare GGUF, AWQ, GPTQ and FP8). If vLLM is what you run today, the vLLM install guide uses the same OpenAI endpoints, so client code moves between the two servers unchanged.
