TensorRT-LLM Production Deployment: FP8, NVFP4, MoE & Disaggregated Serving

Originally published at Servermo

Large language model inference is no longer defined by monolithic executions. Serving frontier Mixture-of-Experts (MoE) architectures demands extreme hardware-software co-design. While general-purpose engines like vLLM (using PagedAttention) or SGLang (using RadixAttention) offer rapid deployment flexibility, NVIDIA’s TensorRT-LLM is frequently evaluated for maximizing token throughput and stabilizing tail latency at enterprise scale on NVIDIA hardware.

This blueprint provides MLOps teams and Site Reliability Engineers (SREs) with a practical architecture for deploying TensorRT-LLM. We map out the PyTorch-native backend transitions, hardware-native FP8/NVFP4 quantization, and the networking infrastructure required for Prefill/Decode (P/D) Disaggregation using NIXL KV cache connectors.

1. TensorRT-LLM Production Architecture & Backend Selection

Historically, TensorRT-LLM required a rigid, ahead-of-time (AOT) engine compilation phase (trtllm-build). As of the v1.2+ release line, TensorRT-LLM has transitioned toward a PyTorch-native runtime as its primary execution backend for standard serving, easing the legacy configuration wall while retaining optimized CUDA kernels.

⚠️ Note on Pre-Releases: Configurations referencing release candidates (such as 1.4.0rc2) or prototype features (like specific NVFP4 KV cache support) should be validated in staging environments before production rollout, as runtime behaviors may shift in stable tags.

Auxiliary-Loss-Free Dynamic Bias

When serving massive MoE models, routing imbalance can degrade throughput. TensorRT-LLM supports the auxiliary-loss-free dynamic bias routing paradigm. By adjusting expert bias vectors dynamically out-of-graph during execution, the router mitigates "router collapse" and helps distribute expert utilization across GPUs without penalizing output quality.

2. FP8 and NVFP4 Quantization

Running a massive MoE model at FP16 is often computationally unviable for high-volume endpoints. TensorRT-LLM leverages the TensorRT Model Optimizer (ModelOpt) for Post-Training Quantization (PTQ):

• Hopper (H100/H200): Utilizes the FP8 E4M3 format. While theoretical throughput gains are frequently cited, actual performance depends heavily on sequence length, batch size, and the specific workload.
• Blackwell (B200/GB200): Unlocks the NVFP4 instruction path. FP4 reduces weight memory requirements theoretically by 4x compared to FP16, allowing large MoE models to fit within smaller NVLink domains.

Always validate quantization impact against a BF16 baseline using golden-prompt accuracy gates before production deployment.

3. TP, PP, and EP Parallelism for MoE Models

Scaling MoE models requires precise Grouped-GEMM mappings across the interconnect fabric:

• Tensor Parallelism (TP): Distributes operations within individual layers across devices.
• Pipeline Parallelism (PP): Distributes model layers across sequential execution stages.
• Expert Parallelism (EP): Shards distinct subsets of MoE experts across separate GPUs. Tokens are dynamically dispatched to their assigned experts and gathered back.

Topology-Dependent Sizing

There is no universally optimal EP size. Select EP, TP, and PP based on the model's architecture, GPU topology, supported parallelism configurations, and measured communication overhead.

An eight-GPU NVSwitch node serves as a baseline for intra-node experiments, while rack-scale systems (like the GB200 NVL72) enable larger distributed configurations. Benchmark alternative configurations using your target model and production request distribution.

4. Prefill/Decode Disaggregation

For workloads with highly variable traffic spikes, Prefill/Decode (P/D) Disaggregation decouples compute-heavy prompt processing (Prefill) from memory-bandwidth-bound token generation (Decode) onto separate dedicated GPU pools.

NIXL Transport Configuration: Avoiding Backend Misconfiguration

TensorRT-LLM uses NIXL as a KV cache transfer mechanism for disaggregated configurations. The environment variable TRTLLM_NIXL_KVCACHE_BACKEND selects the underlying communication backend:

# Use UCX, the documented default
export TRTLLM_NIXL_KVCACHE_BACKEND=UCX

# Explicitly force RDMA (RoCEv2/InfiniBand) to prevent TCP fallback
export TRTLLM_NIXL_USE_RDMA=1

⚠️ The Silent TCP Trap: Do not assume that selecting UCX automatically guarantees RDMA operation. Without explicitly setting TRTLLM_NIXL_USE_RDMA=1, the network layer will silently fall back to standard TCP, destroying sub-millisecond transfer speeds. Validate the active backend using runtime logs and verify the network path with system diagnostics.

5. Diagnosing Host-Memory Exhaustion During Compilation

During ModelOpt quantization or Triton graph compilation, SREs frequently encounter crashes reporting:

LLVM ERROR: out of memory

This error often represents host-side RAM exhaustion during compilation, not a GPU VRAM limit. Heavy swapping to an NVMe drive can substantially slow compilation. To diagnose and mitigate this:

1. Identify the build stage and inspect compiler memory logs.
2. If host RAM is exhausted by concurrent compilation, restrict worker concurrency:

   export MAX_JOBS=4
   

3. If GPU VRAM is exhausted during model loading, adjust GPU memory allocation budgets or parallelism strategies.

6. Complete Deployment Workflow

The official NVIDIA container workflow requires starting the context (Prefill) and generation (Decode) workers, followed by the orchestrator.

Step 1: Launch Container with Host Networking

Crucially, specify --network=host and --ipc=host. Without host network access, internal ports cannot route KV Cache transfers across nodes via the RDMA/RoCEv2 fabric:

docker run --rm -it --ipc=host --network=host --gpus all \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -e TRTLLM_NO_USAGE_STATS=1 \
  -v /mnt/shared/models:/models \
  nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc2 /bin/bash

Step 2: Initialize Nodes and Orchestrator

Execute these processes in separate shell sessions or manage them via a process manager:

# 1. Fire up the Generation Node (Decode)
trtllm-serve --model_dir /models/decode_engine/ --type generation --port 5001

# 2. Fire up the Context Node (Prefill)
trtllm-serve --model_dir /models/prefill_engine/ --type context --port 5002

# 3. Launch the Orchestrator via configuration file
trtllm-serve disaggregated -c /models/disagg_config.yaml

7. Production Security and Compliance

Enterprise deployments must establish strict controls before exposing inference APIs:

• Telemetry Controls: The 1.3+ release line collects anonymous deployment and runtime metadata by default. Organizations handling sensitive data should disable telemetry explicitly:

  export TRTLLM_NO_USAGE_STATS=1
  

• Network Binding: Binding trtllm-serve to 0.0.0.0 exposes the service on all network interfaces. Do not expose inference endpoints publicly without an ingress reverse proxy providing TLS termination, authentication, rate limiting, and network segmentation.
• Data Sovereignty: Evaluate provider jurisdiction, data residency, and contractual terms when hosting sensitive models. Bare-metal deployments grant full control over hardware residency and physical boundaries.

8. GPU Hardware Sizing Framework

Hardware requirements cannot be established from parameter count alone. Infrastructure sizing must account for weight precision, KV cache limits, context lengths, and parallelism strategies:

• Small Dense Models: Single high-memory GPU (e.g., L40S, H200). Sizing depends on weight memory, KV cache capacity, target concurrency, and latency goals.
• Larger Dense Models: Multi-GPU node with high-speed interconnects (NVLink). Requires validating GPU memory budgets, TP requirements, and target throughput.
• Standard MoE Models: Multi-GPU NVSwitch cluster. Sizing requires validating expert placement, all-to-all communication overhead, and EP/TP configurations.
• Frontier-Scale MoE Models: Multi-node or rack-scale clusters (e.g., GB200 NVL72). Sizing requires benchmarking total memory, distributed network topology, and full EP/TP/PP distribution.

9. Benchmarking and Observability

A high-throughput result under a single synthetic benchmark does not prove universal superiority. For a meaningful comparison against alternative engines like vLLM:

• Benchmark using identical model checkpoints, numerical formats, and GPU topologies.
• Replicate realistic production input/output token distributions.
• Measure Time to First Token (TTFT), Inter-Token Latency (ITL), and aggregate tokens-per-second throughput.
• Inspect runtime logs to confirm that the active NIXL backend is executing via RDMA rather than TCP.

10. The Infrastructure Decision: Deployment Model Comparison

Bare-metal GPU infrastructure eliminates virtualization overhead, allowing MLOps teams to validate end-to-end latency and throughput against actual hardware and production workloads.

MLOps & Inference FAQ

Is TensorRT-LLM always faster than vLLM?

On NVIDIA Hopper (H100/H200) and Blackwell (B200) hardware, TensorRT-LLM is frequently evaluated for higher throughput on stable, high-volume workloads due to fused CUDA kernels and hardware-native FP8/NVFP4 quantization. However, vLLM remains superior for rapid iteration, heterogeneous hardware environments, and workloads relying on untested model architectures. Benchmark on your exact workload before selecting an engine.

How do I fix the LLVM ERROR: out of memory in TensorRT-LLM?

This error represents host-side RAM exhaustion during engine compilation or Triton graph unrolling, not a GPU VRAM limit. Do not rely on massive NVMe swap files, which cause swap thrashing and slow builds. Restrict compilation concurrency by setting export MAX_JOBS=4 (or lower) before running the build.

Why consider Bare Metal over Public Cloud for TensorRT-LLM Disaggregated Serving?

Disaggregated serving relies on KV cache connectors (like NIXL) to rapidly transfer memory between Prefill and Decode GPUs. Bare-metal servers provide unthrottled RDMA/RoCEv2 network fabrics, bypassing cloud hypervisors and vSwitches. This guarantees the sub-millisecond all-to-all communication critical for scaling massive MoE models efficiently.