NVIDIA Dynamo Guide: Scaling Distributed LLM Inference on Bare Metal

As the adoption of Large Language Models (LLMs) and complex agentic workflows accelerates, traditional monolithic inference servers hit a hard performance ceiling. Managing inference across thousands of GPUs is a demanding distributed systems problem. To solve it, NVIDIA introduced NVIDIA Dynamo.

In this SRE blueprint, we break down the communication planes, the evolution of KV cache management, topology-aware network transfers, step-by-step deployment on Kubernetes, and the observability practices required for bare-metal GPU clusters.

📌 Version & Compatibility Notice (2026)
This guide uses NVIDIA Dynamo v1.6.0 (GA October 22, 2026) as its reference deployment baseline. All installation commands and example manifests are pinned to v1.6.0 for reproducibility. For new Kubernetes deployments, use the supported nvidia.com/v1beta1 API version for DynamoGraphDeploymentRequest resources, and refer to the official NVIDIA Dynamo v1.6.0 DGDR API Reference. Verify installed operator, CRD, CUDA, NVIDIA Driver, NIXL, and inference-engine versions against official compatibility documentation before applying manifests.

1. NVIDIA Dynamo Architecture: Request, Event, and Transfer Planes

A common technical fallacy is treating Dynamo as a direct replacement for inference engines. Dynamo is an orchestration framework that sits above execution engines (such as vLLM or SGLang), transforming disparate GPU workers into a single coordinated inference system.

Dynamo coordinates these workers across three distinct communication paths alongside an underlying Service Discovery mechanism:

• Request Plane (RPC): Handles incoming traffic between the Frontend and workers (typically utilizing TCP or NATS).
• Event Plane (Telemetry & Signals): Asynchronously broadcasts KV events and worker telemetry (utilizing ZMQ or NATS) to keep the router updated on cache availability.
• KV Transfer Path (NIXL): Provides high-throughput data movement, including KV-cache transfers between Prefill and Decode workers in disaggregated serving. NIXL itself is a broader transfer abstraction supporting multiple memory and storage backends.

2. Aggregated vs. Disaggregated Serving Trade-Offs

LLM inference consists of two distinct phases: Prefill (processing the input prompt, which is typically compute-intensive) and Decode (generating output tokens, which is typically memory-bandwidth-intensive). Dynamo supports two primary serving topologies:

• Aggregated Serving: Prefill and decode execute on the same worker pool, minimizing network dependency and operating efficiently for shorter requests.
• Disaggregated Serving: Separates prefill and decode tasks onto distinct GPU pools tailored to their respective hardware bottlenecks.

💡 SRE Insight: Disaggregated Is Not Automatically Faster
Performance gains depend on prompt length, output length, concurrency, KV payload size, interconnect bandwidth, and pool sizing. If your workload consists of short prompts and short responses, the KV transfer penalty can outweigh the benefits of compute isolation.

3. NIXL Transport Backends and Topology-Aware KV Cache Transfer

In disaggregated serving, KV cache moves from Prefill to Decode workers via NIXL (NVIDIA Inference Xfer Library). NIXL dynamically selects a suitable backend based on source/destination memory types and the backends initialized on both agents:

• UCX: Supports InfiniBand, RoCE, and TCP for inter-node VRAM and DRAM transfers.
• Libfabric: Provides communication through supported OFI providers.
• GDS / GDS-MT: Enables GPU-to-storage transfers through GPUDirect Storage.
• POSIX: Supports conventional DRAM-to-file transfers.
• DOCA GPUNetIO, UCCL-P2P, Mooncake: Alternative high-performance transfer paths.

(Note: NVLink and NVSwitch provide the underlying high-bandwidth hardware fabrics; they should not be confused with NIXL's software backend names).

⚠️ Alert: Transport Selection Risk
When UCX TCP support is available and enabled, transfers may fall back to TCP if an RDMA path is unavailable. In other configurations, backend initialization or transfer setup may fail outright. SREs must verify the active backend and transport in runtime logs rather than assuming automatic fallback.

Topology-Aware KV Transfer

In multi-node deployments, selecting a decode worker based solely on capacity can introduce severe network latency. By publishing topology metadata for domains such as host, rack, or availability zone, operators can constrain or bias decode-worker selection toward a preferred KV-transfer domain.

In Kubernetes deployments, configure topology-aware transfer through the DGD's spec.experimental.kvTransferPolicy:

• Define either labelKey or clusterTopologyName, not both.
• Specify the corresponding domain (e.g., rack or zone).
• If using preferred enforcement, preferredWeight is required (it acts as a strength preference, not a probability).
• In required mode, requests fail if no eligible decode worker exists in the selected domain.

While this reduces unnecessary network hops, actual GPU-NIC affinity and RDMA performance still depend on physical topology, device configuration, and the active NIXL transport.

4. KV-Aware Routing & The Evolution of Cache Management

Dynamo’s Smart Router performs Cache Locality + Load-Aware Placement. Through the Event Plane, workers actively publish KV block lifecycle events, allowing the router to evaluate prefix overlap and active worker load. (If KV events are disabled or event delivery fails, the router falls back to predicting cache states based on past routing decisions).

⚠️ Important: KVBM Deprecation & Transition
NVIDIA's original KV Block Manager (KVBM) introduced a multi-tier caching hierarchy (G1 GPU, G2 Host, G3 Disk, G4 Object Storage). However, KVBM was deprecated in Dynamo v1.5.0, with removal targeted for v1.7.0.

For new deployments, use your selected inference engine's supported native KV-offloading implementation (such as vLLM's CPU offloading) for host or storage tiering. Available tiers vary by backend—do not assume every engine offers an identical GPU-to-CPU-to-disk hierarchy. Cross-node KV cache sharing is currently being addressed by the early-stage KV Cache Runner (KVCR) project.

5. Agentic Inference, Session Identity & Affinity

Agentic workflows frequently append tool outputs and reasoning context across multiple turns. When previous prompt prefixes cannot be reused effectively, repeated context processing increases prefill compute and KV-cache pressure. Dynamo tracks these trajectories using headers including X-Dynamo-Session-ID, X-Dynamo-Session-Final, and X-Dynamo-Parent-Session-ID (for parent-child agent lineage).

Identity vs. Lifecycle and Session Affinity

Session affinity is a routing decision, not a cache-persistence guarantee. A session can remain pinned to a worker while its KV blocks remain subject to the backend engine's cache capacity and eviction policies. Passing a session ID does not automatically preserve KV cache indefinitely or enable sticky routing—you must configure explicit frontend routing policies (e.g., via --router-session-affinity-ttl-secs) in hard or soft binding modes.

The X-Dynamo-Session-Final header signals that an agent trajectory has ended so lifecycle-aware consumers can release associated session state early, though KV block eviction remains governed by configured cache policies.

6. NVIDIA Dynamo Kubernetes Architecture and Orchestration Components

Deploying Dynamo on Kubernetes relies on coordinated orchestration across several components:

• DGDR (DynamoGraphDeploymentRequest): Declares deployment intent (model, hardware constraints, and SLA targets).
• AIConfigurator / AISimulate: Provides hardware metadata, candidate generation, performance estimation, and configuration ranking.
• Profiler: Evaluates candidate configurations to formulate the final deployment plan.
• Dynamo Operator: Reconciles the final DynamoGraphDeployment (DGD) lifecycle within Kubernetes.
• Grove Operator: Coordinates multi-node workload orchestration and gang scheduling.
• KAI Scheduler: Handles strict GPU-aware scheduling and resource allocation.

7. Deploy NVIDIA Dynamo v1.6.0 on Kubernetes

Follow this SRE deployment workflow using the pinned v1.6.0 Dynamo platform and the v1beta1 API for an 8-GPU single-node baseline.

Step 1: Cluster Preparation & Prerequisites

Ensure your bare-metal cluster has the NVIDIA GPU Operator installed:

helm repo add nvidia [https://helm.ngc.nvidia.com/nvidia](https://helm.ngc.nvidia.com/nvidia) --force-update
helm repo update nvidia

# Note: If NVIDIA drivers are already managed by your K8s provider, 
# append `--set driver.enabled=false` to avoid conflicts.
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --wait \
  --timeout=10m

Verify your GPU nodes report allocatable GPU resources:

kubectl get nodes -o custom-columns="NAME:.metadata.name,GPUs:.status.allocatable.nvidia\.com/gpu"

Step 2: Install the Dynamo Platform via Helm

Install the pinned platform chart directly to avoid version drift:

export NAMESPACE="dynamo-system"
export DYNAMO_VERSION="1.6.0"

helm upgrade --install dynamo-platform \
  "[https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$](https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-$){DYNAMO_VERSION}.tgz" \
  --namespace "$NAMESPACE" \
  --create-namespace \
  --wait \
  --timeout=10m

kubectl get pods --namespace "$NAMESPACE"

Step 3: Create Credentials (NGC & Hugging Face)

⚠️ SRE Warning: Provide Image Pull Secrets Properly
Verify both the profiling Job and the generated inference Pods. Confirm that each uses a ServiceAccount or Pod template with the required imagePullSecrets reference—creating the Secret in the namespace alone does not guarantee every generated workload can pull private NGC images.

# 1. NGC Image Pull Secret
kubectl create secret docker-registry nvcr-imagepullsecret \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="<YOUR_NGC_API_KEY>" \
  -n "$NAMESPACE"

# 2. Hugging Face Token for Model Weights (Optional for public models)
kubectl create secret generic hf-token-secret \
  --namespace "$NAMESPACE" \
  --from-literal=HF_TOKEN="<YOUR_HF_TOKEN>" \
  --dry-run=client -o yaml | kubectl apply -f -

Step 4: Deploy the Inference Graph (DGDR)

Submit a DynamoGraphDeploymentRequest (DGDR). Setting autoApply: false is recommended for production so you can manually review the generated configuration before rollout:

# Save this as dynamo-deploy.yaml
apiVersion: [nvidia.com/v1beta1](https://nvidia.com/v1beta1)
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen2-5-inference
  namespace: dynamo-system
spec:
  model: Qwen/Qwen2.5-7B-Instruct
  backend: vllm
  
  image: nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.6.0
  
  # 'rapid' uses AIC-backed simulation. Use 'thorough' for real GPU benchmarking.
  searchStrategy: rapid 
  
  # Illustrative values. Set realistic targets based on expected traffic.
  workload:
    isl: 3000
    osl: 150
    concurrency: 16
    
  hardware:
    gpuSku: h100_sxm
    totalGpus: 8
    numGpusPerNode: 8
    
  # Optimization targets for candidate selection (not guaranteed performance)
  sla:
    ttft: 200.0   
    itl: 20.0
    optimizationType: latency
    
  autoApply: false # Manual review required

kubectl apply -f dynamo-deploy.yaml

Step 5: Review the Generated DGD & Deploy

Once profiling reaches the Ready phase, extract, validate, and apply the generated configuration:

# Watch the status until it hits Ready phase
kubectl get dgdr qwen2-5-inference -n "$NAMESPACE" -w

# Extract the selected configuration
kubectl get dgdr qwen2-5-inference \
  -n "$NAMESPACE" \
  -o jsonpath='{.status.profilingResults.selectedConfig}' \
  > generated-dgd.yaml

# Inspect and validate the generated resource against the cluster API
cat generated-dgd.yaml
kubectl apply --dry-run=server -f generated-dgd.yaml -n "$NAMESPACE"

# Apply after reviewing the generated configuration
kubectl apply -f generated-dgd.yaml -n "$NAMESPACE"

# Verify the inference service components are running
kubectl get dgd -n "$NAMESPACE"
kubectl get pods -n "$NAMESPACE" -o wide
kubectl get svc -n "$NAMESPACE"

Step 6: Verify Inference & Test the Endpoint

Retrieve the dynamically generated frontend service name and send a test completion request:

# Retrieve the frontend service name
DGD_NAME=$(kubectl get dgdr/qwen2-5-inference \
  --namespace "$NAMESPACE" \
  -o jsonpath='{.status.dgdName}')
FRONTEND_SERVICE="${DGD_NAME}-frontend"

# Start port-forward
kubectl port-forward \
  "service/${FRONTEND_SERVICE}" \
  8000:8000 \
  --namespace "$NAMESPACE" \
  >/tmp/dynamo-port-forward.log 2>&1 &
PORT_FORWARD_PID=$!

# Wait for the frontend health endpoint
for i in $(seq 1 30); do
  if curl --silent --fail \
    http://localhost:8000/health >/dev/null; then
    echo "Dynamo frontend is ready."
    break
  fi
  sleep 2
done

# Send a test chat completion request
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Explain Kubernetes gang scheduling."}],
    "max_tokens": 150
  }'

Extending the Deployment to Multiple Nodes

For multi-node distributed inference, your Kubernetes cluster must support gang scheduling—typically via the Grove Operator paired with the KAI Scheduler (or supported alternatives such as LWS + Volcano). Without gang scheduling, multi-node inference graphs risk deadlocking during pod placement.

8. Operational Blueprint: Benchmarking, Planning & Recovery

• Benchmarking Before Production: Compare aggregated vs. disaggregated topologies across concurrency levels, measuring TTFT (Time to First Token) and ITL (Inter-Token Latency) strictly at P50, P95, and P99 percentiles alongside NIXL KV-transfer bandwidth and tokens/sec/GPU.
• Capacity Planning: Size VRAM for model weights plus peak concurrent KV caches, determine optimal Prefill-to-Decode GPU ratios from input/output token distributions, and provision sufficient east-west network bandwidth for NIXL transfers.
• Failure Recovery Testing: Simulate prefill/decode worker crashes, RDMA link degradation during NIXL transfers, and router restarts to validate session-affinity state recovery under load.

9. NVIDIA Dynamo Observability and SRE Troubleshooting

Combine Prometheus, Grafana, and NVIDIA DCGM Exporter to correlate Dynamo metrics, NIXL telemetry, and GPU hardware counters:

• Frontend & Router: Request rate, failure rate, TTFT, routing overhead, and cache-aware routing efficiency.
• Prefill & Decode: Prompt processing latency, ITL, output tokens/sec, and active sequence count.
• KV Cache & NIXL: Cache utilization, eviction pressure, prefix-cache hit rates, and transfer duration.
• Kubernetes Health: Pod restarts, worker readiness, Profiling Job duration, scheduling failures, and node pressure events.

SRE Troubleshooting: Exit Code 137 vs. Permission Errors

• Container Termination (Exit Code 137): Indicates process termination via SIGKILL—commonly triggered by a host/cgroup OOMKilled event, though the exact termination reason should be verified in Kubernetes pod events. Investigate GPU memory exhaustion separately via CUDA errors and DCGM telemetry.
• Filesystem Permission Errors (EACCES / errno 13): Indicates a permission-denied error when accessing a model cache, mounted volume, or configuration file (such as mismatched UID/GID ownership). It is distinct from container exit code 13.

10. Bare Metal vs. Public Cloud: The Infrastructure Reality

Bare-metal dedicated servers provide granular control over GPU placement, PCIe topology, NUMA locality, NIC selection, and low-level network tuning while eliminating hypervisor and noisy-neighbor compute contention. Because public cloud platforms can also offer RDMA-enabled instances, the practical advantage comes down to physical topology transparency, isolation guarantees, and operational control over the underlying hardware fabric.

Conclusion

NVIDIA Dynamo bridges the gap between individual LLM inference engines and datacenter-scale distributed orchestration. Mastering its Request, Event, and Transfer planes—while validating NIXL RDMA paths, transitioning off deprecated KVBM patterns, enforcing P99 observability, and deploying on deterministic bare-metal infrastructure—gives SRE teams a reliable foundation for production AI inference at scale.

NVIDIA Dynamo & Distributed Inference FAQ

What is the difference between NVIDIA Dynamo and vLLM?

vLLM is an inference execution engine that generates tokens on GPUs. NVIDIA Dynamo is the distributed orchestration framework sitting above multiple vLLM or SGLang instances, managing worker discovery, KV-aware routing, NIXL cache transfers, and workload-aware scaling via its Request, Event, and Transfer paths.

Does NIXL always use RDMA for KV cache transfers?

No. NIXL dynamically selects a backend based on source/destination memory types and initialized backends. When UCX TCP support is available and enabled, transfers may use TCP if an RDMA path is unavailable, while other configurations may fail initialization. Always verify active transports in runtime logs.

Does passing a Session ID automatically preserve my agent's KV cache?

No. X-Dynamo-Session-ID provides trajectory identity tracking and enables configured session-affinity routing (such as explicit frontend TTLs via --router-session-affinity-ttl-secs), but cache retention remains governed by backend engine capacity and eviction policies. X-Dynamo-Session-Final signals trajectory completion so state can be released early.

Why is KVBM not recommended for new deployments?

The KV Block Manager (KVBM) was deprecated in Dynamo v1.5.0 and is targeted for removal in v1.7.0. New deployments should rely on the chosen inference engine's native KV-offloading features (such as vLLM CPU offloading) for host/storage tiering, while cross-node KV sharing is being developed under the KV Cache Runner (KVCR) project.

How do you distinguish Exit Code 137 from EACCES during worker initialization?

Exit Code 137 means the container process was killed by SIGKILL (frequently due to a Kubernetes OOMKilled event, which should be confirmed in pod status details). EACCES (errno 13) is a POSIX permission-denied error when reading or writing mounted volumes, model caches, or config files, and is unrelated to process exit code 13.