How to Install and Scale NVIDIA NeMo on Bare Metal

When your objective is to train a 70B or 100B+ parameter Generative AI model from scratch or align it using RLHF, developer-friendly tools often struggle to scale efficiently. At frontier scale, training jobs may span thousands of GPUs, requiring an integrated framework designed to orchestrate complex parallelism, maximize Model Flops Utilization (MFU), and prevent memory bottlenecks. That ecosystem is the NVIDIA NeMo Framework.

This tutorial provides a step-by-step guide on how to install and configure NVIDIA NeMo on a bare-metal GPU server using Docker, while breaking down the architecture, Megatron-Core integration, and strategies for handling Out-of-Memory (OOM) errors.

1. Step-by-Step Installation: Deploying NeMo on Bare Metal

For enterprise deployments, using the official NVIDIA NeMo Docker Container is recommended. It isolates the CUDA userspace and ships with preconfigured, optimized packages such as TransformerEngine and DeepEP.

Step 1: Install Host Prerequisites (Docker CE)

Ensure your host machine has the appropriate NVIDIA Data Center drivers installed and verified via nvidia-smi.

⚠️ SRE Alert: The docker.io Trap
Do not install the default OS package (docker.io), as it is often outdated and can cause cgroup compatibility issues with the latest NVIDIA runtimes. Always install the official Docker CE from Docker's upstream repository.

# Add Docker's official GPG key and repository, then install Docker CE
curl -fsSL [https://get.docker.com](https://get.docker.com) -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

Step 2: Install the NVIDIA Container Toolkit

The toolkit allows Docker to access the physical GPUs on your bare-metal server, acting as the bridge between the container and the NVIDIA drivers:

# Add the NVIDIA GPG Key and Repository
curl -fsSL [https://nvidia.github.io/libnvidia-container/gpgkey](https://nvidia.github.io/libnvidia-container/gpgkey) | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L [https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list](https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list) | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install the toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# Configure Docker runtime to use NVIDIA natively and restart
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 3: Pull the NeMo Container

Download the official NeMo Framework container from the NVIDIA NGC registry:

docker pull nvcr.io/nvidia/nemo:24.07

Step 4: Run NeMo with GPU Support

⚠️ Warning: Persist Your Checkpoints on Fast Storage (NVMe)
Because the --rm flag destroys the container on exit, data stored only inside the container is lost. Always bind-mount a host directory for your checkpoints and datasets.

SRE Best Practice: Never use relative paths like $(pwd) in production. 70B+ model checkpoints exceed 100 GB per save. Bind-mount to an absolute path located on a dedicated, high-speed NVMe RAID array (e.g., /mnt/nvme_storage) to prevent I/O bottlenecks that stall GPU computation during checkpointing.

docker run --gpus all -it --rm \
  --ipc=host \
  --network=host \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -v /mnt/nvme_storage/checkpoints:/workspace/checkpoints \
  -v /mnt/nvme_storage/datasets:/workspace/datasets \
  nvcr.io/nvidia/nemo:24.07

Once inside the container, your bare-metal environment is ready for unthrottled NeMo multi-GPU orchestration.

2. Demystifying NVIDIA NeMo: The Architecture

NVIDIA NeMo is an integrated framework ecosystem for building, customizing, training, evaluating, and aligning large-scale AI models. Built on PyTorch, NeMo features a modular and decoupled architecture—including components such as NeMo Curator, NeMo Megatron-Bridge, NeMo AutoModel, and NeMo RL—that addresses every stage of model development:

• NVIDIA NeMo (The Framework): Brings together model-specific workflows, data preparation components, distributed training integrations, checkpointing, and post-training capabilities.
• Megatron-Core (The Distributed Training Engine): Provides optimized, composable PyTorch building blocks for large-scale transformer training, implementing tensor, pipeline, and context parallelism alongside optimized communication and compute kernels.

NeMo vs. Hugging Face vs. DeepSpeed

Hugging Face provides a broad ecosystem for model development, fine-tuning, and accessibility, while DeepSpeed offers distributed training optimizations including ZeRO-based memory sharding. NVIDIA NeMo delivers an integrated framework ecosystem with deep Megatron-Core integration tailored for large-scale training on NVIDIA GPU infrastructure. Selecting the right stack depends on model architecture, hardware topology, parallelism needs, and engineering priorities.

3. Megatron-Core and 3D Parallelism (How to Scale)

Full-scale training of a 70B-parameter model requires distributed GPU resources because model parameters, gradients, optimizer states, and activations demand far more memory than model weights alone. NeMo utilizes Megatron-Core to orchestrate a 3D Parallelism strategy:

• Tensor Parallelism (TP): Splits individual weight matrices and selected transformer computations across multiple GPUs. Because it involves frequent communication collectives, TP is typically placed within a high-bandwidth, low-latency GPU interconnect domain, such as an NVLink-connected node.
• Pipeline Parallelism (PP): Divides transformer layers into sequential stages distributed across GPUs. Stages can reside within one server or across multiple nodes, with activations transferred between stages over the GPU interconnect or scale-out network.
• Data Parallelism (DP), Distributed Optimizer & FSDP: Megatron-Core supports multiple data-parallel strategies depending on the training architecture. Its Distributed Optimizer shards optimizer states across data-parallel ranks, reducing optimizer memory consumption while preserving the parameter layout required by TP and PP. Sequence Parallelism (SP) complements TP by distributing selected activations across participating GPUs, while Megatron-FSDP shards parameters, gradients, and optimizer states across data-parallel ranks for workloads requiring deeper model-state sharding.

4. SRE Strategies for Eradicating OOM Errors

Even with 80 GB or 141 GB of VRAM per GPU, improper configurations will trigger Out-of-Memory (OOM) crashes. SREs rely on three primary techniques within NeMo:

• Context Parallelism (CP): Distributes sequence-length dimensions and associated activations across multiple GPUs, lowering the per-GPU memory footprint during long-context training using communication strategies such as ring-based exchanges.
• Activation Checkpointing (Recomputation): Instead of retaining every intermediate activation from the forward pass, the training configuration selectively stores checkpoints and recomputes other activations during backpropagation—trading additional compute for substantial activation memory savings.
• Micro-Batch Tuning: Reducing micro_batch_size per GPU lowers activation memory while maintaining global batch size via gradient accumulation (though higher gradient accumulation steps can impact optimizer update frequency, pipeline utilization, and communication overlap).

5. RLHF Fine-Tuning: The Version-Sensitive Memory Trap

PPO-based RLHF workflows can involve four distinct model roles: a trainable Actor, a frozen Reference model, a trainable Critic, and a Reward model—placing heavy demands on GPU memory depending on residency and execution schedules.

NeMo's TensorRT-LLM Integration

In supported NeMo-Aligner PPO workflows, TensorRT-LLM accelerates the generation phase of RLHF. The trainer.ppo.trt_llm.unload_engine_train setting controls whether the TensorRT-LLM engine is unloaded during the training phase.

Setting this parameter to True frees GPU memory during training but introduces engine loading and offloading overhead that can significantly impact end-to-end throughput. NVIDIA's documented guidance recommends keeping this option disabled (False) by default and enabling it only after benchmarking memory and latency trade-offs for your specific workload and release.

6. Virtualization and Bare-Metal Performance Considerations

Virtualized GPU environments can introduce overhead depending on the hypervisor, GPU passthrough or vGPU configuration, CPU scheduling, storage path, and network topology. For distributed training, collective operations such as AllReduce are highly sensitive to network latency, bandwidth, and congestion consistency.

Bare-Metal Infrastructure for Large-Scale AI Training

Bare-metal deployment provides direct access to physical hardware and removes the host hypervisor layer. When configured with compatible GPUs, NVLink bridges, PCIe peer-to-peer support, and RDMA-capable NICs, single-tenant dedicated servers enable optimized GPU communication paths without noisy-neighbor contention.

💡 Important Note on Physical Isolation and Security:
Physical isolation does not automatically provide encryption of data in use. Technologies like Intel TDX and AMD SEV-SNP are CPU-level confidential VM technologies, distinct from NVIDIA GPU Confidential Computing. Organizations requiring confidential computing must verify compatible CPU and GPU security capabilities across the entire deployment stack.

Conclusion

NVIDIA NeMo provides an integrated ecosystem for developing, training, and aligning large-scale generative AI models. By combining NeMo's model workflows with Megatron-Core's distributed training capabilities, engineering teams can scale transformer workloads across multi-GPU and multi-node clusters.

However, production efficiency depends on disciplined GPU memory management, parallelism tuning, high-throughput NVMe checkpoint storage, and low-latency interconnects—making dedicated bare-metal infrastructure an ideal foundation for demanding LLM pretraining and post-training pipelines.

AI Cluster & NVIDIA NeMo FAQ

What is the difference between NVIDIA NeMo and Hugging Face?

Hugging Face provides a broad ecosystem for model development, accessibility, fine-tuning, and distributed training, while DeepSpeed offers distributed training optimizations including ZeRO-based memory sharding. NVIDIA NeMo provides an integrated framework ecosystem with deep Megatron-Core integration and optimizations tailored for large-scale training on NVIDIA GPU infrastructure.

Why do I get OOM errors during RLHF fine-tuning, and should I offload the engine?

RLHF workflows involve multiple models (Actor, Critic, Reward, Reference) that consume substantial VRAM. In supported NeMo-Aligner PPO workflows, TensorRT-LLM engine offloading (trainer.ppo.trt_llm.unload_engine_train=True) can free GPU memory during training, but introduces engine reload overhead. NVIDIA recommends leaving it disabled by default and benchmarking it only when memory pressure requires an alternative configuration.

Does Context Parallelism (CP) change attention complexity to linear?

No. Context Parallelism distributes sequence-length dimensions and associated activations across multiple GPUs, reducing the per-GPU memory footprint during long-context training, but it does not eliminate the underlying computational complexity of attention.

Why is Bare Metal infrastructure recommended over Public Cloud for NVIDIA NeMo?

Bare-metal servers provide direct access to allocated physical hardware and remove the host hypervisor layer. When the GPU, NIC, PCIe topology, drivers, and network fabric are compatible, bare metal simplifies deploying technologies like GPUDirect RDMA and GPUDirect Storage (GDS) without virtualization overhead.

Does Megatron-Core support FSDP for memory optimization?

Yes. Megatron-Core supports multiple data-parallel strategies, including its Distributed Optimizer (which shards optimizer states across data-parallel ranks) and Megatron-FSDP (which shards parameters, gradients, and optimizer states). The optimal strategy depends on model architecture, parallelism configuration, and software release.