How to Deploy Multi-Agent AI: Setup CrewAI on Ubuntu 24.04 Bare Metal GPU

Multi-agent AI frameworks have shifted from experimental prototypes to production-grade automation engines. But running CrewAI or AutoGen agents via cloud APIs introduces crippling costs: a single agent task can trigger 20 to 50 LLM calls due to autonomous thinking loops.

By deploying your multi-agent AI framework on a Dedicated Bare Metal GPU server, your VRAM becomes a fixed cost. Whether your agent loops 10 times or 10,000 times, your infrastructure bill remains flat.

Phase 1: VRAM & KV Cache Math

When multiple agents run concurrently, each agent consumes VRAM for model weights plus KV Cache (Key-Value Cache) for context memory:

• A 14B model (like Qwen2.5 or Llama-3.3) takes ~15GB of VRAM in FP8 precision.
• Each concurrent agent context requires roughly 1.5GB of KV cache.
• Running a 4-agent crew concurrently requires unthrottled GPU access and high PCIe throughput to prevent Out-Of-Memory (OOM) crashes and latency spikes.

Phase 2: Server Hardening & UFW Setup

Never expose your local LLM engine to the public internet. Leaving inference ports open invites GPU hijacking. Lock down your Ubuntu 24.04 firewall:

# Update Ubuntu packages
sudo apt update && sudo apt upgrade -y

# Allow only SSH, HTTP, HTTPS
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

Phase 3: Install Ollama & Pull the Model

Install Ollama to host your sovereign inference engine locally:

curl -fsSL [https://ollama.com/install.sh](https://ollama.com/install.sh) | sh

# Pull a 14B+ model for robust agent reasoning
ollama pull qwen2.5:14b

Engineering Tip: 8B models struggle with complex JSON structured outputs required for CrewAI task delegation. Use at least 14B models for worker agents and 32B/70B models for manager agents.

Phase 4: Deploying CrewAI with uv

Avoid raw pip to prevent Python environment conflicts. Use uv from Astral for ultra-fast environment isolation:

# Install UV package manager
curl -LsSf [https://astral.sh/uv/install.sh](https://astral.sh/uv/install.sh) | sh
source $HOME/.cargo/env

# Initialize project scaffold
uv tool install crewai
crewai create crew servermo_agents
cd servermo_agents

The LiteLLM Trap

CrewAI utilizes LiteLLM under the hood, which crashes if it doesn't detect an OpenAI API key—even when routing strictly to localhost. Bypass this by creating a .env file with dummy values:

OPENAI_API_KEY="NA"
OPENAI_API_BASE="http://localhost:11434/v1"

Phase 5: Production Crew Code (crew.py)

from crewai import Agent, Task, Crew, Process, LLM

# 1. Define Local GPU LLM
bare_metal_llm = LLM(
    model="ollama/qwen2.5:14b",
    base_url="http://localhost:11434",
    temperature=0.2
)

# 2. Architect Agents (Disable infinite delegation loops)
research_agent = Agent(
    role="Infrastructure Security Analyst",
    goal="Discover vulnerabilities in cloud VM networking",
    backstory="Elite SRE who trusts only bare metal servers.",
    llm=bare_metal_llm,
    verbose=True,
    allow_delegation=False
)

writer_agent = Agent(
    role="DevSecOps Technical Writer",
    goal="Draft an actionable security report based on findings",
    backstory="Synthesizes complex security data into Markdown.",
    llm=bare_metal_llm,
    verbose=True,
    allow_delegation=False
)

# 3. Define Tasks
research_task = Task(
    description="Analyze why multi-tenant Cloud VMs are less secure than Dedicated Bare Metal. List 3 key points.",
    expected_output="3 technical bullet points regarding hypervisor vulnerabilities.",
    agent=research_agent
)

write_task = Task(
    description="Write a 2-paragraph security advisory from the 3 bullet points.",
    expected_output="Formatted Markdown security advisory document.",
    agent=writer_agent,
    context=[research_task]
)

# 4. Initialize and Run Crew
production_crew = Crew(
    agents=[research_agent, writer_agent],
    tasks=[research_task, write_task],
    process=Process.sequential
)

if __name__ == "__main__":
    result = production_crew.kickoff()
    print("\n--- FINAL OUTPUT ---\n", result)

Deployment Model Comparison

👉 Ready to deploy sovereign multi-agent AI on bare metal GPUs? Read the full deployment guide on ServerMO:
Setup CrewAI on Ubuntu 24.04 Bare Metal GPU | ServerMO