Self-host Ollama

Before you start

Ollama is the "run open-weight LLMs locally" tool. The install is trivial. The interesting decisions are all about hardware. This guide covers two paths:

• Path A — CPU-only on a Hetzner dedicated-CPU CCX23 (€22.90/mo). Good enough for 7-8B models at 5-15 tokens/sec. Right for personal use or team-of-few.
• Path B — GPU-enabled box (Hetzner GEX44 or your own hardware) for 14B-70B models. Needed for production-grade team use.

We'll do Path A end-to-end and note Path B differences. You'll also add Open WebUI for a ChatGPT-like interface.

Step 1 · Provision the server (5 min)

For Path A, pick a Hetzner CCX23 (dedicated vCPU, 4 cores, 16 GB RAM). Ubuntu 24.04 LTS. SSH key.

ssh root@YOUR_SERVER_IP
adduser deploy && usermod -aG sudo deploy
rsync --archive --chown=deploy:deploy ~/.ssh /home/deploy
sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh

For Path B (GPU), Hetzner's GEX44 has an RTX 4000 SFF Ada (20 GB VRAM). Cost is ~€184/month but you'll run 30B-class models at real speed.

Step 2 · DNS (5 min)

ollama.yourcompany.com → YOUR_SERVER_IP
chat.yourcompany.com → YOUR_SERVER_IP

Two subdomains: one for Ollama's API, one for Open WebUI.

Step 3 · Install Docker (5 min)

sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER && newgrp docker

For Path B (GPU), also install the NVIDIA Container Toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 4 · Compose stack: Ollama + Open WebUI (10 min)

mkdir -p ~/ollama && cd ~/ollama
mkdir -p caddy_data caddy_config ollama_data openwebui_data

~/ollama/docker-compose.yml:

services:
  ollama:
    image: ollama/ollama:latest
    restart: unless-stopped
    volumes:
      - ./ollama_data:/root/.ollama
    expose:
      - "11434"
    # For Path B (GPU), uncomment:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           capabilities: [gpu]

  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    restart: unless-stopped
    depends_on:
      - ollama
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
      WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY}
      WEBUI_URL: https://${CHAT_HOST}
    volumes:
      - ./openwebui_data:/app/backend/data
    expose:
      - "8080"

  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy_data:/data
      - ./caddy_config:/config
    depends_on:
      - openwebui

~/ollama/Caddyfile:

${CHAT_HOST} {
  reverse_proxy openwebui:8080
}

${OLLAMA_HOST} {
  reverse_proxy ollama:11434
  # Protect the API — Ollama has no built-in auth
  basicauth {
    admin ${OLLAMA_BASIC_AUTH_HASH}
  }
}

~/ollama/.env:

CHAT_HOST=chat.yourcompany.com
OLLAMA_HOST=ollama.yourcompany.com
WEBUI_SECRET_KEY=RANDOM_32_CHARS
OLLAMA_BASIC_AUTH_HASH=OUTPUT_OF_CADDY_HASH_PASSWORD

Generate the basic-auth hash:

docker run --rm caddy:2-alpine caddy hash-password --plaintext 'your-secret-here'

Step 5 · Bring it up (5 min)

docker compose up -d
docker compose logs -f openwebui

Open https://chat.yourcompany.com. Sign up as the first user — that account becomes admin.

Step 6 · Pull your first model (5-15 min)

Small model to start — Llama 3.3 8B:

docker exec -it ollama-ollama-1 ollama pull llama3.3:8b

Takes 5-10 minutes on a fast connection (4.7 GB download). For CPU-only inference, this is your ceiling — anything larger will be too slow to enjoy. GPU users can pull larger models (llama3.3:70b, qwen3:32b, deepseek-r1:32b).

Reload Open WebUI. The model appears in the model selector. Start a conversation.

Step 7 · Firewall

sudo ufw allow 22/tcp && sudo ufw allow 80/tcp && sudo ufw allow 443/tcp && sudo ufw --force enable

Common pitfalls

• Ollama has no built-in auth. If you expose the API to the internet, anyone can use your GPU cycles. The Caddy basicauth in the config above is the minimum. Better: keep Ollama internal to Docker and only expose Open WebUI.
• Downloads consume disk fast. Each model is 4-40 GB. Budget 100 GB minimum for a serious model library.
• CPU inference gets slow after 8B. On a 4-vCPU box, a 14B model runs at 2-4 tokens/sec — technically usable, practically painful. Match model size to your hardware honestly.
• Open WebUI's file/RAG features need extra RAM. If you use document upload heavily, the CX22-class 4 GB isn't enough. CCX23's 16 GB gives headroom.
• Chat history uses local SQLite by default. For a shared team instance, migrate to Postgres via DATABASE_URL environment variable.

Updating

cd ~/ollama
docker compose pull
docker compose up -d
docker image prune -f

Model recommendations by hardware

• 4 GB RAM, CPU-only: qwen2.5:3b, phi3:mini. Real-time on modest CPUs.
• 16 GB RAM, CPU-only (CCX23): llama3.3:8b, qwen3:8b, mistral:7b. 5-15 tokens/sec.
• 16-24 GB VRAM (RTX 4000-4090): qwen3:32b, deepseek-r1:32b, llama3.3:70b at 4-bit. 30-80 tokens/sec.
• 48+ GB VRAM (dual 3090s or better): llama3.3:70b at higher quantization, or run multiple models simultaneously.

What you're paying vs ChatGPT Team

• Ollama on Hetzner CCX23 (CPU-only, 5 casual users): €275/year (~$300) + your time.
• Ollama on Hetzner GEX44 (GPU, 20 heavy users): €2,200/year (~$2,400).
• ChatGPT Team, 20 users: $7,200/year.
• Renting cloud GPUs by the hour (Runpod, Vast.ai): $0.30-1.50/hr; often the sweet spot for spiky use.

Self-hosted wins the cost math above 15-20 users, or when data-sovereignty is non-negotiable. Below that, ChatGPT Plus for individuals is often the honest choice.