Proxmox cluster: two mismatched nodes and a quorum lesson

The virtualization layer is two Proxmox VE nodes in one cluster. They are very different machines, and most of what I've learned about running this lab has come from that mismatch.

The nodes

Node A is an old desktop platform that refuses to die. Node B is a small-form-factor box that sips power. Both run the same Proxmox VE version and the same network pattern: a VLAN-aware bridge that VMs attach to with a tag.

What runs where

VMs are spread so that anything with a twin lives on the other node from its twin:

Building VMs from cloud images

No ISO installs. New VMs are AlmaLinux GenericCloud qcow2 images imported directly and configured with Proxmox's built-in cloud-init:

qm create 120 --name lab-vm --memory 4096 --cores 2 \
  --net0 virtio,bridge=vmbr0,tag=<vlan> --scsihw virtio-scsi-single
qm importdisk 120 AlmaLinux-10-GenericCloud-latest.x86_64.qcow2 local-lvm
qm set 120 --scsi0 local-lvm:vm-120-disk-0 --boot order=scsi0 \
  --ide2 local-lvm:cloudinit \
  --ciuser admin --sshkeys ~/.ssh/authorized_keys \
  --ipconfig0 ip=dhcp
qm resize 120 scsi0 +30G
qm start 120

Storage

• Local LVM on each node for VM disks: fast, simple, no shared-storage dependency.
• NFS share on the NAS for backups (vzdump).
• SMB share on the NAS for ISOs and templates, visible to both nodes.

There is no Ceph and no shared VM storage. With two nodes and 1 GbE it would cost more than it gives. Live migration isn't the goal; fast restore is.

The quorum lesson

The cluster was built for three nodes. Right now the third is offline, so the cluster runs on two votes out of three: still quorate, but with zero margin. Lose one node and the other goes read-only for cluster operations until the third comes back.

pvecm status | grep -E "Expected votes|Total votes|Quorate"

For Proxmox itself that's a known, accepted trade-off. For the application HA stack (etcd and Redis Sentinel behind Ops Center) it wasn't acceptable. I added a tiny witness VM (2 vCPU, 2 GB) whose only job is to be the third member. It sits on the smaller node on purpose: the scenario I care about is patching and rebooting the big node, and with the witness on the other side, quorum survives that.

CPU overcommit is real

Node B has 4 threads. At one point its VMs added up to 16 vCPUs, and the host load sat at saturation. The first thing to suffer wasn't the busy VM. It was the primary DNS server on the same node, which started timing out while the secondary on the other node was fine. That asymmetry is what gave it away.

The fixes:

• Cap the noisy VM instead of trusting the scheduler: qm set <vmid> --cpulimit 2
• Move demo/test stacks off the small node entirely
• Treat vCPU totals per node as a budget, not a free-for-all

# quick overcommit check per node
qm list | awk 'NR>1 && $3=="running"{print $1}' | \
  xargs -I{} qm config {} | awk '/^cores:/{s+=$2} END{print s " vCPU allocated"}'
nproc

Backups

Weekly vzdump jobs write every VM to the NAS over NFS, with Proxmox notifications routed into my monitoring. The backup share is the current bottleneck: the biggest Docker host alone is several hundred GB, so the next upgrade is either more space or Proxmox Backup Server with deduplication.