Running AI workloads on NVIDIA GPUs

AI workloads live and die by one thing: whether the GPU is actually visible to the process that wants it. Most of the pain in a GPU host isn't CUDA — it's the driver and container toolkit disagreeing about which reality they're in.

Installing the NVIDIA driver on RHEL-family systems

On AlmaLinux/RHEL, the cleanest path is the official NVIDIA repo rather than the distro's default packages, since you want driver and CUDA toolkit versions you can pin deliberately.

dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
dnf module install -y nvidia-driver:latest-dkms

Reboot, then confirm the driver actually loaded:

nvidia-smi

Getting containers to see the GPU

A driver on the host doesn't automatically mean a container can use it. That's what the NVIDIA Container Toolkit is for — it wires up the runtime hooks so docker run --gpus all actually works.

dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Sanity check with the official CUDA base image:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubi9 nvidia-smi

If that prints the same GPU table as the host command, the whole chain — kernel module, driver, container runtime hook — is working end to end.

Where this actually goes

Once containers can see the GPU reliably, the rest is normal container platform work: scheduling, resource limits (--gpus '"device=0"' for pinning a specific card in a multi-GPU box), and deciding whether workloads run as long-lived inference services or short batch jobs. That's the more interesting part, and it's mostly independent of the driver plumbing above — which is exactly the point of getting the plumbing right once.