↓Skip to main content

NVIDIA CUDA setup and NVIDIA device plugin for Kubernetes on Debian and k3s

957 words·
NVIDIA CUDA Linux Debian Kubernetes K3s Helm Ansible
Table of Contents

Kubernetes Setup
#

I use the following k3s based Kubernetes nodes in this tutorial:

# List nodes
kubectl get node

# Shell output:
NAME        STATUS   ROLES                AGE   VERSION
k3s-1-1     Ready    control-plane,etcd   29d   v1.36.4+k3s1
k3s-w1gpu   Ready    gpu-worker           16m   v1.36.4+k3s1

CUDA Installation: Manual
#

Disable Secure Boot validation (optional)
#

The following steps keep UEFI Secure Boot enabled, but configure the Debian shim bootloader to stop enforcing signature validation. This avoids having to enroll a signing key for the NVIDIA DKMS module.

# List SB state from UEFI and shim bootloader
mokutil --sb-state

# Shell output:
secureBoot enabled

# Disable SB
sudo mokutil --disable-validation

# Shell output:
input password: <set-mok-pw>
input password again: <set-mok-pw>
  • Reboot
  • Boot into MOK management menu
  • Select “Change Secure Boot State”
# List SB state from UEFI and shim bootloader
mokutil --sb-state

# Shell output:
SecureBoot enabled
SecureBoot validation is disabled in shim

Install CUDA
# 

# Install prerequisites
sudo apt update
sudo apt install linux-headers-$(uname -r) dkms wget

# Add repository
wget https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
  • linux-headers-$(uname -r) Headers matching the currently running kernel
  • dkms Builds the NVIDIA kernel module and rebuilds it after kernel updates.

# Build NVIDIA kernel module and install CUDA driver
sudo apt install \
    nvidia-driver-cuda \
    nvidia-kernel-open-dkms

# Reboot server
sudo reboot
  • nvidia-kernel-open-dkms Uses DKMS and builds NVIDIA’s open kernel module
  • nvidia-driver-cuda CUDA driver support and tools

Install Container Toolkit
# 

# Install NVIDIA container runtime
sudo apt install nvidia-container-toolkit
  • nvidia-container-toolkit NVIDIA container runtime
# Restart k3s agent service so k3s detects the NVIDIA runtime
sudo systemctl restart k3s-agent

# Verify container runtime
sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml

# Shell output:
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia']
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.'nvidia'.options]
  BinaryName = "/usr/bin/nvidia-container-runtime"

# List runtimeclass
kubectl get runtimeclass nvidia

# Shell output:
NAME     HANDLER   AGE
nvidia   nvidia    29d



CUDA Installation: Ansible Playbook
# 

- name: Install CUDA on Debian
  hosts: k3s_1_worker
  become: true
  gather_facts: true

  tasks:
    - name: Install prerequisites
      ansible.builtin.apt:
        name:
          - "linux-headers-{{ ansible_facts['kernel'] }}"
          - dkms
          - wget
        state: present
        update_cache: true

    - name: Download NVIDIA CUDA repository keyring
      ansible.builtin.get_url:
        url: https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/cuda-keyring_1.1-1_all.deb
        dest: /tmp/cuda-keyring_1.1-1_all.deb
        mode: '0644'

    - name: Install NVIDIA CUDA repository keyring
      ansible.builtin.apt:
        deb: /tmp/cuda-keyring_1.1-1_all.deb

    - name: Update APT package lists
      ansible.builtin.apt:
        update_cache: true

    - name: Install NVIDIA driver and open kernel module
      ansible.builtin.apt:
        name:
          - nvidia-driver-cuda
          - nvidia-kernel-open-dkms
          - nvidia-container-toolkit
        state: present

    - name: Restart k3s agent to detect the NVIDIA container runtime
      ansible.builtin.systemd_service:
        name: k3s-agent
        state: restarted
      become: true



Verify GPU Usage
# 

# Nvidia GPU monitoring
nvidia-smi

# Shell output:    
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.71.09              KMD Version: 615.71.09     CUDA UMD Version: 13.4     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070 ...    On  |   00000000:01:00.0 Off |                  N/A |
|  0%   30C    P8              1W /  220W |       1MiB /  12282MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+



NVIDIA device plugin for Kubernetes
#

The NVIDIA device plugin for Kubernetes is a Daemonset that allows you to automatically:

  • Expose the number of GPUs on each nodes of your cluster
  • Keep track of the health of your GPUs
  • Run GPU enabled containers in your Kubernetes cluster.

Link: https://github.com/NVIDIA/k8s-device-plugin/tree/main


Helm Overview
# 

# Add NVIDIA Device Plugin Helm repository
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin &&
helm repo update

# List available Helm chart versions
helm search repo nvdp/nvidia-device-plugin --versions

# Shell output:
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
nvdp/nvidia-device-plugin       0.20.1          0.20.1          A Helm chart for the nvidia-device-plugin on Ku...
nvdp/nvidia-device-plugin       0.20.0          0.20.0          A Helm chart for the nvidia-device-plugin on Ku...

# Save Helm chart values
helm show values nvdp/nvidia-device-plugin --version 0.20.1 > nvidia-device-plugin.yml

Ansible Setup
#

Playbook
# 

- name: NVIDIA Device Plugin Helm setup
  hosts: localhost
  connection: local
  gather_facts: false
  become: false
  vars:
    # Helm Configuration
    helm_repo_url: "https://nvidia.github.io/k8s-device-plugin"
    helm_repo_name: "nvdp"
    helm_chart: "nvdp/nvidia-device-plugin"
    helm_chart_version: "0.20.1"
    helm_release_name: "nvdp"
    # Kubernetes Configuration
    kubernetes_namespace: "nvidia-device-plugin"
    # GPU Node
    gpu_nodes: 
      - k3s-w1gpu

  roles:
    - k8s_nvidia_device_plugin

# Run playbook: 
ansible-playbook playbooks/k8s_nvidia_device_plugin.yml

Tasks
#

k8s_nvidia_device_plugin/tasks/main.yml

- name: Label GPU workers for the NVIDIA device plugin
  kubernetes.core.k8s:
    state: patched
    api_version: v1
    kind: Node
    name: "{{ item }}"
    definition:
      metadata:
        labels:
          nvidia.com/gpu.present: "true"
  loop: "{{ gpu_nodes }}"
  delegate_to: localhost

- name: Create namespace
  kubernetes.core.k8s:
    api_version: v1
    kind: Namespace
    name: "{{ kubernetes_namespace }}"
    state: present

- name: Add Helm repository
  kubernetes.core.helm_repository:
    name: "{{ helm_repo_name }}"
    repo_url: "{{ helm_repo_url }}"
    force_update: true

- name: Install Helm Chart
  kubernetes.core.helm:
    name: "{{ helm_release_name }}"
    chart_ref: "{{ helm_chart }}"
    chart_version: "{{ helm_chart_version }}"
    release_namespace: "{{ kubernetes_namespace }}"
    create_namespace: false
    wait: true  # Ansible waits till all resources are ready
    wait_timeout: 5m0s
    atomic: false  # Auto-rollback on failure
    values: "{{ lookup('template', 'helm-values.yml.j2') | from_yaml }}"

Templates
#

k8s_nvidia_device_plugin/tasks/main.yml

runtimeClassName: nvidia

devicePlugin:
  enabled: true

gfd:
  enabled: false

nfd:
  enabled: false



Kubernetes Resources
# 

# List default resources
kubectl -n nvidia-device-plugin get all

# Shell output:
NAME                                  READY   STATUS    RESTARTS   AGE
pod/nvdp-nvidia-device-plugin-7sggl   1/1     Running   0          104s

NAME                                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
daemonset.apps/nvdp-nvidia-device-plugin                      1         1         1       1            1           <none>                        104s
daemonset.apps/nvdp-nvidia-device-plugin-mps-control-daemon   0         0         0       0            0           nvidia.com/mps.capable=true   104s

Verify the GPU is available to the node:

# List node details
kubectl describe node k3s-w1gpu

# Shell output:
...
Capacity:
  cpu:                24
  ephemeral-storage:  191135628Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             64686324Ki
  nvidia.com/gpu:     1  # Check
  pods:               110
Allocatable:
  cpu:                24
  ephemeral-storage:  185936738773
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             64686324Ki
  nvidia.com/gpu:     1  # Check
  pods:               110

Verify Worker Node Label:

# Verify worker node label
ubectl get node k3s-w1gpu -L nvidia.com/gpu.present

# Shell output:
NAME        STATUS   ROLES        AGE     VERSION        GPU.PRESENT
k3s-w1gpu   Ready    gpu-worker   4h12m   v1.36.4+k3s1   true