[Question]: `driver.enabled=true` when the driver is already installed correct behavior?

[e-minguez]

driver.enabled=true is the default value https://github.com/NVIDIA/gpu-operator/blob/main/deployments/gpu-operator/templates/nvidiadriver.yaml#L1 on the chart (so it is enabled even if not specified)

What should be the driver.enabled=true flag behavior if the NVIDIA driver is in fact already installed?

In my tests I can observe the nvidia drivers are not being installed by the gpu operator as it seems the operator recognizes they are installed already.
It seems the init container k8s-driver-manager (https://github.com/NVIDIA/k8s-driver-manager ) does some checks on the host and if the driver is there, it doesn't install it. The driver container image nvcr.io/nvidia/driver:580.105.08-sles15.6 doesn't seem to exist, but again, it is not pulled because the driver is already installed.

These are the crd objects created by the gpu-operator:

kubectl get crd |grep -i nvidia
clusterpolicies.nvidia.com                                 2026-02-18T12:55:43Z
nvidiadrivers.nvidia.com                                   2026-02-18T12:55:43Z

kubectl get nvidiadrivers.nvidia.com -A
No resources found

kubectl get clusterpolicies.nvidia.com -A
NAME             STATUS   AGE
cluster-policy   ready    2026-02-18T12:55:45Z

kubectl get clusterpolicies.nvidia.com cluster-policy -o yaml
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  annotations:
    meta.helm.sh/release-name: gpu-operator
    meta.helm.sh/release-namespace: gpu-operator
  creationTimestamp: "2026-02-18T12:55:45Z"
  generation: 1
  labels:
    app.kubernetes.io/component: gpu-operator
    app.kubernetes.io/instance: gpu-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: gpu-operator
    app.kubernetes.io/version: v25.10.1
    helm.sh/chart: gpu-operator-v25.10.1
  name: cluster-policy
  resourceVersion: "2161"
  uid: 0a047789-bb7f-479d-ac7a-a4da6a83f614
spec:
  ccManager:
    defaultMode: "off"
    enabled: false
    env:
    - name: CC_CAPABLE_DEVICE_IDS
      value: 0x2339,0x2331,0x2330,0x2324,0x2322,0x233d
    image: k8s-cc-manager
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/cloud-native
    version: v0.1.1
  cdi:
    default: false
    enabled: true
  daemonsets:
    labels:
      app.kubernetes.io/managed-by: gpu-operator
      helm.sh/chart: gpu-operator-v25.10.1
    priorityClassName: system-node-critical
    rollingUpdate:
      maxUnavailable: "1"
    tolerations:
    - effect: NoSchedule
      key: nvidia.com/gpu
      operator: Exists
    updateStrategy: RollingUpdate
  dcgm:
    enabled: false
    image: dcgm
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/cloud-native
    version: 4.4.2-1-ubuntu22.04
  dcgmExporter:
    enabled: true
    image: dcgm-exporter
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/k8s
    service:
      internalTrafficPolicy: Cluster
    serviceMonitor:
      additionalLabels: {}
      enabled: false
      honorLabels: false
      interval: 15s
      relabelings: []
    version: 4.4.2-4.7.0-distroless
  devicePlugin:
    enabled: true
    image: k8s-device-plugin
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia
    version: v0.18.1
  driver:
    certConfig:
      name: ""
    enabled: true
    image: driver
    imagePullPolicy: IfNotPresent
    kernelModuleConfig:
      name: ""
    kernelModuleType: auto
    licensingConfig:
      nlsEnabled: true
      secretName: ""
    manager:
      image: k8s-driver-manager
      imagePullPolicy: IfNotPresent
      repository: nvcr.io/nvidia/cloud-native
      version: v0.9.1
    rdma:
      enabled: false
      useHostMofed: true
    repoConfig:
      configMapName: ""
    repository: nvcr.io/nvidia
    startupProbe:
      failureThreshold: 120
      initialDelaySeconds: 60
      periodSeconds: 10
      timeoutSeconds: 60
    upgradePolicy:
      autoUpgrade: true
      drain:
        deleteEmptyDir: false
        enable: false
        force: false
        timeoutSeconds: 300
      maxParallelUpgrades: 1
      maxUnavailable: 25%
      podDeletion:
        deleteEmptyDir: false
        force: false
        timeoutSeconds: 300
      waitForCompletion:
        timeoutSeconds: 0
    useNvidiaDriverCRD: false
    usePrecompiled: false
    version: 580.105.08
    virtualTopology:
      config: ""
  gdrcopy:
    enabled: false
    image: gdrdrv
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/cloud-native
    version: v2.5.1
  gds:
    enabled: false
    image: nvidia-fs
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/cloud-native
    version: 2.26.6
  gfd:
    enabled: true
    image: k8s-device-plugin
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia
    version: v0.18.1
  hostPaths:
    driverInstallDir: /run/nvidia/driver
    rootFS: /
  kataManager:
    config:
      artifactsDir: /opt/nvidia-gpu-operator/artifacts/runtimeclasses
      runtimeClasses:
      - artifacts:
          pullSecret: ""
          url: nvcr.io/nvidia/cloud-native/kata-gpu-artifacts:ubuntu22.04-535.54.03
        name: kata-nvidia-gpu
        nodeSelector: {}
      - artifacts:
          pullSecret: ""
          url: nvcr.io/nvidia/cloud-native/kata-gpu-artifacts:ubuntu22.04-535.86.10-snp
        name: kata-nvidia-gpu-snp
        nodeSelector:
          nvidia.com/cc.capable: "true"
    enabled: false
    image: k8s-kata-manager
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/cloud-native
    version: v0.2.3
  mig:
    strategy: single
  migManager:
    config:
      default: all-disabled
      name: default-mig-parted-config
    enabled: true
    gpuClientsConfig:
      name: ""
    image: k8s-mig-manager
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/cloud-native
    version: v0.13.1
  nodeStatusExporter:
    enabled: false
    image: gpu-operator
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia
    version: v25.10.1
  operator:
    defaultRuntime: docker
    initContainer:
      image: cuda
      imagePullPolicy: IfNotPresent
      repository: nvcr.io/nvidia
      version: 13.0.1-base-ubi9
    runtimeClass: nvidia
  psa:
    enabled: false
  sandboxDevicePlugin:
    enabled: true
    image: kubevirt-gpu-device-plugin
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia
    version: v1.4.0
  sandboxWorkloads:
    defaultWorkload: container
    enabled: false
  toolkit:
    enabled: true
    env:
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock
    image: container-toolkit
    imagePullPolicy: IfNotPresent
    installDir: /usr/local/nvidia
    repository: nvcr.io/nvidia/k8s
    version: v1.18.1
  validator:
    image: gpu-operator
    imagePullPolicy: IfNotPresent
    plugin:
      env: []
    repository: nvcr.io/nvidia
    version: v25.10.1
  vfioManager:
    driverManager:
      image: k8s-driver-manager
      imagePullPolicy: IfNotPresent
      repository: nvcr.io/nvidia/cloud-native
      version: v0.9.1
    enabled: true
    image: cuda
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia
    version: 13.0.1-base-ubi9
  vgpuDeviceManager:
    config:
      default: default
      name: ""
    enabled: true
    image: vgpu-device-manager
    imagePullPolicy: IfNotPresent
    repository: nvcr.io/nvidia/cloud-native
    version: v0.4.1
  vgpuManager:
    driverManager:
      image: k8s-driver-manager
      imagePullPolicy: IfNotPresent
      repository: nvcr.io/nvidia/cloud-native
      version: v0.9.1
    enabled: false
    image: vgpu-manager
    imagePullPolicy: IfNotPresent
status:
  conditions:
  - lastTransitionTime: "2026-02-18T12:56:53Z"
    message: ClusterPolicy is ready as all resources have been successfully reconciled
    reason: Reconciled
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-02-18T12:56:53Z"
    message: ""
    reason: Ready
    status: "False"
    type: Error
  namespace: gpu-operator
  state: ready

There was a daemonset driver pod running for a little short amount of time. This is the daemonset describe for the nvidia-driver-daemonset:

kubectl describe daemonset -n gpu-operator nvidia-driver-daemonset
Name:           nvidia-driver-daemonset
Namespace:      gpu-operator
Selector:       app=nvidia-driver-daemonset
Node-Selector:  nvidia.com/gpu.deploy.driver=true
Labels:         app=nvidia-driver-daemonset
                app.kubernetes.io/component=nvidia-driver
                app.kubernetes.io/managed-by=gpu-operator
                helm.sh/chart=gpu-operator-v25.10.1
                nvidia.com/precompiled=false
Annotations:    deprecated.daemonset.template.generation: 1
                nvidia.com/last-applied-hash: 2814262311
                openshift.io/scc: nvidia-driver
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=nvidia-driver-daemonset
                    app.kubernetes.io/component=nvidia-driver
                    app.kubernetes.io/managed-by=gpu-operator
                    helm.sh/chart=gpu-operator-v25.10.1
                    nvidia.com/precompiled=false
  Annotations:      kubectl.kubernetes.io/default-container: nvidia-driver-ctr
  Service Account:  nvidia-driver
  Init Containers:
   k8s-driver-manager:
    Image:      nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.9.1
    Port:       <none>
    Host Port:  <none>
    Command:
      driver-manager
    Args:
      uninstall_driver
    Environment:
      NODE_NAME:                    (v1:spec.nodeName)
      NVIDIA_VISIBLE_DEVICES:      void
      ENABLE_GPU_POD_EVICTION:     true
      ENABLE_AUTO_DRAIN:           false
      DRAIN_USE_FORCE:             false
      DRAIN_POD_SELECTOR_LABEL:
      DRAIN_TIMEOUT_SECONDS:       0s
      DRAIN_DELETE_EMPTYDIR_DATA:  false
      OPERATOR_NAMESPACE:           (v1:metadata.namespace)
    Mounts:
      /host from host-root (ro)
      /run/mellanox/drivers from run-mellanox-drivers (rw)
      /run/nvidia from run-nvidia (rw)
      /sys from host-sys (rw)
  Containers:
   nvidia-driver-ctr:
    Image:      nvcr.io/nvidia/driver:580.105.08-sles15.7
    Port:       <none>
    Host Port:  <none>
    Command:
      nvidia-driver
    Args:
      init
    Startup:  exec [sh -c [ -f /sys/module/nvidia/refcnt ] && nvidia-smi && touch /run/nvidia/validations/.driver-ctr-ready] delay=60s timeout=60s period=10s #success=1 #failure=120
    Environment:
      NODE_NAME:            (v1:spec.nodeName)
      NODE_IP:              (v1:status.hostIP)
      KERNEL_MODULE_TYPE:  auto
    Mounts:
      /dev/log from dev-log (rw)
      /etc/SUSEConnect from subscription-config-0 (ro)
      /etc/zypp/credentials.d from subscription-config-1 (ro)
      /host-etc/os-release from host-os-release (ro)
      /lib/firmware from nv-firmware (rw)
      /run/mellanox/drivers from run-mellanox-drivers (rw)
      /run/mellanox/drivers/usr/src from mlnx-ofed-usr-src (rw)
      /run/nvidia from run-nvidia (rw)
      /run/nvidia-fabricmanager from run-nvidia-fabricmanager (rw)
      /run/nvidia-topologyd from run-nvidia-topologyd (rw)
      /sys/devices/system/memory/auto_online_blocks from sysfs-memory-online (rw)
      /sys/module/firmware_class/parameters/path from firmware-search-path (rw)
      /var/log from var-log (rw)
  Volumes:
   run-nvidia:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia
    HostPathType:  DirectoryOrCreate
   var-log:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:
   dev-log:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/log
    HostPathType:
   host-os-release:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/os-release
    HostPathType:
   run-nvidia-fabricmanager:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia-fabricmanager
    HostPathType:  DirectoryOrCreate
   run-nvidia-topologyd:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia-topologyd
    HostPathType:  DirectoryOrCreate
   mlnx-ofed-usr-src:
    Type:          HostPath (bare host directory volume)
    Path:          /run/mellanox/drivers/usr/src
    HostPathType:  DirectoryOrCreate
   run-mellanox-drivers:
    Type:          HostPath (bare host directory volume)
    Path:          /run/mellanox/drivers
    HostPathType:  DirectoryOrCreate
   run-nvidia-validations:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/validations
    HostPathType:  DirectoryOrCreate
   host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:
   host-sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  Directory
   firmware-search-path:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/module/firmware_class/parameters/path
    HostPathType:
   sysfs-memory-online:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/devices/system/memory/auto_online_blocks
    HostPathType:
   nv-firmware:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/driver/lib/firmware
    HostPathType:  DirectoryOrCreate
   subscription-config-0:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/SUSEConnect
    HostPathType:  FileOrCreate
   subscription-config-1:
    Type:               HostPath (bare host directory volume)
    Path:               /etc/zypp/credentials.d
    HostPathType:       Directory
  Priority Class Name:  system-node-critical
  Node-Selectors:       nvidia.com/gpu.deploy.driver=true
  Tolerations:          nvidia.com/gpu:NoSchedule op=Exists
Events:
  Type    Reason            Age    From                  Message
  ----    ------            ----   ----                  -------
  Normal  SuccessfulCreate  5m28s  daemonset-controller  Created pod: nvidia-driver-daemonset-2p46d
  Normal  SuccessfulDelete  5m22s  daemonset-controller  Deleted pod: nvidia-driver-daemonset-2p46d

The steps to reproduce it are:

• Deploy a SLES 15.7 host with NVIDIA GPU
• Register, install, update:

sudo SUSEConnect --url=https://scc.suse.com -e xx@example.com -r XXX
sudo zypper ref
sudo zypper up -y
sudo SUSEConnect -p PackageHub/15.7/x86_64
sudo SUSEConnect -p sle-module-containers/15.7/x86_64
sudo zypper in -y acl yq git-core btop tmux podman helm pciutils
sudo setfacl -m u:$(id -nu):r /etc/zypp/credentials.d/*
sudo reboot

• Install the NVIDIA drivers. As per https://documentation.suse.com/suse-ai/1.0/html/AI-deployment/index.html#ai-nvidia-gpu-driver

sudo zypper ar \
  https://developer.download.nvidia.com/compute/cuda/repos/sles15/x86_64/ \
  cuda-sle15
sudo zypper --gpg-auto-import-keys refresh

sudo zypper install -y --auto-agree-with-licenses \
  nv-prefer-signed-open-driver
export version=$(rpm -qa --queryformat '%{VERSION}\n' \
  nv-prefer-signed-open-driver | cut -d "_" -f1 | sort -u | tail -n 1)

sudo zypper install -y --auto-agree-with-licenses \
  nvidia-compute-utils-G06=${version} \
  nvidia-persistenced=${version}

sudo reboot

• RKE2 installation:

sudo mkdir -p /etc/rancher/rke2
cat << EOF | sudo tee /etc/rancher/rke2/config.yaml
token: "foobar123."
write-kubeconfig-mode: "0644"
EOF
curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION="v1.34.3+rke2r3" sh -
sudo systemctl start --now rke2-server
sudo ln -sf /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml" | sudo tee /etc/profile.d/rke2.sh
sudo chmod +r /etc/profile.d/rke2.sh
source /etc/profile.d/rke2.sh

• Install gpu-operator

cat << EOF | kubectl apply -f -
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock
EOF

This is basically the same as running helm install but leveraging RKE2 helm's operator.

After a few seconds, the nodes are labeled properly:

kubectl get nodes -o custom-columns="NAME:.metadata.name,Gpus:.status.allocatable.nvidia\.com/gpu"
NAME                                       Gpus
ip-10-0-1-154.us-east-2.compute.internal   2
ip-10-0-1-89.us-east-2.compute.internal    2

And you can schedule and run containers using gpus as expected.

[e-minguez]

Turns out I was re-reading the docs here https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#pre-installed-nvidia-gpu-drivers and in fact says:

If you do not specify the driver.enabled=false argument and nodes in the cluster have a pre-installed GPU driver, the init container in the driver pod detects that the driver is preinstalled and labels the node so that the driver pod is terminated and does not get re-scheduled on to the node. The Operator proceeds to start other pods, such as the container toolkit pod.

So it seems the behavior observed. The node has the driver, init container detects it and skips the driver installation.

[rajathagasthya]

Closing this as driver installation behavior works as expected. Please reopen if necessary.

[jeremych1000]

What if you WANT the operator to manage the driver? Do you have to uninstall the driver from the underlying host first? Can there be an "overwrite the driver" mode?

[rajathagasthya]

@jeremych1000 Yes, you need to uninstall the driver from the host first. The operator cannot "overwrite" the driver. The k8s-driver-manager checks for pre-installed driver on the host and gets terminated. Please see https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#pre-installed-nvidia-gpu-drivers