Error enabling MPS with k3s and Helm (nvdp): ERROR_LIBRARY_NOT_FOUND on NVML initialization

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

Hello,
I am encountering an error when trying to enable MPS (Multi-Process Service) while installing the NVIDIA Device Plugin (nvdp) using Helm on a k3s cluster. I would appreciate any guidance on resolving this issue.
There is some information about my k3s and jetson orin nano.

root@nano:~/model# uname -a Linux nano 5.15.148-tegra #1 SMP PREEMPT Sun Jul 6 16:03:40 CST 2025 aarch64 aarch64 aarch64 GNU/Linux root@nano:~/model# cat /etc/nv_tegra_release # R36 (release), REVISION: 4.4, GCID: 41062509, BOARD: generic, EABI: aarch64, DATE: Mon Jun 16 16:07:13 UTC 2025 # KERNEL_VARIANT: oot TARGET_USERSPACE_LIB_DIR=nvidia TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia 
cat << EOF > /opt/dp-mps.yaml version: v1 sharing: mps: resources: - name: nvidia.com/gpu replicas: 2 EOF helm install nvdp nvdp/nvidia-device-plugin \ --namespace nvidia-device-plugin \ --create-namespace \ --set config.default=mps \ --set-file config.map.mps=/opt/dp-mps.yaml 
root@k3s-master:~# kubectl logs -n nvidia-device-plugin nvdp-nvidia-device-plugin-mps-control-daemon-ggf4m -c mps-control-daemon-ctr I1110 10:03:43.912793 448393 main.go:80] "NVIDIA MPS Control Daemon" version=<	3c9ffca9	commit: 3c9ffca9491f0d2d362a7064138dfcd71bb57592 > I1110 10:03:43.913035 448393 main.go:109] Starting OS watcher. I1110 10:03:43.913225 448393 main.go:123] Starting Daemons. I1110 10:03:43.913263 448393 main.go:166] Loading configuration. I1110 10:03:43.913818 448393 main.go:181] Updating config with default resource matching patterns. I1110 10:03:43.913883 448393 main.go:192] Running with config: { "version": "v1", "flags": { "migStrategy": "none", "failOnInitError": null, "gdrcopyEnabled": null, "gdsEnabled": null, "mofedEnabled": null, "useNodeFeatureAPI": null, "deviceDiscoveryStrategy": null, "plugin": { "passDeviceSpecs": null, "deviceListStrategy": null, "deviceIDStrategy": null, "cdiAnnotationPrefix": null, "nvidiaCTKPath": null, "containerDriverRoot": null } }, "resources": { "gpus": [ { "pattern": "*", "name": "nvidia.com/gpu" } ] }, "sharing": { "timeSlicing": {}, "mps": { "failRequestsGreaterThanOne": true, "resources": [ { "name": "nvidia.com/gpu", "devices": "all", "replicas": 2 } ] } }, "imex": {} } I1110 10:03:43.913897 448393 main.go:196] Retrieving MPS daemons. E1110 10:03:43.914324 448393 main.go:83] error starting plugins: error getting daemons: failed to initialize NVML: ERROR_LIBRARY_NOT_FOUND

Thanks for your help!

2

*** Please note that this reply is generated by LLM automatically ***

I’m happy to help you with your issue, but I have to say that the provided context doesn’t seem to be directly related to your question about enabling MPS MultiProcess Service while installing the NVIDIA Device Plugin using Helm on a k3s cluster on a Jetson Orin Nano.

However, I can try to provide some general guidance on the topic.

MPS (MultiProcess Service) is a feature provided by NVIDIA that allows multiple processes to share the same GPU context, which can improve performance and reduce memory usage.

To enable MPS on a Jetson Orin Nano, you typically need to:

  1. Ensure that you are running a compatible version of the NVIDIA driver and JetPack.
  2. Configure the MPS service to start automatically on boot.
  3. Modify the NVIDIA device plugin configuration to enable MPS.

Here are some general steps that might help:

  1. Check the NVIDIA driver version: nvidia-smi --query-gpu=driver_version --format=csv
  2. Check the JetPack version: cat /etc/nv_tegra_release
  3. Configure MPS to start automatically on boot: sudo systemctl enable nvidia-mps-control.service
  4. Modify the NVIDIA device plugin configuration to enable MPS: sudo helm install --set mps.enabled=true nvdp

However, without more specific information about your k3s cluster and Jetson Orin Nano setup, it’s difficult to provide more detailed guidance.

Some questions that might help me better understand your issue:

  • What version of k3s are you running?
  • What version of the NVIDIA driver and JetPack are you running on your Jetson Orin Nano?
  • Can you provide more details about the error message you are seeing when trying to enable MPS?
  • Have you tried modifying the NVIDIA device plugin configuration to enable MPS?

Comparing to the provided context, I can see that there are some discussions about Jetson Orin Nano and NVIDIA containers, but they don’t seem to be directly related to your question about enabling MPS.

If you can provide more information about your setup and the error message you are seeing, I’ll do my best to help you resolve the issue.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4 
root@nano:~/model# nvidia-smi --query-gpu=driver_version --format=csv driver_version 540.4.0 root@k3s-master:~# kubectl get nodes NAME STATUS ROLES AGE VERSION k3s-master Ready control-plane,etcd,master 23h v1.33.5+k3s1 nano Ready <none> 23h v1.33.5+k3s1
6

Hi,

The library is not available on the Jetson platform.
Please see below for the details:

Thanks.

7

So is there any way to enable MPS on a k3s cluster, or not?

8

Hi,

We don’t have the information about MPS + k3s.
But MPS is supported on Jetson, while NVML is not supported.
If you need NVML, then it won’t work.

Thanks.