:How do I install the NVMe driver for a custom image?

Non-Volatile Memory Express (NVMe) is a high-speed interface protocol designed for solid-state storage, such as flash-based SSDs. It provides faster response times and higher bandwidth than traditional storage protocols such as SCSI and virtio-blk.

Procedure

1. Create an ECS instance from your existing custom image. This instance serves as an intermediate environment. Then, remotely connect to the instance.

   Important

   You are charged for the intermediate instance that you create. After the new custom image is created, we recommend that you promptly release the instance to avoid ongoing charges.

2. Check for the NVMe driver and install it based on the operating system of the instance.

   Automatic configuration using Cloud Assistant

   Cloud Assistant provides the ecs_nvme_config plugin to help you quickly configure NVMe drives. However, the plugin has operating system limitations.

   ecs_nvme_config Plugin: Supported Operating Systems

   • Operating systems supported by the ecs_nvme_config plugin:

     • Alibaba Cloud Linux

     • Anolis OS

     • CentOS 6.6 or later

     • CentOS Stream

     • Debian 9 or later

     • Ubuntu 16 or later

     • openSUSE 42 or later

     • SUSE Linux Enterprise Server 11.4 or later

     • Red Hat Enterprise Linux

     • Fedora

     • Rocky Linux

     • AlmaLinux

   1. Run the following command to check whether the Cloud Assistant Agent is installed on the instance and whether the ecs_nvme_config plugin is available in the client.

      acs-plugin-manager --list

      

      • If no Cloud Assistant information is returned, you must first install the Cloud Assistant Agent on the instance.

      • If a list of Cloud Assistant plugins is returned and includes the ecs_nvme_config plugin, proceed to the next step.

   2. Use the ecs_nvme_config plugin to complete the NVMe configuration.

      1. Run the following command to use the ecs_nvme_config plugin to check whether an NVMe module exists on the current instance and whether it can be configured.

         sudo acs-plugin-manager --exec --plugin ecs_nvme_config --params --check

         • If the following message is returned, the NVMe driver is already installed on the instance. You do not need to perform any NVMe-related configuration and can create a custom image directly.

           [SUCCESS] Summary: Your image can run on an NVMe instance.

         • If the result contains a message similar to the following [ERROR], you need to proceed to the next step.

           [ERROR] 1.nvme module missing from initrd/initramfs, please run acs-plugin-manager --exec --plugin ecs_nvme_config --params -f/--fix to enable nvme;

      2. Based on the message in the check result, configure the NVMe driver and related parameters.

         sudo acs-plugin-manager --exec --plugin ecs_nvme_config --params --fix

      3. After the configuration is complete, run the following command to restart the instance.

         sudo reboot

      4. After restarting the instance, check whether the NVMe module is configured successfully.

         sudo acs-plugin-manager --exec --plugin ecs_nvme_config --params --check

         The following is a sample success response:

         [OK] 1.initrd/initramfs already contains the nvme module; [OK] 2.fstab file looks fine and does not contain any device names; [OK] 3.The nvme parameters are already included. [SUCCESS] Summary: Your image can run on an nvme instance

   Manual configuration (CentOS/Alibaba Cloud Linux)

   1. Run the following command to check whether the system kernel has loaded the NVMe driver.

      cat /boot/config-`uname -r` | grep -i nvme | grep -v "^#"

      

      • If the result is CONFIG_BLK_DEV_NVME=y, the operating system can boot directly on instance families that support the NVMe protocol. You can then proceed to automatic configuration using Cloud Assistant.

      • If the result is CONFIG_BLK_DEV_NVME=m, perform the following steps.

   2. Run the following command to check whether the initial RAM file system (initramfs) contains the NVMe driver.

      sudo lsinitrd /boot/initramfs-`uname -r`.img | grep -i nvme | awk '{print $NF}'

      

      • If a result similar to the one in the figure is returned, the operating system can start on an instance from an instance family that supports the NVMe protocol. You can proceed directly to step d.

      • If no such result is returned, proceed to the next step (step c).

   3. Run the following command to add NVMe driver support to initramfs.

      mkdir -p /etc/dracut.conf.d echo 'add_drivers+=" nvme nvme-core "' | sudo tee /etc/dracut.conf.d/nvme.conf > /dev/null sudo dracut -v -f

      Note

      If the dracut tool is not installed on your operating system, run sudo yum -y install dracut to install the tool.

   4. Add the NVMe-related io_timeout parameter to the GRand Unified Bootloader (GRUB).

      Note

      Adding the io_timeout parameter to the GRand Unified Bootloader (GRUB) helps prevent timeout faults related to NVMe devices in Linux. Setting this parameter to the maximum value prevents the system from failing when it processes I/O requests for NVMe devices.

      In most Linux distributions, the io_timeout parameter is set to 30 seconds by default. Before you set the io_timeout parameter, you need to run the echo 4294967295 > /sys/module/nvme_core/parameters/io_timeout or echo 4294967295 > /sys/module/nvme/parameters/io_timeout command to verify whether the io_timeout parameter can be set to the maximum value of 4,294,967,295 seconds. If the command returns -bash: echo: write error: Invalid argument, this indicates that your current Linux version supports a maximum value of only 255 seconds.

      Method 1: Use the grubby command

      1. Check whether the system supports the grubby command.

         which grubby

         • If the output shows the path for grubby (such as /usr/sbin/grubby), the system supports grubby. Proceed to the next step.

         • If no output is returned, the system does not support grubby. To add the parameters, see Method 2.

      2. Use the grubby command to add kernel parameters.

         sudo grubby --update-kernel=ALL --args="nvme_core.io_timeout=4294967295 nvme_core.admin_timeout=4294967295"

      Method 2: Modify the GRUB configuration file

      1. Run the following command to open the grub file.

         sudo vi /etc/default/grub

      2. Press the i key to enter edit mode. In the GRUB_CMDLINE_LINUX= line, add the nvme_core.io_timeout=4294967295 nvme_core.admin_timeout=4294967295 parameters.

         After you add the parameters, the file content looks similar to the following:

         

         Note

         If the configuration file already contains these parameters, you do not need to add them again.

      3. Press the Esc key to exit edit mode, enter :wq, and press the Enter key to save and exit the file.

      4. Run the following command to apply the GRUB configuration.

         Depending on the instance startup mode of the ECS instance, run the corresponding command:

         • Legacy BIOS boot mode

           sudo grub2-mkconfig -o /boot/grub2/grub.cfg

         • UEFI boot mode

           1. View the grub configuration file.

              cat /boot/efi/EFI/centos/grub.cfg

           2. Run the corresponding command based on the content of the grub configuration file.

              • If the configuration file content specifies another path for configfile, such as /boot/grub/grub.cfg as shown in the following figure, run the following command to apply the configuration.

                sudo grub2-mkconfig -o /boot/grub2/grub.cfg

              • Otherwise, run the following command to apply the configuration.

                sudo grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

   5. Run the following command again to verify that the NVMe driver is included in the initramfs.

      sudo lsinitrd /boot/initramfs-`uname -r`.img | grep -i nvme | awk '{print $NF}'

      If a result similar to the one in the following figure is returned, the configuration is complete. The operating system can now start on an instance from an instance family that supports the NVMe protocol.

      

   Manual configuration (Ubuntu/Debian)

   1. (Optional) You can run the following command to view the NVMe drivers included in the initrd.

      lsinitramfs /boot/initrd.img-`uname -r` | grep -i nvme

      The following output shows that the NVMe driver is loaded by default in the initrd for Ubuntu operating systems.

   2. Add the NVMe-related io_timeout parameter to GRUB.

      Note

      The primary purpose of adding the NVMe-related io_timeout parameter to the GRand Unified Bootloader (GRUB) is to prevent timeout faults related to NVMe devices in Linux. Setting the timeout value to the maximum prevents the operating system from failing due to a timeout when it processes I/O requests from NVMe devices.

      In most Linux distributions, the io_timeout parameter is set to 30 seconds by default. Before you set the io_timeout parameter, run the echo 4294967295 > /sys/module/nvme_core/parameters/io_timeout or echo 4294967295 > /sys/module/nvme/parameters/io_timeout command to verify whether the io_timeout parameter can be set to the maximum value of 4,294,967,295 seconds. If the command returns -bash: echo: write error: Invalid argument, this indicates that your current Linux version supports a maximum value of only 255 seconds.

      1. Run the following command to open the /etc/default/grub file.

         sudo vi /etc/default/grub

      2. Press the i key to enter edit mode. In the GRUB_CMDLINE_LINUX= line, add the nvme_core.multipath=n nvme_core.io_timeout=4294967295 nvme_core.admin_timeout=4294967295 parameters.

         After you add the parameters, the file content looks similar to the following:

         Note

         If the configuration file already contains these parameters, you do not need to add them again.

      3. Press Esc to exit edit mode, enter :wq, and press Enter to save and exit the file.

   3. Run the following command to apply the GRUB configuration.

      Depending on the instance boot mode of your ECS instance, run one of the following commands:

      • Any boot mode (This command applies only to Ubuntu/Debian systems)

        sudo update-grub2

      • Legacy BIOS boot mode

        sudo grub-mkconfig -o /boot/grub/grub.cfg

      • UEFI boot mode

        sudo grub-mkconfig -o /boot/efi/EFI/debian/grub.cfg

3. After you install the driver on the instance, create a new custom image from the instance. Then, modify the properties of the new custom image and set the NVMe Driver property to Supported.

4. (Optional) Redeploy your services by creating an instance from the new custom image that supports NVMe, as described in Create an instance using a custom image. When you create the ECS instance, select an instance type that supports NVMe.

References

• You can attach only Enterprise SSDs (ESSDs) and ESSD AutoPL disks to NVMe-based ECS instance types. You can enable the Multi-attach for cloud disks feature for these disks to share data by attaching them to multiple ECS instances.

• If the io_timeout parameter of an NVMe disk on an ECS instance is improperly configured, the disk may become unavailable. For more information about how to resolve this issue, see What do I do if an NVMe disk on a Linux ECS instance is unavailable due to an improperly configured I/O timeout parameter?