Kernel panic when trying to boot from encrypted NVME on TX2 NX without WI-FI module

I encountered weird issue when TX2 NX on DevKit carrier board refusing to boot into encrypted NVME when WiFi module removed (M2 key E module).
JetPack 4.6.6 and JetPack 4.6.1

Initially I created EKB key:
echo "00000000000000000000000000000000" > ekb.key

created user and password
sudo ./tools/l4t_create_default_user.sh -u nvidia -p nvidia -a --accept-license

Updated flash_l4t_nvme_rootfs_enc.xml for proper size of APP_ENC partition.

Flashed with this command
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1 -S 100GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only jetson-xavier-nx-devkit-tx2-nx nvme0n1p1

And additionally flashed EKS partition on the emmc
sudo ./tools/kernel_flash/l4t_initrd_flash.sh -k eks jetson-xavier-nx-devkit-tx2-nx mmcblk0p1
(got instruction from here Kernel panic when trying to boot from encrypted NVME - #3 by JerryChang)

And as long wifi module installed it works properly fine, but if wifi module removed or it was flashed without it (also tested on Auvidea JN30D, same story). It just crashing with this error from Debug UART console

Starting kernel ... [ 0.000000] Booting Linux on physical CPU 0x100 [ 0.000000] Linux version 4.9.337-tegra (buildbrain@mobile-u64-5497-d8000) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revisio4 [ 0.000000] Boot CPU: AArch64 Processor [411fd073] [ 0.000000] OF: fdt:memory scan node memory@80000000, reg size 80, [ 0.000000] OF: fdt: - 80000000 , 70000000 [ 0.000000] OF: fdt: - f0200000 , 85600000 [ 0.000000] OF: fdt: - 175e00000 , 200000 [ 0.000000] OF: fdt: - 176600000 , 200000 [ 0.000000] OF: fdt: - 177000000 , 200000 [ 0.000000] earlycon: uart8250 at MMIO32 0x0000000003100000 (options '') [ 0.000000] bootconsole [uart8250] enabled [ 0.000000] Found tegra_fbmem: 00800000@96088000 [ 0.000000] Found lut_mem: 00002008@96085000 [ 1.523049] imx219 9-0010: imx219_board_setup: error during i2c read probe (-121) [ 1.530673] imx219 9-0010: board setup failed [ 1.558679] imx219 10-0010: imx219_board_setup: error during i2c read probe (-121) [ 1.566353] imx219 10-0010: board setup failed [ 2.108967] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 [ 2.108967] [ 2.118135] CPU: 5 PID: 1 Comm: bash Not tainted 4.9.337-tegra #1 [ 2.124243] Hardware name: lanai-3636 (DT) [ 2.128348] Call trace: [ 2.130807] [< (ptrval)>] dump_backtrace+0x0/0x198 [ 2.136220] [< (ptrval)>] show_stack+0x24/0x30 [ 2.141288] [< (ptrval)>] dump_stack+0xa0/0xc4 [ 2.146353] [< (ptrval)>] panic+0x128/0x2a4 [ 2.151157] [< (ptrval)>] complete_and_exit+0x0/0x30 [ 2.156743] [< (ptrval)>] do_group_exit+0x40/0xa8 [ 2.162067] [< (ptrval)>] __wake_up_parent+0x0/0x40 [ 2.167566] [< (ptrval)>] el0_svc_naked+0x34/0x38 [ 2.172892] SMP: stopping secondary CPUs [ 2.176838] Kernel Offset: disabled [ 2.180333] Memory Limit: none [ 2.183394] trusty-log panic notifier - trusty version Built: 23:39:30 Nov 4 2024 [ 2.194908] Rebooting in 5 seconds.. 

hello vdoom.heretic,

it’s used sdmmc3: sdhci@3440000.
please try modify device tree to disable this node if you’re going to remove M.2 wifi module.
for instance,

diff --git a/kernel-dts/common/tegra186-p3636-0001-a00-comms.dtsi b/kernel-dts/common/tegra186-p3636-0001-a00-comms.dtsi @@ -19,7 +19,7 @@ uhs-mask = <0x10>; /* SDR104 */ only-1-8-v; /delete-property/ iommus; - status = "okay"; + status = "disabled"; }; bcm4354: bcmdhd_wlan { @@ -30,6 +30,6 @@ nv_path = "/lib/firmware/brcm/nvram.txt"; sdhci-host = <&sdmmc3>; pwr-retry-cnt = <3>; - status = "okay"; + status = "disabled"; }; 

Hello @JerryChang !
Somehow it didn’t help and the same error persists.

I reviewed documentation once again and found this picture.


(source NVIDIA Jetson Linux Developer Guide : Security | NVIDIA Docs )
In my understanding TX2 NX with JP 4.6.6 cannot use “Generic LUKS key“ (according to this Kernel panic when trying to boot from encrypted NVME - #15 by JerryChang), and I need to use ECID option. I see it says “per-device unique info from fuse“. Does it mean that I need to burn something to some fuses to make it work without M.2 wifi module?
And I don’t remember if I mentioned this before but this issue happens only when I try to boot from encrypted NVME. When bootting from encrypted EMMC it works fine with/without M.2 wifi module.

hello vdoom.heretic,

all right, it looks you’ve disk encryption enabled.
could you please have confirmation, what’s the scenario to reproduce the failure?

Hello @JerryChang
Yes, Here is the HW I used:

  • TX2 NX module.
  • Nvidia’s devkit’s carrier board from Xavier NX.
  • 128Gb M.2 NVME SSD.
  • Workstation with Ubuntu 18.04

Steps:

  1. Download and prepare L4T package with JetPack 4.6.6 on the workstation with SDKManager.
  2. Download and install “Secure boot” package into L4T ( Jetson Linux R32.7.6 | NVIDIA Developer )
  3. Download BSP sources and unpack “kernel_src” to Linux_for_Tegra/source/public folder
  4. Modified JetPack_4.6.6_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/source/kernel_src/hardware/nvidia/platform/t18x/lanai/kernel-dts/common/tegra186-p3636-0001-a00-comms.dtsi according to your post( Kernel panic when trying to boot from encrypted NVME on TX2 NX without WI-FI module - #3 by JerryChang ).
  5. Build device tree blobs and copied all of them to the Linux_for_Tegra/kernel/dtb folder.

I tried both options with two clean setups, but outcome was the same:

  • To build only dtbs:
# Navigate to kernel source cd Linux_for_Tegra/source/public/kernel/kernel-4.9/ # Configure for Tegra (this creates the .config file) make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- tegra_defconfig # Now build device trees make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- dtbs 
  • Setup GCC from Jetson Linux R32.7.6 package and build dtbs together with kernel with the single script nv_src_build.sh
  1. Update flash_l4t_nvme_rootfs_enc.xml for bigger SSD. (flash_l4t_nvme_rootfs_enc.xml.txt (9.1 KB))
  2. Create user
sudo ./tools/l4t_create_default_user.sh -u nvidia -p "nvidia" --accept-license 
  1. Create EKB key filled with ZEROES.
echo "00000000000000000000000000000000" > ekb.key 
  1. Flash NVME with ENCRYPTION enabled.
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1 -S 100GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only jetson-xavier-nx-devkit-tx2-nx nvme0n1p1 
  1. Flash Device Tree Blob partition
sudo ./flash.sh -r -k kernel-dtb jetson-xavier-nx-devkit-tx2-nx mmcblk0p1 
  1. Flash EKS partition
sudo ./tools/kernel_flash/l4t_initrd_flash.sh -k eks jetson-xavier-nx-devkit-tx2-nx mmcblk0p1 
  1. Configure UBoot to boot from NVME.

I tried different order for the EKS, DTB partition and full NVME flashing steps but the outcome was the same. EMMC was already flashed with ECRYPTION enabled.

hello vdoom.heretic,

your step-9 looks incorrect. please refer to developer guide, Disk Encryption Implementation in Jetson Linux.
please also check the commands to generate EKS image, Tool for EKB Generation.
that <sym2_key_file> is the LUKS key for disk encryption support. it’s the -i options in flash commands to assign key for EKB.
so, you should also assign the same disk encryption key to the flash command-line,
for instance, $ sudo ROOTFS_ENC=1 ./flash.sh -i ./ekb.key <board> <rootdev>

Hello @JerryChang

I made a new clean instalation, but problam still persists. What else should I try?

Steps:

  1. Download and prepare L4T package with JetPack 4.6.6 on the workstation with SDKManager.
  2. Download and install “Secure boot” package into L4T R32.7.6 ( Jetson Linux R32.7.6 | NVIDIA Developer )
  3. Download BSP sources and upacking “kernel_src” to Linux_for_Tegra/source/public folder
  4. Modified JetPack_4.6.6_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/source/kernel_src/hardware/nvidia/platform/t18x/lanai/kernel-dts/common/tegra186-p3636-0001-a00-comms.dtsi according to your post( Kernel panic when trying to boot from encrypted NVME on TX2 NX without WI-FI module - #3 by JerryChang ).
  5. Build device tree blobs and copied all of them to the Linux_for_Tegra/kernel/dtb folder.

I tried both options with two clean setups:

  • To build only dtbs:
# Navigate to kernel source cd Linux_for_Tegra/source/public/kernel/kernel-4.9/ # Configure for Tegra (this creates the .config file) make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- tegra_defconfig # Now build device trees make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- dtbs 
  • Setup GCC from Jetson Linux R32.7.6 package and build dtbs together with kernel with the single scipt nv_src_build.sh
  1. Update flash_l4t_nvme_rootfs_enc.xml for bigger SSD. (

    flash_l4t_nvme_rootfs_enc.xml.txt (9.1 KB)

    )

  2. Create user

sudo ./tools/l4t_create_default_user.sh -u nvidia -p "nvidia" --accept-license 
  1. Create EKB key filled with ZEROES.
echo "00000000000000000000000000000000" > ekb.key 
  1. Flash EMMC with ENCRYPTION enabled
sudo ROOTFS_ENC=1 ./flash.sh -i "./ekb.key" jetson-xavier-nx-devkit-tx2-nx mmcblk0p1 
  1. Flash NVME with ENCRYPTION enabled.
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1 -S 100GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only jetson-xavier-nx-devkit-tx2-nx nvme0n1p1 
  1. Flash Device Tree Blob partition
sudo ./flash.sh -r -k kernel-dtb jetson-xavier-nx-devkit-tx2-nx mmcblk0p1 
  1. Flash EKS partition
sudo ./tools/kernel_flash/l4t_initrd_flash.sh -k eks jetson-xavier-nx-devkit-tx2-nx mmcblk0p1 
  1. Configure UBoot to boot from NVME.
  2. Tried to repack initrd (https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3276/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/bootloader_disk_encryption.html#wwpID0E0HE0HA). inird in the boot partition on NVME already had UUID of encrypted partition of the same disk. I tried to add “crypt_root UUID” of encrypted partition of NVME to initrd from the EMMC /boot partition, but after that it just stop booting at some point.
  3. I also tried to generate eks.img with gen_ekb.py, placed it to Linux_for_Tegra/bootloader folder and tried to reflash EKS partition with command sudo ./flash.sh -r -k eks jetson-xavier-nx-devkit-tx2-nx mmcblk0p1. But this is also didn’t help. But I noticed that eks.img generated by gen_ekb.py was different from eks.img generated by flash.sh after flashing EMMC. In both cases I was using the same zero filled key. sym2.key was “00000000000000000000000000000000“.

hello vdoom.heretic,

let’s check whether it’s related to disk encryption, or wifi modules.
for instance, is it able to boot from NVME without WI-FI module?

When it’s flashed to NVME without encryption, it boots and working. With and without wifi module

hello vdoom.heretic,

do you have other M.2 NVMe SSD hardware for quick testing?

I already tried patriot 256gb nvme ssd and samsung 128gb with original Xavier nx carrier board and chinese copy of the carrier board from waveshark (jetson-io-base-b) with the same outcome. Only difference that when I am flashing 256gb disk with config for 128gb it take some extra step to fill disk with zeroes, which takes extra 2-3 hours.

I also tried to flash nvme with encryption with wifi module plugged in and without.

If wifi module plugged in during flashing, and stays there during booting - it boots. If wifi module unplugged after flashing- it shows error during boot process and restarts.

If wifi module unplugged before flashing - it show the same error during booting. Plugging wifi module back in doesn’t solve the issue in this case, and issues remains even if wifi module plugged back after the flashing.

hello vdoom.heretic,

is there strong request by removing wifi module?
according to JetPack 4 Reaches End of Life, we may not dig into this topic to resolve the bugs.

Hello @JerryChang

Well, too bad. Probably I will consider different options then. But just in case, if you have any hints where and what I can investigate regarding this problem, please let me know.

I made some more testings and seems like this issue doesn’t happend with XavierNX on the same carrier board with the same JP4.6.6.