Cannot boot from Sandisk Corp Western Digital WD Black SN850X NVMe SSD (rev 01)

[stevefalco]

Describe the bug

I have a Pimoroni NVMe base connected to my Pi 5. If I use a WD_BLACK SN770 2TB the system boots, but if I use a WD_BLACK SN850X 2000GB it does not boot. I've attached three files: bad.txt shows the unsuccessful boot of the SN850X,
good.txt shows the successful boot of the SN770, and log.txt shows more diagnostics of the SN850X after booting from a separate USB thumb drive.

Please note that I had to add a 32.768 kHz oscillator to the SUSCLK pin of the M.2 connector of the Pimoroni board, because without that the PCIe link of the SN850X would never come up. (Pin 68).

In bad.txt at line 83 we see the drive, and the MBR shows the boot and root partitions. In good.txt we see the same at line 78.

For both drives, the vfat partition is clearly readable by the bootrom, because we can see the kernel being started. Additionally we can confirm that the vfat partition was read because the kernel command line in bad.txt at line 146 shows the root partition ID is PARTUUID=ee2da036-02. Similarly good.txt at line 141 shows the ID is PARTUUID=8ca68f5e-02.

At bad.txt line 266 we see the rootfs image being unpacked. And good.txt at line 261 is the same.

The trouble first shows up at line 344 in bad.txt, because the pcie link is reported as "down". In contrast, line 339 in good.txt shows the link is up.

It seems to me that the bootrom brings the link up and can read from the SN850X drive, but apparently the bootrom doesn't leave the interface in a state whereby the kernel can use it.

If I boot from USB, the SN850X drive is usable. See file log.txt. At line 81 the "lspci -vv" shows that Linux can see the drive, and at line 286 "lsblk" sees the partitions. I am able to mount and list the contents.

bad.txt
good.txt
log.txt

Steps to reproduce the behaviour

Connect a WD_BLACK SN850X 2000GB to a Pi 5 using a Pimoroni base. You will have to "white wire" a 32.768 kHz oscillator to pin 68 (SUSCLK) of the M.2 connector for the SN850X to turn on.

Try to boot and observe that the vfat partition is readable by the bootloader but Linux won't be able to mount the root filesystem.

Boot from an SD card or USB thumb drive, and you will be able to mount the SN850X.

Device (s)

Raspberry Pi 5

Bootloader configuration.

# rpi-eeprom-config
[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf146
NET_INSTALL_AT_POWER_ON=1

System

raspi-info.txt

Bootloader logs

Please see the files attached earlier (bad.txt, good.txt, log.txt). If you need additional informat please let me know.

USB boot

N/A

NVMe boot

# nvme list
Node                  Generic               SN                   Model                                    Namespace Usage                      Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            24204D800091         WD_BLACK SN850X 2000GB                   1           2.00  TB /   2.00  TB    512   B +  0 B   620361WD

id-ctrl.txt

# nvme list-ns /dev/nvme0
[   0]:0x1

id-ns.txt

Network (TFTP boot)

N/A

[bootc]

I've just run into a problem that sounds very similar indeed, with some differences in my setup:

• I'm using a 1TB SN850X
• I've got a CM5 WiFi / 8GB RAM / No eMMC + IO Board
• I didn't have to do a wire mod to have it show up in Raspberry Pi OS

I can install an OS using the Imager running from RPi OS booted from an SD card just fine.

I can't boot Ubuntu 24.04 or RPi OS from the NVMe.

I also can't boot Ubuntu 24.04 from an SD card either, with similar symptoms: the SD card isn't detected during boot. However, if I boot Ubuntu 24.04 from the SD card, the NVMe is detected.

Let me know if you'd like my logs for comparison.

[bootc]

For completeness, I have now managed to successfully boot Ubuntu 24.04 by writing the same OS to both the NVMe and an SD card. With this setup the CM5 firmware boots off the SD and loads the kernel and initramfs from there. The kernel then successfully mounts the root filesystem from the NVMe and carries on. This is in no way ideal as I can't now update the kernel or initramfs, I merely mention it to illustrate the functionality.

[stevefalco]

That does sound like the same problem. The reason I had to add the low-frequency oscillator is because the SN850X needs it for some sleep modes, and the Pimoroni base doesn't come with that oscillator. But other boards do - the official one from RPi includes a low-frequency oscillator, for example.

I've replaced the Pi 5 with a Beelink S12 Pro Mini, which boots perfectly from the SN850X. I also have an SN850Xin my Asus desktop machine as the boot drive, and it works properly there too.

Conclusion: apparently the Pi 5 F/W is missing something fairly standard for NVMe interfaces.

[lurch]

I also can't boot Ubuntu 24.04 from an SD card either... However, if I boot Ubuntu 24.04 from the SD card...

Huh? 🙃

Obviously this will be much easier for us to diagnose if you test using Raspberry Pi OS 🙂 Sometimes Ubuntu lags behind with software / firmware / kernel updates, which makes diagnosis much more tricky.

[bootc]

I also can't boot Ubuntu 24.04 from an SD card either... However, if I boot Ubuntu 24.04 from the SD card...

Huh? 🙃

Obviously this will be much easier for us to diagnose if you test using Raspberry Pi OS 🙂 Sometimes Ubuntu lags behind with software / firmware / kernel updates, which makes diagnosis much more tricky.

If I boot Raspberry Pi OS from the SD card it works fine and can see both the SD and NVMe just fine.

[pelwell]

[ Off topic, but thank's for all those kernel drivers at the start of Raspberry Pi, Chris! ]