Running ZFS on a Raspberry Pi offers powerful features for home servers and personal NAS setups. But with power comes complexity, and in recent months, a quiet pitfall has emerged for Pi users who track kernel updates closely: upgrading to an unsupported kernel version breaks ZFS.

⚠️ Important Warning: If you've just upgraded your kernel and you're using ZFS, do not reboot yet. Rebooting without a working ZFS module can leave your system unbootable—even if you don't have critical partitions on ZFS. The system may hang because it tries to load the missing ZFS kernel module during boot and fails, halting the boot process.

If you're already facing a broken setup or can't boot — don't panic. This guide outlines both the worst-case scenario and the ideal recovery strategy. Whether you're locked out or preparing for a smooth upgrade, we've got you covered.

The Problem: A Kernel Update Too Far

As of ZFS version 2.2.3 (used in Debian-based Raspberry Pi OS), the filesystem supports Linux kernels up to 6.7. However, Raspberry Pi OS backports recently began shipping 6.12.x kernels. If you upgrade to 6.12 without precautions:

• ZFS will fail to compile against the new kernel
• DKMS errors will appear during package updates
• Your ZFS module will be missing after reboot
• Your Raspberry Pi may fail to boot, even without ZFS-mounted root or critical paths, simply due to systemd or boot scripts expecting the kernel module to be present

Example ZFS Compilation Error:

During package updates or installs, you'll see something like:

checking whether bdev_open_by_path() exists... configure: error:
        *** None of the expected "blkdev_get_by_path()" interfaces were detected.
        *** This may be because your kernel version is newer than what is
        *** supported, or you are using a patched custom kernel with
        *** incompatible modifications.
        ***
        *** ZFS Version: zfs-2.2.3-1~bpo12+1~rpt1
        *** Compatible Kernels: 3.10 - 6.7

This error means ZFS cannot build its kernel module against Linux 6.12.x, making it unusable.

Why This Happens: The Version Trap

The key problem is that ZFS 2.2.3 remains installed even after a kernel upgrade, and it doesn't get upgraded automatically during apt upgrade. Since ZFS 2.2.3 only supports up to kernel 6.7, it fails to compile against 6.12.x.

However, if you remove and then reinstall ZFS, the package manager installs ZFS 2.3.1, which does support Linux kernel 6.12.x. This version transition is what resolves the issue — but only if you explicitly purge and reinstall ZFS.

Worst Case: The System That Wouldn’t Boot

After upgrading the kernel and rebooting, the system failed to boot properly. It showed errors like:

cannot open access to console, the root account is locked

Although the system did not have critical filesystems on ZFS, the boot process still stalled because system services attempted to load the ZFS kernel module, which was no longer available. This resulted in an unrecoverable boot failure.

The only way forward was to reformat and reinstall Raspberry Pi OS. However, the default Raspberry Pi OS image still comes with a 6.6.x kernel, which is incompatible with ZFS 2.3.1 and newer kernels unless upgraded. Therefore, the recovery process requires one crucial step:

1. First, perform a full system upgrade:

   sudo apt update && sudo apt full-upgrade
   

   This brings the kernel up to 6.12.x.

2. Then, reinstall ZFS:

   sudo apt install zfs-dkms zfsutils-linux
   

Once this is complete, your system will be running kernel 6.12 with ZFS 2.3.1, and you can safely import your pools and resume operations.

Best Case Recovery: Clean Cut, Clean Upgrade

For users who can still log into their system, here's the cleanest and safest sequence to move forward:

1. Stop all services using ZFS, including Docker, NFS, Samba, backup tools, etc.

2. Export all ZFS pools:

   sudo zpool export -a
   

3. Disable swap if it lives on a ZFS vdev:

   sudo swapoff /dev/sdXn
   

4. Purge ZFS packages:

   sudo apt purge zfsutils-linux zfs-dkms zfs-zed
   sudo rm -rf /usr/src/zfs* /var/lib/dkms/zfs
   

5. Update the kernel to the desired version:

   sudo apt update && sudo apt full-upgrade
   

6. Reboot into the new kernel:

   sudo reboot
   

7. Reinstall ZFS:

   sudo apt install zfs-dkms zfsutils-linux
   

8. Import your pool(s):

   sudo zpool import poolname
   

9. Restart services that depend on ZFS.

Final Notes: Prevention Is Better Than Recovery

To avoid this issue in the future:

• Hold your current working kernel version:

  sudo apt-mark hold linux-image-rpi-v8 linux-headers-rpi-v8
  

• Or track ZFS GitHub for kernel compatibility before upgrading

• Or test upgrades on a second Pi or cloned SD card before rolling them out to production

Conclusion: A Solvable Trap

ZFS on the Raspberry Pi remains a powerful option, but it demands careful version tracking. If you upgrade responsibly, or recover cleanly as described above, you can continue benefiting from advanced features like snapshots, send/receive, and compression even on this tiny powerhouse.

Don’t let a kernel update ruin your storage plans—with preparation, the Pi + ZFS combo can remain stable and strong.

• Hardware: Raspberry Pi 5 with PCIe NVMe HAT and 2TB NVMe SSD
• Filesystem: ZFS with separate datasets for each service
• Networking: Docker bridge networks for service segmentation
• Privacy: Tor and I2P routing for anonymous communication
• Public Access: Cloudflare Tunnel to securely expose LNbits

📊 Architecture Diagram

🛠️ Setup Steps

1. Prepare the System

• Install Raspberry Pi OS (64-bit)
• Set up ZFS on the NVMe disk
• Create a ZFS dataset for each service (e.g., bitcoin, lnd, rtl, lnbits, tor-data)
• Install Docker and Docker Compose

2. Create Shared Docker Network and Privacy Layers

Create a shared Docker bridge network:

docker network create \
  --driver=bridge \
  --subnet=192.168.100.0/24 \
  bitcoin-net

Note: Connect bitcoind, lnd, rtl, internal lnbits, tor, and i2p to this bitcoin-net network.

Tor

• Run Tor in a container
• Configure it to expose LND's gRPC and REST ports via hidden services:

  HiddenServicePort 10009 192.168.100.31:10009
  HiddenServicePort 8080  192.168.100.31:8080
  

• Set correct permissions:

  sudo chown -R 102:102 /zfs/datasets/tor-data
  

I2P

• Run I2P in a container with SAM and SOCKS proxies
• Update bitcoin.conf:

  i2psam=192.168.100.20:7656
  i2pacceptincoming=1
  

3. Set Up Bitcoin Core

• Create a bitcoin.conf with Tor/I2P/proxy settings and ZMQ enabled
• Sync the blockchain in a container using its ZFS dataset

4. Set Up LND

• Configure lnd.conf to connect to bitcoind and use Tor:

  [Bitcoind]
  bitcoind.rpchost=bitcoin:8332
  bitcoind.rpcuser=bitcoin
  bitcoind.rpcpass=very-hard-password
  bitcoind.zmqpubrawblock=tcp://bitcoin:28332
  bitcoind.zmqpubrawtx=tcp://bitcoin:28333
  
  [Application Options]
  externalip=xxxxxxxx.onion
  

• Don’t expose gRPC or REST ports publicly
• Mount the ZFS dataset at /root/.lnd
• Optionally enable Watchtower

5. Set Up RTL

• Mount RTL-Config.json and data volumes
• Expose RTL's web interface locally:

  ports:
    - "3000:3000"
  

6. Set Up Internal LNbits

• Connect the LNbits container to bitcoin-net
• Mount the data directory and LND cert/macaroons (read-only)
• Expose the LNbits UI on the local network:

  ports:
    - "5000:5000"
  

• In the web UI, configure the funding source to point to the LND REST .onion address and paste the hex macaroon
• Create and fund a wallet, and copy its Admin Key for external use

7. Set Up External LNbits + Cloudflare Tunnel

• Run another LNbits container on a separate Docker network
• Access the internal LNbits via the host IP and port 5000
• Use the Admin Key from the internal wallet to configure funding
• In the Cloudflare Zero Trust dashboard:
  • Create a tunnel
  • Select Docker, copy the --token command
  • Add to Docker Compose:

    command: tunnel --no-autoupdate run --token eyJ...your_token...
    

💾 Backup Strategy

• Bitcoin Core: hourly ZFS snapshots, retained for 6 hours
• Other Services: hourly snapshots with remote .tar.gz backups
  • Retention: 7d hourly, 30d daily, 12mo weekly, monthly forever
• Back up ZFS snapshots to avoid inconsistencies

🔐 Security Isolation Benefits

This architecture isolates services by scope and function:

• Internal traffic stays on bitcoin-net
• Sensitive APIs (gRPC, REST) are reachable only via Tor
• Public access is controlled by Cloudflare Tunnel

Extra Security: Host the public LNbits on a separate machine (e.g., hardened VPS) with strict firewall rules:

• Allow only Cloudflare egress
• Allow ingress from your local IP
• Allow outbound access to internal LNbits (port 5000)

Use WireGuard VPN to secure the connection between external and internal LNbits:

• Ensures encrypted communication
• Restricts access to authenticated VPN peers
• Keeps the internal interface isolated from the public internet

✅ Final Notes

• Internal services communicate over bitcoin-net
• LND interfaces are accessed via Tor only
• LNbits and RTL UIs are locally accessible
• Cloudflare Tunnel secures external access to LNbits

Monitor system health using monit, watchtower, or Prometheus.

Create all configuration files manually (bitcoin.conf, lnd.conf, RTL-Config.json), and keep credentials secure. Test every component locally before exposing it externally.

⚡