SOLVED: FreeBSD 12.2 ZFS: can only boot from disk, mirror, raidz1, raidz2, and raidz3 vdevs

SOLVED: FreeBSD 12.2 ZFS: can only boot from disk, mirror, raidz1, raidz2, and raidz3 vdevs

After upgrading a file server to FreeBSD 12.2 this last week we ran into an interesting issue where the boot loader immediately errored with the following message:

ZFS: can only boot from disk, mirror, raidz1, raidz2, and raidz3 vdevs
        vdev_init_from-nvlist: 5

It seemed to be a similar issue to this: https://reviews.freebsd.org/D20319 but not quite the same.

On reading more I ran across an article that described an issue where a ZFS array which formerly had ZIL (ZFS intenet log) disks but were removed left “hole” entries which the boot loader couldn’t handle.

Sure enough, that is what had happened on this array:

>>>> zdb -h zroot |grep zil
        2015-04-14.16:05:14 zpool create -f -m /mnt zroot raidz2 /dev/gpt/root0.nop /dev/gpt/root1.nop /dev/gpt/root2.nop /dev/gpt/root3.nop /dev/gpt/root4.nop /dev/gpt/root5.nop log /dev/gpt/zil0.nop /dev/gpt/zil1.nop
        2017-02-04.22:15:24 zpool remove zroot gpt/zil0
        2017-02-04.22:15:35 zpool remove zroot gpt/zil1

You can see the ‘hole’ entry via this as well at the bottom:

>>> zdb -C
        
        zroot:
            version: 5000
            name: 'zroot'
            state: 0
            txg: 38108897
            pool_guid: 10082749584082723291
            hostid: 780332240
            hostname: 'xxxxxxxxxx'
            com.delphix:has_per_vdev_zaps
            hole_array[0]: 1
            vdev_children: 2
            vdev_tree:
                type: 'root'
                id: 0
                guid: 10082749584082723291
                children[0]:
                    type: 'raidz'
                    id: 0
                    guid: 3591333108397056969
                    nparity: 2
                    metaslab_array: 36
                    metaslab_shift: 37
                    ashift: 12
                    asize: 48009336324096
                    is_log: 0
                    create_txg: 4
                    com.delphix:vdev_zap_top: 416
                    children[0]:
                        type: 'disk'
                        id: 0
                        guid: 2053196431247621779
                        path: '/dev/gpt/root0'
                        phys_path: 'id1,enc@n3061686369656d30/type@0/slot@1/elmdesc@Slot_00/p2'
                        whole_disk: 1
                        DTL: 53
                        create_txg: 4
                        com.delphix:vdev_zap_leaf: 52
                    children[1]:
                        type: 'disk'
                        id: 1
                        guid: 14128656964775907171
                        path: '/dev/gpt/root1'
                        phys_path: 'id1,enc@n3061686369656d30/type@0/slot@2/elmdesc@Slot_01/p2'
                        whole_disk: 1
                        DTL: 48
                        create_txg: 4
                        com.delphix:vdev_zap_leaf: 44
                    children[2]:
                        type: 'disk'
                        id: 2
                        guid: 6203050471626383213
                        path: '/dev/gpt/root2'
                        phys_path: 'id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02/p2'
                        whole_disk: 1
                        DTL: 50
                        create_txg: 4
                        com.delphix:vdev_zap_leaf: 49
                    children[3]:
                        type: 'disk'
                        id: 3
                        guid: 16517585206204717828
                        path: '/dev/gpt/root3'
                        phys_path: 'id1,enc@n3061686369656d30/type@0/slot@4/elmdesc@Slot_03/p2'
                        whole_disk: 1
                        DTL: 54
                        create_txg: 4
                        com.delphix:vdev_zap_leaf: 51
                    children[4]:
                        type: 'disk'
                        id: 4
                        guid: 4119283768170704307
                        path: '/dev/gpt/root4'
                        phys_path: 'id1,enc@n3061686369656d30/type@0/slot@5/elmdesc@Slot_04/p2'
                        whole_disk: 1
                        DTL: 57
                        create_txg: 4
                        com.delphix:vdev_zap_leaf: 55
                    children[5]:
                        type: 'disk'
                        id: 5
                        guid: 11827461088804713021
                        path: '/dev/gpt/root5'
                        phys_path: 'id1,enc@n3061686369656d30/type@0/slot@6/elmdesc@Slot_05/p2'
                        whole_disk: 1
                        DTL: 56
                        create_txg: 4
                        com.delphix:vdev_zap_leaf: 41
                children[1]:
                    type: 'hole'
                    id: 1
                    guid: 0
                    metaslab_array: 0
                    metaslab_shift: 24
                    ashift: 9
                    asize: 0
                    is_log: 0
                    is_hole: 1
            features_for_read:
                com.delphix:hole_birth
                com.delphix:embedded_data

So it seems there is an issue with FreeBSD 12.2’s /boot/loader.

The solution was to roll back the /boot/loader file from 12.1, and make sure the boot block was using 11.2 /boot/gptzfsboot — newer than that other errors were thrown.

We’ll work on getting this setup and reproducible on a VM so we can open a PR on this soon, but in the mean time hopefully this helps someone!