SOLVED: FreeBSD 12.2 ZFS: can only boot from disk, mirror, raidz1, raidz2, and raidz3 vdevs

After upgrading a file server to FreeBSD 12.2 this last week we ran into an interesting issue where the boot loader immediately errored with the following message:

ZFS: can only boot from disk, mirror, raidz1, raidz2, and raidz3 vdevs
vdev_init_from-nvlist: 5

It seemed to be a similar issue to this: https://reviews.freebsd.org/D20319 but not quite the same.

On reading more I ran across an article that described an issue where a ZFS array which formerly had ZIL (ZFS intenet log) disks but were removed left “hole” entries which the boot loader couldn’t handle.

Sure enough, that is what had happened on this array:

>>>> zdb -h zroot |grep zil
2015-04-14.16:05:14 zpool create -f -m /mnt zroot raidz2 /dev/gpt/root0.nop /dev/gpt/root1.nop /dev/gpt/root2.nop /dev/gpt/root3.nop /dev/gpt/root4.nop /dev/gpt/root5.nop log /dev/gpt/zil0.nop /dev/gpt/zil1.nop
2017-02-04.22:15:24 zpool remove zroot gpt/zil0
2017-02-04.22:15:35 zpool remove zroot gpt/zil1

You can see the ‘hole’ entry via this as well at the bottom:

>>> zdb -C

zroot:
    version: 5000
    name: 'zroot'
    state: 0
    txg: 38108897
    pool_guid: 10082749584082723291
    hostid: 780332240
    hostname: 'xxxxxxxxxx'
    com.delphix:has_per_vdev_zaps
    hole_array[0]: 1
    vdev_children: 2
    vdev_tree:
        type: 'root'
        id: 0
        guid: 10082749584082723291
        children[0]:
            type: 'raidz'
            id: 0
            guid: 3591333108397056969
            nparity: 2
            metaslab_array: 36
            metaslab_shift: 37
            ashift: 12
            asize: 48009336324096
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 416
            children[0]:
                type: 'disk'
                id: 0
                guid: 2053196431247621779
                path: '/dev/gpt/root0'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@1/elmdesc@Slot_00/p2'
                whole_disk: 1
                DTL: 53
                create_txg: 4
                com.delphix:vdev_zap_leaf: 52
            children[1]:
                type: 'disk'
                id: 1
                guid: 14128656964775907171
                path: '/dev/gpt/root1'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@2/elmdesc@Slot_01/p2'
                whole_disk: 1
                DTL: 48
                create_txg: 4
                com.delphix:vdev_zap_leaf: 44
            children[2]:
                type: 'disk'
                id: 2
                guid: 6203050471626383213
                path: '/dev/gpt/root2'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02/p2'
                whole_disk: 1
                DTL: 50
                create_txg: 4
                com.delphix:vdev_zap_leaf: 49
            children[3]:
                type: 'disk'
                id: 3
                guid: 16517585206204717828
                path: '/dev/gpt/root3'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@4/elmdesc@Slot_03/p2'
                whole_disk: 1
                DTL: 54
                create_txg: 4
                com.delphix:vdev_zap_leaf: 51
            children[4]:
                type: 'disk'
                id: 4
                guid: 4119283768170704307
                path: '/dev/gpt/root4'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@5/elmdesc@Slot_04/p2'
                whole_disk: 1
                DTL: 57
                create_txg: 4
                com.delphix:vdev_zap_leaf: 55
            children[5]:
                type: 'disk'
                id: 5
                guid: 11827461088804713021
                path: '/dev/gpt/root5'
                phys_path: 'id1,enc@n3061686369656d30/type@0/slot@6/elmdesc@Slot_05/p2'
                whole_disk: 1
                DTL: 56
                create_txg: 4
                com.delphix:vdev_zap_leaf: 41
        children[1]:
            type: 'hole'
            id: 1
            guid: 0
            metaslab_array: 0
            metaslab_shift: 24
            ashift: 9
            asize: 0
            is_log: 0
            is_hole: 1
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

So it seems there is an issue with FreeBSD 12.2’s /boot/loader.

The solution was to roll back the /boot/loader file from 12.1, and make sure the boot block was using 11.2 /boot/gptzfsboot — newer than that other errors were thrown.

We’ll work on getting this setup and reproducible on a VM so we can open a PR on this soon, but in the mean time hopefully this helps someone!