FreeBSD / pfSense on a Zotac ZBox: link state down and ‘re0: watchdog timeout’ errors

FreeBSD / pfSense on a Zotac ZBox: link state down and ‘re0: watchdog timeout’ errors

Recently we purchased a Zotac ZBOX-CI321NANO to replace our existing pfSense PC which was in a much larger case.

There are a few variations but ours has dual 1 Gbps NICs, many USB 3.0 ports, 2 GiB of RAM, a 32 GiB SSD and is powered by a Celeron 2961Y (1.10 GHz). It has plenty of horsepower to spare running as a router/firewall for an office and is positively tiny at roughly 5″ x 5″ and 2″ tall. It is the perfect pfSense box and looks great on our equipment rack.

However after a day in service we started noticing drops on one of its interfaces, the internal one to be specific, which had a lot of VLANs:

re1: watchdog timeout
        re1: link state changed to DOWN
        re1_vlan1: link state changed to DOWN
        re1_vlan3: link state changed to DOWN
        re1_vlan4: link state changed to DOWN
        re1_vlan5: link state changed to DOWN
        re1_vlan6: link state changed to DOWN
        re1_vlan10: link state changed to DOWN
        re1: link state changed to UP
        re1_vlan1: link state changed to UP
        re1_vlan3: link state changed to UP
        re1_vlan4: link state changed to UP
        re1_vlan5: link state changed to UP
        re1_vlan6: link state changed to UP
        re1_vlan10: link state changed to UP

Our outside interface (in our case re0) was not affected. We tried turning off MSI and MSIX via the following sysctls:

hw.re.msi_disable=1
        hw.pci.enable_msix=0
        hw.pci.enable_msi=0

However the timeouts persisted as did the link drops.

What did work was disabling hardware checksum offloading:

Interestingly this did not turn off the VLAN_HWCSUM option though in ifconfig:

re1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
                options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
                ether 00:01:2e:xx:xx:xx
                nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                media: Ethernet autoselect (1000baseT <full-duplex>)
                status: active

But it did solve the issue!