Tuesday, June 15, 2021


SOLVED: linker_load_file: /boot/modules/openzfs.ko – unsupported file type

While trying to recover and upgrade a FreeBSD install we ran into an issue where the kernel loader would hang when loading openzfs.ko: linker_load_file: /boot/modules/openzfs.ko – unsupported file type In this case /boot/loader.conf had auto-importing enabled via this: vfs.zfs.autoimport_disable=0 The default is 1, or disable auto import, since at least 2014 according to this commit.… Read more »

Saturday, May 15, 2021


SOLVED: Magento / PHP datefmt_create: no such time zone: ‘UTC’: U_ILLEGAL_ARGUMENT_ERROR

After upgrading a host’s operating system we ran into this error on a Magento site: There has been an error processing your request datefmt_create: no such time zone: ‘UTC’: U_ILLEGAL_ARGUMENT_ERROR After much debugging it turned out that the version of ICU inside the PHP-FPM chroot had not been updated. Updating the chroot copies of the… Read more »

Wednesday, April 21, 2021


SOLVED: FreeBSD nfsv4 client/server protocol prob err=10026

While working with high-load and high-lock contention NFSv4 with FreeBSD as the NFS client you may see this error and have the NFS mount hang: nfsv4 client/server protocol prob err=10026 Make sure your mount parameters do *not* include the intr keyword, per this PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=178231. We found this was especially true when using NFS appliances… Read more »

Tuesday, February 16, 2021


SOLVED: FreeBSD 12.2 ZFS: can only boot from disk, mirror, raidz1, raidz2, and raidz3 vdevs

After upgrading a file server to FreeBSD 12.2 this last week we ran into an interesting issue where the boot loader immediately errored with the following message: ZFS: can only boot from disk, mirror, raidz1, raidz2, and raidz3 vdevs vdev_init_from-nvlist: 5 It seemed to be a similar issue to this: https://reviews.freebsd.org/D20319 but not quite the… Read more »

Sunday, September 6, 2020


Dedicated and Cloud Hosting Comparison: AWS, Azure, Google Cloud, Vultr, LiquidWeb

A-Team Systems works with dozens of different hosting providers in almost every environment conceivable. In a unique way this broad experience has given our engineers an inherent “gut feeling” on the performance of these providers and how they compare to each other. The primary goal of this benchmark and hosting review is to replace that… Read more »

Monday, February 24, 2020


SSH Pipe Speed 48 KB/sec FreeBSD 12 PF

We recently ran into an issue after upgrading a server from FreeBSD 11.2 to 12.1 where throughput via SSH pipes would crash to 48 KB/sec almost instantly after starting. The odd thing was scp (which uses SSH) was not affected. After ruling out Internet and remote host issues we discovered that the PF rule was… Read more »

Thursday, November 8, 2018


FreeBSD Jenkins ERROR: Failed to parse POMs

We recently ran into an issue where the version of Maven under FreeBSD wasn’t updated enough (ports/pkg was behind) for the version of Jenkins in ports/pkg. It produced an error like the following: [INFO] Installing /usr/local/jenkins/workspace/bdd_framework_test/pom.xml to /usr/local/jenkins/.m2/repository/Cucumber_Framework/BDD_SearchLock_Safelist_Final/0.0.1-SNAPSHOT/BDD_SearchLock_Safelist_Final-0.0.1-SNAPSHOT.pom [WARNING] Attempt to (de-)serialize anonymous class hudson.maven.reporters.MavenArtifactArchiver$2; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)… Read more »

Thursday, October 18, 2018


PHP-FPM Chroot “security library: invalid algorithm”

After an OS update on one of our client servers we witnessed a perplexing error in their chrooted PHP-FPM environment: security library: invalid algorithm This is produced when the chroot’s libraries were out of sync with the base system, in our case updating usr/lib64 fixed it. A simple fix but took a significant amount of… Read more »

Monday, June 11, 2018


Expand ZFS To Use Larger Disks / Free Space

One of the great features of ZFS is to be able to expand it using new, larger disks — even while it is online. To do this you’ll replace each disk, one by one with a larger disk. After each replacement you’ll rebuild the zpool, and then move on to the next disk. Once all… Read more »

Tuesday, February 13, 2018


Debian 8 Under Bhyve Networking Problem: Packets Over 230 Bytes Truncated

We’re a big fan of FreeBSD’s Bhyve hypervisor, including writing an open source lightweight management interface for it called BMT. While setting up a Debian 8 guest VM we ran into a perplexing issue where networking was not working correctly. After much troubleshooting we narrowed down the symptom to be that packets larger than 230… Read more »

A-Team Systems