Does the problem go away again if you go back to below 80%?
So turns out it's a bit more involved than what's been commonly told as a straight up 80% == bad scenario. ZFS by default divides[1] each vdev (RAIDZ or mirror set) into ~200 allocation regions called metaslabs[2].
When allocating from a metaslab[3] it will check if the free space in that metaslab is below the threshold defined by metaslab_df_free_pct. It seems the threshold was changed to 4% free space at some point[4].
If the free space is above the limit it will use the fast first-fit search, if not it will use the expensive best-fit search.
However, as noted that threshold is per metaslab. So if the pool is fragmented, even though the overall free space in the pool is above the 4% threshold, there might be metaslabs with less than that free, which will lead to the expensive best-fit search.
So it's not a hard limit, but it should start to be noticeable above 80%.
[1]: https://www.delphix.com/blog/delphix-engineering/openzfs-cod...
[2]: http://dtrace.org/blogs/ahl/2012/11/08/zfs-trivia-metaslabs/
[3]: https://github.com/openzfs/zfs/blob/master/module/zfs/metasl... (note metaslab_df_free_pct)
[4]: https://www.truenas.com/community/threads/zfs-tweak-for-firs...
When it hits that threshold it becomes immediately noticeable - I/O performance will basically fall right down. At my old job we noticed it around 93% on Solaris 11.x, and maybe 85% on Solaris 10 off the top of my head, on two different production systems. Best practice was considered to keep pools on any ZFS system (OpenZFS or Solaris) below 80%.
If you have say a couple of 4TB disks in a mirror vdev, get it say 98% full, then expand adding new vdev with two 8TB disks mirrored, then you still have to pay the price whenever ZFS allocates from the first vdev.
ZFS tries to spread the load between vdevs in a weighted manner, which means most of the writes would go to the new vdev. But if it allocates from the first vdev you'll pay a latency price.
This means the latency in this case can be quite unpredictable, and not something that'll go away until enough blocks are freed from the first vdev.