OK, I found part of the problem. I resurrected my old setup and switched to using fio (was using iozone and some others). When you specify --filename= in fio but have multiple jobs, they are all sharing the same file (https://fio.readthedocs.io/en/latest/fio_doc.html). So instead of 28 jobs reading 28 distinct files, you have reading the same file 28 times. So even when dropping the buffer-cache, reboot clean, etc. etc. before the benchmark, this is 27 times a test of the read caches and 1 time a test of the actual disk, which will obviously seriously inflate the results. I'm thinking --directory= would be the best option.

I also did the following to drop (most) cached data before the test:

  sync; echo 3 > /proc/sys/vm/drop_caches; zpool export $pool; zpool import $pool