Good reply, you are right.
I have even written ("Be aware that output from ls without options differ, if redirected") that it is so but was somewhat fooled by my experiment and the different timings for "ls" and "ls -1" when using XFS.
Of course I was not able to reproduce this difference and assume that I did work on the system and it was not idle.
I have repeated the the tests, on another mostly idle system. (Therefor the timing is not comparable to the first test).
This time both filesystems live in a loop device and the benchmarks were run with hyperfine (3 warmup loops, 3 measurements each).
XFS
ls -l >/dev/null Time (mean ± σ): 5.890 s ± 0.394 s [User: 1.096 s, System: 3.491 s]
ls -lU >/dev/null Time (mean ± σ): 2.254 s ± 0.386 s [User: 0.648 s, System: 1.602 s]
ls >/dev/null Time (mean ± σ): 542.7 ms ± 4.4 ms [User: 443.8 ms, System: 98.7 ms]
ls -1 >/dev/null Time (mean ± σ): 534.4 ms ± 2.3 ms [User: 450.1 ms, System: 84.1 ms]
ls -1U >/dev/null Time (mean ± σ): 115.8 ms ± 1.8 ms [User: 80.9 ms, System: 34.6 ms]
ls -U >/dev/null Time (mean ± σ): 121.0 ms ± 14.6 ms [User: 80.9 ms, System: 40.0 ms]
find >/dev/null Time (mean ± σ): 225.0 ms ± 16.2 ms [User: 150.1 ms, System: 74.7 ms]
EXT4
ls -l >/dev/null Time (mean ± σ): 10.493 s ± 0.392 s [User: 2.495 s, System: 5.032 s]
ls -lU >/dev/null Time (mean ± σ): 8.395 s ± 0.157 s [User: 0.735 s, System: 5.049 s]
ls >/dev/null Time (mean ± σ): 1.966 s ± 0.010 s [User: 1.760 s, System: 0.202 s]
ls -1 >/dev/null Time (mean ± σ): 2.014 s ± 0.032 s [User: 1.798 s, System: 0.216 s]
ls -1U >/dev/null Time (mean ± σ): 221.4 ms ± 1.3 ms [User: 83.7 ms, System: 137.6 ms]
ls -U >/dev/null Time (mean ± σ): 222.5 ms ± 2.0 ms [User: 78.9 ms, System: 143.6 ms]
find >/dev/null Time (mean ± σ): 531.0 ms ± 4.0 ms [User: 377.3 ms, System: 153.7 ms]