Good reply, you are right. I have even written ("Be aware that output from ls without options differ, if redirected") that it is so but was somewhat fooled by my experiment and the different timings for "ls" and "ls -1" when using XFS.

Of course I was not able to reproduce this difference and assume that I did work on the system and it was not idle.

I have repeated the the tests, on another mostly idle system. (Therefor the timing is not comparable to the first test). This time both filesystems live in a loop device and the benchmarks were run with hyperfine (3 warmup loops, 3 measurements each).

  XFS
  ls -l >/dev/null   Time (mean ± σ):   5.890 s ±  0.394 s  [User: 1.096 s,  System: 3.491 s]
  ls -lU >/dev/null  Time (mean ± σ):   2.254 s ±  0.386 s  [User: 0.648 s,  System: 1.602 s]
  ls >/dev/null      Time (mean ± σ):  542.7 ms ±   4.4 ms  [User: 443.8 ms, System: 98.7 ms]
  ls -1 >/dev/null   Time (mean ± σ):  534.4 ms ±   2.3 ms  [User: 450.1 ms, System: 84.1 ms]
  ls -1U >/dev/null  Time (mean ± σ):  115.8 ms ±   1.8 ms  [User: 80.9 ms,  System: 34.6 ms]
  ls -U >/dev/null   Time (mean ± σ):  121.0 ms ±  14.6 ms  [User: 80.9 ms,  System: 40.0 ms]
  find >/dev/null    Time (mean ± σ):  225.0 ms ±  16.2 ms  [User: 150.1 ms, System: 74.7 ms]

  EXT4 
  ls -l >/dev/null   Time (mean ± σ):  10.493 s ±  0.392 s  [User: 2.495 s,  System: 5.032 s]
  ls -lU >/dev/null  Time (mean ± σ):   8.395 s ±  0.157 s  [User: 0.735 s,  System: 5.049 s]
  ls >/dev/null      Time (mean ± σ):   1.966 s ±  0.010 s  [User: 1.760 s,  System: 0.202 s]
  ls -1 >/dev/null   Time (mean ± σ):   2.014 s ±  0.032 s  [User: 1.798 s,  System: 0.216 s]
  ls -1U >/dev/null  Time (mean ± σ):  221.4 ms ±   1.3 ms  [User: 83.7 ms,  System: 137.6 ms]
  ls -U >/dev/null   Time (mean ± σ):  222.5 ms ±   2.0 ms  [User: 78.9 ms,  System: 143.6 ms]
  find >/dev/null    Time (mean ± σ):  531.0 ms ±   4.0 ms  [User: 377.3 ms, System: 153.7 ms]