Using the latest kitty release 0.19.3 vs. st, both already loaded:

  ~ >=> time st ls

 real   0m0.048s
 user   0m0.041s
 sys    0m0.008s
  ~ >=> time kitty ls

 real   0m0.239s
 user   0m0.173s
 sys    0m0.059s

If kitty isn't nicely cached it takes over 500ms on my machine. Using your suggested flag it still takes twice as long.