It feels like kitty takes _slightly_ longer to start up but it's still a tiny amount of time.
There's also the different configuration options, kitty has a config file but st (if memory serves) needs to be recompiled for changes.
I've found that initially kitty can have trouble rendering fonts with correct spacing but that takes a couple of minutes to tweak to your liking. And I think the colour configurations are a little more complex? I haven't had to change the config for a while though.
The GPU acceleration does make stdout much faster if it's unbuffered (in my experience) and on a 144hz monitor it looks so much smoother.
Hopefully that's enough of a comparison? Sorry I couldn't give more feedback, as mentioned I've only used st for a short time.
For me the difference was large enough to switch to st. Kitty's startup takes 5-10 times as long and it's very noticable for me. As I use terminals a lot this meant a delay every time I opened an instance.
But this is a subjective thing of course. I have all the features I need with st and use a terminal multiplexer for the rest.
My lightly patched st /feels/ better IME, but it doesn't have anywhere near the same featureset. kitty has its fancier font support and cool kittens which make the comparison very uneven.
I limit my kitty use to a single long running instance as my vim terminal, taking advantage of its far better rendering for pretty symbols and inline images via the icat kitten. For everything else I find st/urxvt far more useful, in spite of the occasional rendering errors and font selection problems. Even going so far as to fiddle with sxiv's Xembed support to weakly imitate kitty's icat from time to time.
Simply run the tool within the terminal under test. It's quite easy to use.