I use xterm and i3wn on debian and I never experienced anything faster. Surely the idea of waste GPU for the terminal never even crossed my mind so alacritty is IMO overkill.
I think I feel the same way, lag has never occurred to me when using xterm -- and I use the heavier features like translations and sixels day-to-day. Maybe it's just the Athena widget style that is making people dismiss it, because it's great.