When I play a game and there are frame drops, stuttering, lag, dropped inputs, etc., it reduces my fun just as much as if the game were poorly designed. Maybe that's not the case for you, maybe you don't care, but I do, so do others.
I don't think Nintendo should make a console that rivals the best machine money can buy. I do think they took too long to refresh the hardware in the Switch lineup and their customers are worse off for it.