I think the main difference is that on IRC you'd usually be invited to a specific channel and you'd slowly work your way to other channels as your interests grow. If a given channel has too much activity and is too "off topic" you'll usually find like-minded people who'll just fork to a smaller, more manageable channel. But on Discord you're invited to what they abusively call a "server" and from there you usually see dozens of groups with various purposes and features and you're bombarded with gifs and notification icons all over the place.
I think I agree with you, if I was an ~18yo right now and was willing to invest time into a community I'd love all this... stimulation. But as a boring old 35yo it's just overwhelming and feels like the signal-to-noise ratio is abysmal.