Specialization is great, if the specialty you choose ends up being important or highly in-demand. For example, everyone wants to be the Geoffrey Hinton of the 21st century, a luminary in deep learning. Not too many people want to be the Geoffrey Hinton of 1986, publishing back-propagation into an AI winter and subsequent decades-long disinterest. It's hard to know which one you'll be (or in the above, both!)
Side projects and hustles allow you to broaden your toolbox and see potentially interesting crossovers. "It's like [X] for [Y]!" is a cliche startup pitch at this point, but it's true that a lot of cool ideas arise from applying principles of one field in a widely disparate area.
I don't think that it's particularly wrong to choose one route or the other--I admire the specialists I know, even as I know that's not something I can replicate. I'm too enamored of the possibilities, and cognizant of the brevity of my lifespan, to be able to commit like that.