One the one hand, yes, you can absolutely amuse your life into ruins. 24/7 access to fun is the great affliction of our times. Video games are part of it, but so is netflix and social media. If you're spending 20 hours a week partaking in any of these things, that's 20 hours you aren't spending building stuff or honing your skills. It really does add up over time.
On the other hand, you do not need to join the desert fathers to be productive, in fact, taking a moment every now and then to unwind and do something mindless and enjoyable instead helps you last the course, and a break from work can be very helpful in allowing your brain to process what you're working on, often with new inspiration and ideas as a result.
The best approach is to structure your life a bit, instead of having this all-or-nothing mentality where you either always work or constantly binge on amusements, I have a particular day of the week I play video games and do stuff like that. Works great.
My problem is it always seems like I'm building skills to help some capitalist be more rich or make more money. If I want to build something to contribute to a community sans profit, there's the burdens of popularity if I'm successful.
I think people are afraid to admit that amusement is what brings them fulfillment. Having to work on something or build more skills sounds similar to a protestant work ethic to me, which was much more important when the essentials of living were not as assured as it is today (or you need to fit into certain social circles today).
I suppose it helps your personal character and having more things to talk about, but I don't know if worrying whether or not a majority of people I'd meet find me interesting because of my skill(s) is also a good selling point.
It feels a bit like saying "I never watch TV", "I never listen to music", or "I never read fiction" - "I only ever code, I am purely productive."
I remember when I was in high school when World of Warcraft dropped, like half the guys in my class got super hooked and would stay up late every night and phone in the school work every day. Their grades just never recovered from WoW, even some of the really bright ones graduated with barely passing grades.
Though admittedly WoW was a bit of a killer cocktail, because of the social pressure to partake in raids. That kind of game was also so new, like yeah there had been MMOs before, but nothing like that, so the "WoW effect" may admittedly be a relatively unique event in gaming history.
You simply have a different perspective, and are overinflating their position.