Lenin eventually took the same view - that chess was distracting him from a higher purpose. Perhaps a harmless game would have been a better use of his time.
Why do we feel the need to prefer or not prefer what others do with their time? Isn’t that something that concerns only the person and their own life, which really shouldn’t concern us at all?
I think your point really highlights this ethical mindset that has become so prominent today, especially with the rise of the internet.
The world would have been a better place had he kept playing chess then.