I agree. Putting in hard work usually pays off rather than stifle your chances, under the right conditions.
I think what detractors are reacting to is the attitude that putting in hard work or long hours are a path to success.
They would prefer that having a desire for success would be enough to be successful so that the kid who rolls out of bed at 11 has the same chance at success as the person putting in 80 hours.
Now, I don’t personally think 80 hours is sustainable but for some circumstances it does help, such as early stages of running a business or similar situations.
Of course good planning is better than having to crunch, but sometimes you have to crunch. None the les, I still don't think this is their quibble. Their quibble is that working hard for success makes the workshy look bad.