There are obviously times and certain items that do matter and require some stress to handle at that time, but day in and day out those should be rare events. I see people go on long arguments over the color of a button, or (as we see on HN a lot) argue over the minutia of a particular programming language. For the most part those things don't matter. Does the button have enough contrast to its accessible? Cool, let's move on. Go, Java, <insert pl here> - I can sing the praises or yell the problems of many languages, but I've solved problems and delivered great value in everything from VB5 to php to c++. Does the language fit the problem space? Cool, let's move on.
This is a long way of saying that people blame hours working instead of how they approach work. Someone might work 4 hours/day, but if those 4 hours are super stressful every single day, that can't be a good thing.
Caring too much leads to overwork (sapping your ability to do good work in the long term) and/or analysis paralysis.