I understand your point completely, as I used to feel the same way in the past. Then I realized it's just that being forced to do a commute for work makes it OK to potentially put extra load on your spouse, or to leave your kids with some other caretaker, but this doesn't have to be this way. It should be fine to take some time if you need it even if there's no job-related reason behind it.

So now both my wife and myself take time for ourselves to get that decompression time when needed.