I agree with you completely, but you didn't even mention kids. The other stuff, chores, eating, exercise, and so on, can be moved around in time, but kids require constant attention in the moment even when they are a few years old. I'm lucky to get an hour in the evening for me time efter the kids go to sleep, not even enough to watch a full movie if I want 8 hours of sleep. This is every day of the week, since they don't sleep in on weekends but wake up even earlier.
If I don't get to to stuff I enjoy every day I'm miserable, so I usually play video games past bed time and pay for it by walking around in a fog at work. A day with doing something fun feels like a wasted day, like a day closer to the grave with nothing to show for it.
It's not healthy living like this, but the only way I see doing anything different is by skipping doing anything just for fun. I look forward to when the kids are teenagers and I can relax again. /s