When I worked in an office I never worried about being the first to leave, and didn't even realize other people do.
For me, the biggest downside of working from home is how easy it is to get distracted by non-work. I go grab a coffee and end up putting dishes away, cleaning up, etc. and next thing I know an hour's passed by so I'm working late or feeling guilty.
I've been working on my time management, in general, and that's helping.