Do you find that in the hours you're not working you're constantly worried that you'll be "found out"? If so the problem is those are wasted hours, you can't just sit down and read or travel because you're still supposedly working. Which means it doesn't really matter that you're not working, you're just not doing anything productive.

That leads to doing nothing but mucking about on the Internet, not actually furthering anything for yourself. So it can have an impact on you mentally. I've felt that myself before.

Now, some people have a well-defined work output where the boss just says "I want this, that's all". So if you do it in 3 hours in one day or 40 hours in one week, it doesn't matter, meaning you can actually use those other hours for yourself.