1) We forget our dreams and risks because if we have children or just people depending on us to put a roof over their heads and food on the table, we can't just drop everything.
2) Once you live in luxury, it is hard to drastically cut back a few months (like spend 1000 a month vs 3000) and then have enough money saved to just take a risk and hope for the best.
3) Even if YOU can achieve (2), your spouse might not be able to.
4) Loans, assist with this risk averseness, since you don't want to lose your house/invested money. Honestly this is why I am debt free. When a company I worked for went under, people were ripping their hair out, I was just smiling "cool, I got time to look for a better job".
5) In reality, you DO HAVE FREE TIME. I have 1 child (3yo), and a full time job, and a wife in college, AND I find time to r&r AND work on a startup, AND spend time with child/wife. If I have enough hours, no excuse for you. I have co-workers with even more responsibilities than I do AND they manage to find time for a side company.
In reality people who say "I don't have time" are just not willing to sacrifice anything for their dreams. Even if you fail you can at least live your life knowing you tried, if you succeed it feels even better, otherwise your whole life is spent envious of someone else's success.