1) It's not just about making a big financial bet. No one is the same person they were 10 years earlier. The decisions we make along the way change us. You decide to make a big bet, that means you face a whole bunch of different decisions as a result, and then those lead to more, all of which change & become who you are. Many times it's fine, but when it's not, that is something lost beyond money. Do you think no one has ever failed in business and not had an awful impact on them? Some people emerge more determined, some people end up broken.
2) Success changes people. Failure changes people. It also changes how other people look at you. It's not uncommon for people who achieve a certain level of success to make a break with their former lives in some way. Even before success or failure, simply making a big bet then requires a certain amount of focus & drive that can make it difficult, unless you are very careful (and many people are not) to keep your same relationships, to not allow friendships & family connections to lapse. Or let's use something more concrete: many people finance their big bet through family & friends, asking them to make a bet as well. Failure in those cases can be very difficult.
3) You seem to agree with me here: time is always lost, always lost beyond any monetary gain or loss. And it's not just the time lost on the project: Starting from zero means time spent working your way back up. (Perhaps tapping those family & friends for more resources: see above) You only have time for a very small number of big bets. Doing that casually because you believe the potential downside is capped at money is... an incomplete understanding of the risks. If you choose the "big bet" path, it will likely define your life, and what you do day in & day out. And what life is defined purely in monetary terms? None.
So take those risks, all decisions are risks. But consider the costs as well. It's always more than just money.