In most cases the companies that fail after X years have been failing for at least X-1 years. This means these guys were under total stress for X-1 years.
One has no idea what that feels like if they haven't gone through the process.
Another thing is, you're pretty much locked into that company for X years. Even if you realize on year 1 that it's not what you started out to build, you still need to go with it, otherwise you're seen as an irresponsible prick, and people won't see you as "experienced entrepreneur". Most of the times what happens is you either trick yourself into thinking that it will work, or compromise your initial vision into building something mediocre that generates money (For example you may start out thinking you will build the next generation social network that will change everything, and end up with a behavioral targeting ad network)
Imagine wasting X-1 years of your life working on something you no longer "truly" believe in. That part is failing.
If you still don't agree, try raising money and starting a company, and work on it for X years. You will understand.