Note that most of the open source / hobbyist web actually works properly.

I don't think the answer is primarily that it's a matter of money - I think most large companies end up with "web design by committee".

If Zuckerberg redesigned the entire Facebook homepage from top to bottom it'd probably be a hell of a lot faster than it is now. But instead it's the product of an enormous corporate structure and there is no-one to step in and say "guys, this is shit, start again and let's do it properly, no you can't have your 1 meg tracker in there, not even one of them, write a barebones one".

One man or small team software projects in general are better for this reason as well, with a massive team over time cruft accumulates.