1. TOFU won't scale - if I have a single SSH host, I can easily verify the server key out of band, and then never change it (though that's quite an insecure proposition if you ask me). How would you do this to _every single website you visit?_ And if you don't verify out-of-band (like calling up the host), how do you know that you're not being massively MITMed? And then when you leave your house and go to an airport and get a "your certificate changed", all it means is that _now_ you're connecting to the real page, and not MITM page. And even if you verified the original cert and now get a "site certificate has changed". It could mean that the owner rotated his private key. What do you do? 99% will just say "false alarm, ignore, move on".
2. WoT - It's a false sense of security. You're trusting that random people on the internet will 1. Bother doing _any_ verification before signing someone's key and 2. People will keep their private keys safe from botnets. And the network has perverse incentive - the less verification a group does, the more cross-signing they'll do, the more "trusted" it is.