As someone old enough to remember: one of the main reasons Google won was that the other engines (shedding here a tear for Lycos) simply couldnt handle the increasing amount of web spam. They were built in a trusted web environment, but suddenly things became cheap enough for less scrupulous people to start creating effectively spam sites, and the engines somehow didn't manage to react in time.