Hacker News Reader

Comment by dumbfounder | Hacker News Reader

Yes, we can do better than DDG. But if you are expecting to fund a real search engine with a few hundred thousand dollars you are insane. It will take a ton of development and a ton of hardware to create an index that isn't a pile of garbage. This isn't 2000 anymore. You need to index >100 billion pages and you need it updated and you need great crawling and parsing and you need great algorithms and probably an entirely proprietary engine and you need to CONSTANTLY refine all the above until it isn't garbage. Maybe you could muster something passable for $1B over 5 years with a strong core team that attracts great talent. If Apple actually does this, as they are rumored to, I bet they dump $10b into it just for the initial version.

View on HN

Agreed, it's going to require significant investment in hardware and software.

The recent UK Competitions and Market Authority report evaluating Google and the UK search market came to the conclusion a new entrant would require about 18 Billion GBP in capital to become a credible alternative search engine, in terms of size, quality, hardware, man hours making it.

Remember Cuil? Had the size, the fanfare but unfortunately not the quality.

Open Street Map is a nice analogy for what could work. Aside from the open source maintenance of the map, there's also tons of corporate help in the background. Companies that are delivering OSM as a service or relying on it for their own services have an interest in making it better. MapBox, for example, apparently pays tons of people a salary who are contributing upstream to Open Street Map. If we can get an Apple/Microsoft/Other players collab maybe a venerable alternative can actually be built.

> If Apple actually does this, as they are rumored to, I bet they dump $10b into it just for the initial version.

Google pays Apple more than that every year just to set Google as the default search engine on iPhones.

In a way, Google is funding its future competitor.

I agree and I have been hoping Apple builds a serious competitor. I welcome any competition at this point. Let's be real, not many people are using bing. People _would_ actually use apple search.

>I welcome any competition at this point.

Microsoft tried and failed to build a competitor and it's not like they have shallow pockets.

They grossly underestimated a number of aspects:

   - The huge number of man-years invested in Google's search quality stack hand-tuning and what it would take to replicate it.

   - The fact that the machine learning field was simply not ready to tackle the search quality problem.

   - The infrastructure required to build a crawler / indexer stack as good as Google's

I think in 2020, the second problem is within reach of many companies technically. It's mostly a matter of throwing enough money at optimized infrastructure.

However, replicating the search quality stack is going to be very hard, unless someone makes a huge breakthrough in machine learning / language modeling / language understanding at a thousandth of the cost it currently takes to run something like GPT-3.

The most likely candidate to execute properly on that last bit is - unfortunately - Google.

Sure but Microsoft has tried and failed to build a competitor to google not to DDG.

I don't see it that hopeless. I feel it kinda is like starting Open Street Maps. It won't be perfect for a long time but there will be people who'd prefer it and help out.

> People would actually use apple search.

Of course they would - it would be set as the default search on their iPhones with no clear-cut way to change it. You know, "security". The users don't know what's best for them, etc. as Apple seems to think.

If you want a good engine there is no need to index 100B pages, since 99% of the pages are blogspam.

How are you going to identify what's blogspam and what's legitimate without indexing it all in the first place?

You load, detect and discard it?

We’re already talking about building a search engine, might as well talk about a model to convincingly detect blogspam too.

if(has_ads) { spam = likely; }

I think that is why the idea would be to have the tier 1 sites so you don't have to index as much.

I said ISN'T a pile of garbage :)

Reply on news.ycombinator.com