Comment by vonklaus | Hacker News Reader

Comment by vonklaus | Hacker News Reader

Here is an unsolicited email I sent last night. I sent it to spinn3r, but any company doing "competitive intelligience", market research, big data, ect can be subbed in here for their name, because the browser and search engine wars are starting. The email:

Just read the blog post on optimizing the elasticsearch cluster. It sounds like you have a pretty awesome deployment setup. Now, I know competitive intelligience is "hot" and "trendy", I mean, for sure it isn't mobile-first omni-channel blockchain bitcoin disruptive "trendy", but it's prety close. Now, I jest, but what I am about to say next I am dead serious about: your product is a search engine, just let people use it as a goddamn search engine.

Now, let me explain. For like probably over a year now I have been making the case that between google, elastic and apache there is just a goddamn open source version of google sitting out there. Would a consumer facing pay for search engine be profitable? idk, is your product profitable? I suspect there is a lot of people who would pay a $9.99 a month to upload and control page rank and not have google own them. Lets assume you were able to capture even 0.01% of googles customers e.g. 1 billion people and got them to pay you like $9.99, then you'd have just shy of $1M MRR, and $11.98M annually. Obviously, you could do some nice caching of resources like stack overflow, reddit, hacker news, and wikipedia and simply store peoples preferences (sort of a heirarchical git like system) and then you would have some pretty swiggity sweet optimizations.

You know what might really add value? Having NLP on your own custom tailored information database. I bet people (who search and use the internet daily) would probably see much higher leverage from this than the sentiment analytics you can give them on a competitor. Also, as an aside, you should be mining the competitors customer support channels not only for sentiment but for their inabiliy to service feature requests/complaints. Proabably one of the biggest predictors of why wealthfront lost MS to betterment. Regardless though, if you guys are looking for an expansion strategy or a pivot, this is it.

tl;dr. Peter Theil was wrong (no, not just about Clinkle) but it is true that:

The next larry page & sergei brin will build a search engine.

Having "been there, done that, got the t-shirt" I can tell you that no, there currently is no market for an unbiased search engine. We (Blekko) couldn't even get enough people to give up an email address in exchange for more control over their search experience.

I respect that. The site just has "acquired by IBM watson team" so I don't know. However, it is possible you were too early. Also, it is impertive that the expereince is built inside a browser. Not 100% sure whate happened to blekko (although looks like you guys landed on your feet) however, if anyone is building a search engine (or you yourself are) I will write up a full marketing plan and strategy and product playbook for free, because it is just so goddamn important.

I think the only Blekko team member stupid enough to still be interested in building a search engine is me, building search for the Wayback Machine at the Internet Archive. And we have a business plan, we're a charity! :-)

Thanks for weighing in. Disappointing that Blekko didn't work out better ... but hey, at least you have the t-shirt!

Do you think that Duck Duck Go's (relative) success shows that there are still possibilities here?

I liked DuckDuckGo's approach, they have really benefited from a user first strategy. That said, it has been interesting how the cost of compute and storage going down has led to such a large number of non-pages on the web. I was reading a paper which estimated that there were probably 10 trillion web documents reachable on the web, and fewer than 10 billion of them were "meaningful" (meaning they had accurate and unique information). If their estimates are correct, literally 99.9% of the web is crap pages. That is a huge discrimination problem.

do you have a link? this fits with my hypothesis, and would be interested to read it.

>We (Blekko) couldn't even get enough people to give up an email address in exchange for more control over their search experience.

The venn diagram of "people who want more control over their search experience" and "people who are sick up to the back teeth with companies who insist on an email address even to take their product for a trial spin" is pretty much a cross section of an egg, and control over search is the yolk.

> Lets assume you were able to capture even 0.01% of googles customers ...

Reminds me of this story:

http://www.musicthinktank.com/blog/and-if-only-1-of-those-pe...

you aren't wrong. It was an off the cuff email I sent at 3am, it was purely a rediculous top down analysis comletely devoid of research, simply to ilustrate a point. The point is, if you were able to get a million people to use your service and pay you $9.99, you would be making decent money. This is important, because there is a strong revenue stream tied to a service you provide whereas many companies struggle to get to 1 million users which if achieved, will complicate engineering and increase costs. After those challeneges are faced, well, then they need to convert those users into some sort of payment scheme...

FWIW, having worked in Google Search before and now working with the open-source/IaaS big data equivalents, there's no comparison. Elasticsearch is light years behind where Google ranking is, and the way the APIs are setup and the service is architected, it's doubtful you could ever use it to build a system that rivals Google. Some of the key signals that Google uses just require a different data organization than all the open-source tools support out of the box.

You have to understand that google is solving a much harder problem, but as a user, a personal solution is much easier to develop for and has higher leverage. If you gave me the ability to curate resources and browse sites I was interested in and upload pools of information as seed data, I would be able to tell (and build) the initial training set much much easier than google.

I only have to be correct for my own use case (with nearly 28 years experience being myself) and google has to be correct for everyone with (at best) 18 years of search results, of which you have to back out what the user is searching for. e.g. If I look for George Washingtons birthday I could look for information about the person george washington, the actual query george washington birthday, where I might find the data wikipedia:george washington, or an event that could contain it preseidents day.

I know both what my query is, and what a successful answer to my query is (at least the format of an acceptable return). Google has to infer that a query like presidents day and then a query like george washington are related and a successful result was the resource: wikipedia, because it contained george washington's birthday.

The point is:

* these examples are contrived. However, a person can be given the tooling to organize their own data & potential data.

* a person will have insight into what they want, almost certainly better than google.

* if they are wrong/the resources hasn't been cached for them, well, the personalized engine can just simply default to google if no results exist and return them

* privacy.

edit: not only this but having every site I visit (if I wanted this and private & personally encrypted) available and searchable is hugely valuable. Google receives takedown requests, resources move, or I forgot how I received them. Caching/downloading/storing video, images and web results and having them remain private and searchable is pretty much reason enough for me to tell you to take my money.,

Why couldn't Google build those tools themselves? And if they could - why haven't they?

Because people won't trust google soon (google != alphabet). When adblockers are fully implemented and used by the entire population, google will look like societ russia after the collapse of the CCCP. It is going to start moving into any business where information is valuable (every business) and it will be super scary as they have the entire worlds data and they have been using it to train a very complex and highly sophisticated AI for nearly 2 decades with billions of records of training data a second in every category you can imagine.

So people will want to protect their data & privacy, especially companies.

People will want to have a better search experience, which means that things that are on the internet that they want to see will be there. Google is probably the most monitored for takedown requests/deindexing and they control most of the video portals and certainly the window you view the interent through so they are a target to legitimate and illegitimate takedown requests.

edit: they also have built the components and outsourced them. Hadoop/Map Reduce, most of Apache's products, elasticsearch, and another google offering tensorflow can certainly be made into a search engine. Obviously not google scale, but certainly an individual.

Ordinary people, particularly young people, clearly care far, far, far, far, far less about privacy than you think. The average middle-class ten year old literally has more pictures of herself than the number of days she's been alive, and an enormous number of those pictures are already online. Moreover, she's constantly putting more and more of her life online.

If anything, I think people want to have more of the interesting parts of their lives be more public (in other words, they want to be "famous", whether just among their friends or on a bigger stage). They do want many things to be "private" in the sense that they don't want certain people to see certain things online. But I think we're deluding ourselves if we think they care at all about the fact that google is scanning their emails or search history, or that twitter is mining their tweets, or that snapchat is using their location data.

There's also a big irony about talking about privacy on Hacker News, where our political views are preserved for eternity on the Internet for anyone to read. ;-)

Reply on news.ycombinator.com