Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

Nadella isn't making a particularly strong case for MS either with his seeming distaste for any product of theirs that isn't either mobile or cloud-based (like the ones where they actually have a monopoly to build off and no competition worth speaking of), which seems to arise from that 'chasing growth numbers' mindset.

Loss of direction is what stalls big tech companies 9 times out of 10. If you have 10 decision making actors on the board of directors, you have to effectively do 10 different things that each of that guys want, and all of them with compromises made to accommodate the 9 remaining other things that had to be done simultaneously.

My father frequently says that "a public company is a like a woman with 10 husbands, all trying to make love to her at the same time"

https://m.youtube.com/watch?v=5lutHF5HhVA

Somehow, over time, this message was lost, and it was no longer ok to be #2 or #3 in some verticals (like search), and having very healthy profit margins wasn't enough either. The Microsoft search deal was a bad deal, poorly executed, as well: Microsoft couldn't meet the monetization and performance requirements from the start (Microsoft paid out of pocket for a while for this), y! lost control of an important product, and the staffing reductions to search that were supposed to be enabled never came.

Hopefully Verizon's open source legacy will become even bigger and make it easier for these kinds of innovations to happen.

We've had some success a few cities over in Verizon Labs... https://verizon.github.io/

Generally when you disclose something (like the fact that you run the open source process at Yahoo) that's a disclosure, rather a disclaimer.

A disclaimer might be considered the opposite "I don't run the open source process at Yahoo, but...", or, more commonly, "IANAL".

Very optimistic calculation...

https://github.com/vespa-engine/vespa/tree/master/vespa-hado...

It's a datastore in its own right (just like ES), but I imagine that e.g. you wouldn't use it to handle transactions.

Let me try myself answering my own question, I hope someone hops in and tells me where I'm wrong or how else to improve :)

     1) Get PostgresSQL exntensions via "package manager" pgxnclient
     1.1) pg_bouncer - For connetion pooling
     1.2) yoke - As a high-availability cluster manager with auto-failover and automated cluster recovery
     1.3) prestodb.io - Distributed SQL query engine for pgsql
     1.4) pglogical - Logical streaming replication for using a publish/subscribe model
     1.5) pg_lambda - To create your own AWS (meta) Lambda
     1.6) pg_strom - To offload tasks to the GPU
     1.7) zombodb - To utilize full-text searching via indexes backed by Elasticsearch
     2) Put all together with pglogical and presto to seperate GPU/CPU intensive tasks.
     2.1) "Build Missing Middleware" - To design/fuse a query visually that combines multiple backends
     2.1.1) Create a binary data-stream by integrating pg_lambda, pg_strom, presto and zombodb
     2.1.2) "Build Missing Middleware" - A tensor processing extension to use ML Model evaluations
     2.1.3) "Use Missing Middleware" - For data-processing via Machine-Learning models
     2.1.4) "Use Missing Middleware"- To output ML processed results into the database
     2.2) Partition these queries using "pg_lambda + middleware" to create accelerated and fused query results

This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for...

Disclamer: I work on the Vespa team in Trondheim, Norway.

is it used inside Yahoo - because Vespa comes with its own tensor processing engine. I wondered who would use one over the other.

Smart companies commoditize their products' complements (something that needs to be bought with the product) so that whoever wants buy their product has a large variety of offerings to select from. For instance, MS-DOS's ability to run on any standard PC architecture machine commoditized the PC.

It's not clear how commoditizing data search makes selling media to consumers easier, because consumers of media don't buy data search. In fact they expect it to come for free from the media company.

Might it not have the opposite effect instead? That is, it makes starting up a media company cheaper and allows competitors to spend more money on acquiring media?

Plus answers was done at a time when we were extremely naive about the wisdom of crowds and open source knowledge. Later efforts learned from those early 2000s failures. I'm pretty sure Joel Spolsky specifically called out Yahoo Answers when he was talking about what Stack Overflow was going to do right in his early announcements.

As for the current Answers site, well, we'll see what happens. I know the PM, a delightful person. I don't know the plan. But if there is a plan to make something useful from it, apply. It apparently makes money (otherwise it would have been killed long ago), and that means there's something to work with. When I started at Yahoo I was hoping to get onto the Groups team for the same reason -- a huge challenge to fix something that could be made cool again.