The main reason why Seldon is different from other open-source prediction engines: * Seldon have come from a background of high-scale enterprise deployments before releasing an open source platform, not the other way around. In other words, we already optimized for real-time low latency high throughput production environments. * Seldon provides an end-to-end componentized setup - i.e. front end UI, real-time prediction server, offline machine learning jobs, web-scraping, etc. * Seldon allows developers to run A/B tests and change algorithms with no downtime. * Seldon will provide enterprise features like automated performance optimization of customer's own algorithms using micro-services.

Presently most of our CF algorithms utilize Apache Spark but we intend to be agnostic on this and allow any machine learning platform to be integrated. I believe that Spark can easily handle this size of data set.

With regards to horizontal scaling, there are two parts to consider. Creating the models and serving the recommendations (the Seldon server project).

Model creation is done in a variety of ways, but but can be managed with scalable technologies such as Spark.

The Seldon Server project can be deployed on as many machines are you require and they will work together to provide recommendations behind a load balancer. We have experience working with some very large news websites so this part of our technology is well developed.

It sounds like you basically want a form of document clustering. Seldon has integrated the Semantic Vectors https://code.google.com/p/semanticvectors/ project which can be used to do this or depending on your technical level to look directly at Semantic Vectors or toolkits such as gensim https://radimrehurek.com/gensim/

Seldon is an open-source predictive machine learning platform that includes a high-performance recommendation engine and data enrichment. It can run multiple algorithms and configurations to optimise KPIs. It will shortly include a pluggable architecture to allow data scientists to deploy their custom algorithms.

Please check:

http://docs.seldon.io/

http://www.seldon.io/algorithms/

He offered to answer technical questions, so let me step in on this one: it's a recommendation engine. E.g., it can do things like Amazon's "you may also like" feature.

Try this:

http://docs.seldon.io/index.html