Hacker News Reader

Comment by malux85 | Hacker News Reader

Hey this looks sweet. I'm still new to this field, but I'm looking to do some collaborative filtering across large datasets (4 billion rows+)

I was looking at using Prediction.IO and this looks suitable too, can you elaborate on the high level differences between this project and prediction.io?

I'm also curious about horizontal scaling.

View on HN

The main reason why Seldon is different from other open-source prediction engines: * Seldon have come from a background of high-scale enterprise deployments before releasing an open source platform, not the other way around. In other words, we already optimized for real-time low latency high throughput production environments. * Seldon provides an end-to-end componentized setup - i.e. front end UI, real-time prediction server, offline machine learning jobs, web-scraping, etc. * Seldon allows developers to run A/B tests and change algorithms with no downtime. * Seldon will provide enterprise features like automated performance optimization of customer's own algorithms using micro-services.

Presently most of our CF algorithms utilize Apache Spark but we intend to be agnostic on this and allow any machine learning platform to be integrated. I believe that Spark can easily handle this size of data set.

With regards to horizontal scaling, there are two parts to consider. Creating the models and serving the recommendations (the Seldon server project).

Model creation is done in a variety of ways, but but can be managed with scalable technologies such as Spark.

The Seldon Server project can be deployed on as many machines are you require and they will work together to provide recommendations behind a load balancer. We have experience working with some very large news websites so this part of our technology is well developed.

Great thanks for your feedback, I will setup an installation in my home lab this weekend, super excited!

Reply on news.ycombinator.com