Could you expand on how prediction.io would handle a real world data set containing a few million items/users? How long would it take to generate a single user<->user recommendation at this scale? Does prediction.io require that I keep the whole dataset in main memory and how much memory would I need?

I'm asking, because for us (dawanda.com, one of the biggest ecommerce platforms in germany) most of the development effort on our soon-to-be-opensourced recommendation engine was spent on scaling the CF up from a few thousand test records to a 150 million record production data set.

In the first iteration we also built it completely in scala, but as we were putting more and more data into it, memory usage was exploding. We realized that boxed types had too much overhead and that we had to implement the whole sparse rating/similarity matrix in C [1]. Also we decided to go for a hybrid memory/disk approach which allowed us to process 80GB datasets on a machine with only 64GB main memory.

How did you manage to solve the memory consumption issue for prediction.io in scala? Did you use java raw memory access or did you also swap out data to disk/ssd?

[1] http://github.com/paulasmuth/libsmatrix