Comment by 3pt14159 | Hacker News Reader

Comment by 3pt14159 | Hacker News Reader

They are asking you to build a recommendation engine. One tailored for a specific type of problem.

Using some startups api aside (spirit of the competition and all that), anyone interested in completing this should pick up the Springer book Recommender Systems Handbook, it's a very good place to start.

Depending on the size of the Facebook dataset, you will need either a graph library like networkx or a graph database (there are many).

You should probably use python because, let's face it, nothing else will have as many opensource libraries, be as fast, and as accessible from C as python is (use cython for painless compiled code and to link to other C libraries).

Don't use recursion unless your language is specifically built for it. Stay away from ILP graph traversal unless you really know what you are doing.

If you are using some sort of context token/vector similarity approach, be sure to know that the number of tokens you are going to need will be huge, and furthermore you will likely need some hard coded rule sets for low follower/following users.

Try to introduce a time-based decay factor or something similar, and of course take advantage of degrees of "closeness" if there are repeated interactions between nodes.

One last thing: if your approach uses some sort of map reduce solution, it might be better for real world applications, but it will significantly slow down your progress. Just load a box up with RAM and use fast algorithms.

Best of luck to all you out there!

Then again, you could try something completely left field and creative.

People who know anything about machine learning will follow conventional wisdom and potentially miss out on a step change in quality of results.

I've seen a couple step changes in the quality of results and they are usually from people that have immersed themselves in current methods, identified the problems with the current methods, then stepped back 100 meters to try to reason about the problem in a different way.

Unlike, say, selling something online, machine learning has strong underpinnings in mathematics. You do really need to stand on the shoulders of giants in order to move the needle forward. It is more in common with building a bridge than it is with coming up with something revolutionary like renting dvds through the mail instead of a brick and mortar store.

Imagine a bridge building contest where the goal was to something that could take a train across a river. I should hope someone coming along and suggesting the rough concepts to an aspiring bridge builder would be lauded. "If you use suspension, be sure to check for systemic wind loads" "Don't forget about the foundations at the end of the bridge" and "If you use concrete, remember to give the concrete enough cover over the reinforcing steel bars so that it doesn't become unsightly" are all good pieces of advice that someone should know.

Now, once you learn the basics and can do what has already been done - absolutely push the field forward.

You seem to be very knowledgeable in this area. May I ask what your background is? I would like to take on some self study in this area. Would you have any advice?

Besides the excellent suggestions by the parent, I might also look at the awesome work done by Jon Kleinberg for a more theoretical understanding. One may start with his pretty concise Networks book: http://www.cs.cornell.edu/home/kleinber/networks-book/

Excellent book. Definitely recommend if you want to develop a deeper understanding.

Just dive in man. Get some datasets, intelligently break them apart, and just try to start predicting who should like what movie. It isn't too hard if you've gone to university for a STEM degree or learned the math some other way.

I would recommend to pick the free Stanford class... great intro to (classical) ML: https://class.coursera.org/ml/class/index

Reply on news.ycombinator.com