Hacker News Reader

Comment by moab | Hacker News Reader

You can compute non-trivial statistics about the entire web-graph (roughly |V|=3B, |E|=100B)on a single high-end server these days. Speaking from personal experience, a machine with > 64 cores and 1TB of RAM can compute connectivity, MSTs, and partition this graph in minutes. Are there graphs out there (with the exception of Facebook and Google's private graphs) that are so large as to be intractable without resorting to external-memory/distributed computing?

View on HN

LinkedIn + Outlook + Dynamics?

Sorry, I should clarify: publicly accessible graphs. The hyperlink graphs are the largest I've found.

http://webdatacommons.org/hyperlinkgraph/index.html#toc1

Also see https://snap.stanford.edu/data/

Extrapolating from a few data sets at that link, the road network for the entire US is probably on the order of 100M nodes and around 250-500M edges.

I would imagine real biological data isn't quite there yet, but it's certainly not for want of trying. Simulations might get pretty close. Blue Brain was aiming for 100M neurons and (presumably) a few orders of magnitude more connections and I think there's another simulation floating around with 10^11 neurons (but presumably less biophysical accuracy).

Obviously, graph partitioning is a useful and practical problem whether or not there are publicly accessible graphs that require more ram than an EC2 X1 instance gives you.

But yes, assuming your data is "big data" when it can easily be processed on a dev workstation or beefy server is very much in vogue right now.

Reply on news.ycombinator.com