He is limited by not having an "expensive macbook with 4 real cores" ?? He gave up on a test because it "paged out the 15GB of data".
Of course at these scales a cluster is x5 slower.
Can someone please explain - am i missing something here ?
Note that he already demonstrated that for many of the tests, his older MacBook was faster than these 160 core clusters. There were a couple where the older MacBook was not faster than a 160 core cluster, but he suspects that a newer one might be.
The one that paged out happened to be one where he artificially broke it up into 160 pieces to simulate what it would be like if he had 160 similar cores, but that caused enough issues with memory locality that swapping dominated the results. This artificial partitioning was to give a sense for what results on a 160 core cluster "should" be, but the swapping distorts the results enough that for that problem it wouldn't even be a very good rough estimate.
> Is this guy for real?
This isn't supposed to be a publication-quality paper. This is a blog post, using the resources he has immediately at hand, to show why if you follow cutting edge distributed database research you might be led down a wrong, and very costly, path, which you could avoid with a much simpler implementation on much cheaper hardware.
Note that he references an earlier paper that he did publish on this topic; this blog post is mostly just doing a similar check on some more recent published work, to see if much has improved in the field:
https://www.usenix.org/system/files/conference/hotos15/hotos...
So this complaint should be levelled at the papers and the academic communities- why is it acceptable to only provide empirical evidence for these distributed algorithms at data scales where they cannot out perform simpler implementations? Why not require empirical performance measurement on truly large scale examples for these kind of algorithms?
Strictly speaking, I think that term attributes more organization to the academic CS establishment than my experience supports.
The point here is that, if they're trying to establish that Smart Technique 2 is better than Smart Technique 1, they need to first establish that Stupid Technique Q doesn't work at all. Which they haven't.
If they don't have access to "truly big data" to do so, well, most of the people who want to use the Smart Techniques probably don't either.
BUT
With only 2TB of local storage, your 4TB dataset (or output set) has to pass over the net for every execution. At 10Gbs - this alone can take 20 minutes to 1hr.
If you want to write it to the local SSD - multiply that by x3 or x10 ?! ....
This 4TB of memory is still bounded by 4 x 30MB of L3 cache. Which means your single thread implementation will be slowed down by x3 or x5 due to memory latency.
Your multi core implementation will probably suffer even more.
Distributed system are VERY hard, but dealing with them is inevitable for certain workloads.
That's what I mean. You have to get really big to have Big Data. Fine, you can't do PageRank on a petabyte of web crawls using one machine. But the datasets that people use for benchmarking, at least the benchmarks that are made public, you definitely can. You can go far larger than a typical benchmark dataset and still do it on one machine.