COST in the land of databases

Just as in e.g. relational domain you can propose well defined operations (such as join two relations based on a common foreign key) and devise benchmarks for these operations, it should be possible to do the same precise thing for graph databases.

For a graph database, you would want (a) ingress, (b) comps, and (c) recovery benchmarks. The (b) comps per above can be based on a set of common patterns of queries applied to a parametric graph spec.

> In fairness, graph workload performance, compared to other types of workloads, is unusually sensitive to a diverse set of details that are rarely captured adequately by researchers.

That reads: we don't understand graphs.

https://github.com/frankmcsherry/blog/blob/master/posts/2017...

I think that answering the "fastest way to do X in a cluster" question is only particularly interesting if X is something that you need to do in a cluster. But clearly they are testing X that is capable of being done on a mid-range laptop. There's a real question of whether the X that they are testing will actually scale to larger datasets than the laptop (or a decently beefy single node) could handle; it's entirely possible that the cluster solution will blow up at that point. And of course, if the single-threaded implementation is faster, then you could presumably just do "X in a cluster" by doing "X on a single node" and using the rest of the nodes to mine your CPU bound cryptocurrency of choice.

>None of them present any evidence that they are any better than a single-threaded implementation on any problem at any scale. So, if nothing else the published papers are not yet right. But I suspect they are mostly wrong.

It's (a) always there, and (b) always hilarious. From operating systems comparisons that need "we didn't actually slow down the processor physically" comparisons to look good to these "we look great, compared to that guy over there licking the floor."

Also, downloading 1GB is no big deal, downloading 1TB is a big deal, so I think your example has more merit in the > 1TB range. i.e. things that don't fit on a laptop easily.

His "smarter algorithms" aren't.

You are conflating a couple of things here.

Of course for real analyses, you want to run them on standardized environments with appropriate access controls.

But nothing says that "standardized environment" has to use a distributed database that, from all accounts, seems to harm more than help.

You could just have a RDBMS or even filesystem, which is appropriately backed up or replicated, on a single beefy VM or physical machine.

> The real opportunity lies in ability to add arbitrary compute/memory capacity to DB query execution flows.

Only if the tools used to provide that ability actually allow you to take advantage of it. If scaling up to 160 nodes doesn't allow you to be any faster than 1 single laptop, you probably could have spent a lot less money on just adding more RAM to a single beefy server than a 160 node cluster.

Athena is built on Presto, which has full support for transactions. Each connector provides varying support (isolation level, multi-statement writes, etc.) for transactions, limited by the underlying data source.

"Our first paper is Scalable Distributed Subgraph Enumeration or "SEED". We will have a future post about rules for picking acronyms."

I suspect the author would agree.

Note that he already demonstrated that for many of the tests, his older MacBook was faster than these 160 core clusters. There were a couple where the older MacBook was not faster than a 160 core cluster, but he suspects that a newer one might be.

The one that paged out happened to be one where he artificially broke it up into 160 pieces to simulate what it would be like if he had 160 similar cores, but that caused enough issues with memory locality that swapping dominated the results. This artificial partitioning was to give a sense for what results on a 160 core cluster "should" be, but the swapping distorts the results enough that for that problem it wouldn't even be a very good rough estimate.

> Is this guy for real?

This isn't supposed to be a publication-quality paper. This is a blog post, using the resources he has immediately at hand, to show why if you follow cutting edge distributed database research you might be led down a wrong, and very costly, path, which you could avoid with a much simpler implementation on much cheaper hardware.

Note that he references an earlier paper that he did publish on this topic; this blog post is mostly just doing a similar check on some more recent published work, to see if much has improved in the field:

https://www.usenix.org/system/files/conference/hotos15/hotos...

So this complaint should be levelled at the papers and the academic communities- why is it acceptable to only provide empirical evidence for these distributed algorithms at data scales where they cannot out perform simpler implementations? Why not require empirical performance measurement on truly large scale examples for these kind of algorithms?