Digg Saying Yes to NoSQL; Going Steady with Cassandra
about.digg.com
about.digg.com
The above site has been discussed today on HN: http://news.ycombinator.com/item?id=1181714
also no copy paste in the browser version AFAICT
The data is stored in files of a special format, just like an SQL database.
The basic idea is that data is stored as key > value. You can sort of relate it to the MySQL practice of denormalization and storing a bunch of data from different tables as an array in one cell.
You don't want to (or can't) do joins because data is not normalized, but instead have to process the results you would want separately in a batch process called map/reduce.
It all boils down to this one fact: NoSQL is write heavy in order to improve read performance. Because your data is in multiple places and you are doing batch processing to get the right format/calculated data then you are saving time when the user requests it from you. This is one of the reasons its good for scalability. The other reason seems to be that denormalized data is easier to scale out then normalized.
You should also look up CAP theorem and NoSQL.
A few links that I've been saving to teach myself: - http://stackoverflow.com/questions/1189911/non-relational-da... - http://stackoverflow.com/questions/2170152/nosql-best-practi... - http://www.linux-mag.com/cache/7579/1.html - http://blogs.computerworld.com/15510/the_end_of_sql_and_rela...
Overview: http://www.rackspacecloud.com/blog/2009/11/09/nosql-ecosyste...
Think BASE (Basically Available, Soft state, Eventual consistency) instead of ACID (Atomicity, Consistency, Isolation, Durability).
(I believe this is where the term originated)
PS: Why aren't we in the habit of citing papers like this? So many links are to websites rather than direct sources, and the info in papers is usually highly readable and extremely informative about the details.
The same with RFCs and W3C documents. All that stuff is written very well, from the technical view as well as regarding their readability. Why putting a layer of blogs and websites around them, which actually lower the perceived quality?
Also, I'm frequently annoyed by articles that write about what some other people (e.g. RMS) wrote. These aren't much shorter than the source article, and aren't written nearly as clear as the original.
I appeciate it when an author just links to the source and elaborates on that topic, stating his own opinion, and perfers to quote rather than to paraphrase what's already clearly written in the source article.
But yeah, Dynamo is the most important of those.
If anyone can relate their experience with either of these two platforms, would either be a good choice for live querying for these types of applications? I know you can use MapReduce to eventually get the data you need, but I need to support queries that respond in (well) less than a second, even for very large data sets.
1) What kind of API do you need? It'll have to be simpler than SQL, likely key-value with secondary indices managed at either the API layer or application layer.
2) Which 2 of Consistency, Availability and Partition Tolerance do you need? Is "gets there eventually" enough or do you need "can read it right after I write it" guarantees?
Cassandra does ad-hoc queries quite well if you're willing to use mapreduce.
If you have a smaller number of frequently done queries and are willing to use MR for the others maybe that is a happy place for you.
If Digg are using Cassandra as a big key-value store then how do they look up comments for posts? If they're storing one entry per comment with an index on post ID then it's not really key based any more.
We hear a lot of case studies about performance increases, but I never seem to see any practical details on how the databases should be used correctly.
The plain CF ("Comments") uses the comment ID (which is a timestamp+salt) as the row key. The fields of the comment are columns in the row (username, text, date_created etc).
The SuperCF ("StoryComments") uses the story ID as the row key. Each row contains one SuperColumn per comment. The SC name is the comment_id, and the columns are the fields of the comment.
So, say you want to get the first 50 comments for a given story, you'd do a get_slice on StoryComments, passing the story ID in as the row key, "" as the start column, and a count of 50. You get back 50 SuperColumns, each of which contains one comment.
Cassandra is extremely good for sequential reads, not just random lookups. Just about any list of things can be efficiently stored and retrieved with its batch insertion and slicing operations.