Apache HBase 1.0 released

Hot keys can be a problem, performance wise. So people sometimes "shard the key".

Yeah, you'll see it. Operations are basically atomic on a single row, so it often makes sense to have fat rows with tons of columns. You can use the version as an extra dimension, or encode multi-dimensional information into the column to make it pretty nice.

There are (approximately) 31 million seconds per year.

Using column-based storage for metrics at one-second granularity makes for a very fast and useful time-series store.

However, for storing relational data (such as customers and orders) you are correct, HBase is not the right tool for the job.

The problem is that "denormalization" is an overloaded term.

Often, it's used to mean storing multiple copies of data; for instance, to work around a key-value store's lack of secondary indexing. In that case, you're paying a penalty in terms of space (and code complexity) to make certain operations faster.

In other cases, denormalization just means structuring your data non-relationally. For example, you might want to add a set-valued field to one of your tables. The relational way would be to split that field into a separate table and access it with a join operation, but it's not obvious that that's more maintainable or efficient than storing the set inline.

HBase's support for wide rows is just a mechanism that gives you that kind of flexibility in organizing your data. As norkakn alluded to, the distinction between "row" and "column" in HBase isn't nearly as fundamental as in an RDBMS. Data is indexed by a tuple of (row, column family, column), where the row determines atomicity and the column family controls storage locality.

There are solid technical reasons for wanting to go with an RDBMS or a distributed key-value store in different situations. Metaphors like "data junkyard" don't add anything productive to the discussion.

Schema design tends to be a bit different than in a traditional relational database due to the different constraints and capabilities.

For instance, IIRC Facebook has a video talking about how they use HBase for messages, and they have an example of a table indexing the messages for search:

Each row contains the index of one user. For each word in a message they insert an entry in the row with the word as the column name, the message id as the version (a column can have several values with different versions), and maybe some additional info in the value.

You can see how you'd quickly reach a million columns when every word in every message you send or receive becomes a column entry.

It may have been this presentation: http://www.infoq.com/presentations/HBase-at-Facebook/

Cassandra is another NoSQL database that is known for having lots of columns (I believe around 2 billion columns per row). It is typically used for storing time-series. The row is taken to be the "base" epoch, and each column in that row a particular sensor reading.

I've thought about using HBase for storing genomic variants-- each row is a genomic position (3.1 billion for humans), and each column represents a sample or variant. At the current speed of sequencing, a million human genomes might be reached within a 5 years (a number of projects are attempting this).

Typical usage is an inverted index for full text search. You have search token as a key, and doc_id as columns.

Think of HBase (and also Google's BigTable) a a sorted hash table containing hashes, so sparse columns (that is a million possible columns, but in any row a much smaller number of columns used) is efficient and makes sense.

There are some real use-cases when you stop confining yourself to a single tenant architecture. Apart from that, think of any categorization that's dynamic - timeline based data, versioning of data, etc.

I've seen tables with 10^3 and 10^4 columns if I remember correctly. As others have pointed out it's an efficient way to do timeseries, and as HBase tables can be broken into "column families", you may often associate many types of data with the same key, whereas in a relational database you're encouraged to split them into separate tables. Denormalized doesn't just mean duplicated.