Hacker News Reader

Comment by phamilton | Hacker News Reader

While not a fan of mongoDB for many reasons, any sufficiently large scale application will have denormalization. Caching layers cause all sorts of cache invalidation bugs because of this exact problem. Indexes, while often managed by the DB, are essentially denormalization. One could even argue that syncing data between a client and server is a class of the same problem.

My point is that we deal with these problems out of necessity. If you don't get a ton of traffic, then sticking with a normalized RDBMS is probably best. However, large scale systems will have redundant logic. Generic solutions (letting the normalized DB do the work) will often be slower than specific ones (denormalizing your schema and moving logic into the application layer). These are pretty simple principles that I've seen on almost every large-scale application I've worked on.

View on HN

Indexes are not an example of denormalization. Normalization means that there is one canonical element that describes an object, not that there's only one copy or representation of that element.

Indexes are basically a copy of the original data, only sorted in the right order.

They are denormalized data, sort of.

Indexes are often only a pointer to that data, and they are not denormalized, they are better described as projections of your data (this way saving the work of ordering/filtering on that same projection in the future.)

Not only that, they're projections which you don't have to spend effort on keeping up to date, and which the engine automatically uses to optimize your queries. While there's an argument to be made that the same can be accomplished in a non-relational database with suitable application code or a translation layer, all else equal it's better not to need those things in the first place.

And as we saw from a recent Uber post about Postgresql, that automatic updating is not actually free.

Was going to say the same, but I realized the OP was talking about in your code, not so much as overhead.

Of course it's not, but if you're not Uber, that almost certainly isn't a problem for you. When you use MongoDB, you get to roll your own indexes no matter who you are.

Yes, they are pointers. But the indexed fields have to be duplicated for efficient reading. From Wikipedia:

"An index is a copy of select columns of data from a table that can be searched very efficiently that also includes a low-level disk block address or direct link to the complete row of data it was copied from." [1]

When you reference a field in an index, you're effectively duplicating this column on disk.

We can discuss semantics, but at the end of the day, indexes are glorified copies of your columns in a specific order.

[1] https://en.wikipedia.org/wiki/Database_index

Are you suggesting that RAID-1 and RAID-5 are also examples of denormalized data, sort of?

Not really, for two reasons:

- It's done at the hard disk level, so no database or data schema is involved.

- It's not done for performance reasons, but for security reasons (not to lose any data).

"In computing, denormalization is the process of trying to improve the read performance of a database, at the expense of losing some write performance, by adding redundant copies of data or by grouping data."

It's a technique that involves copying data for some very particular reason, an abstraction to hide the nasty duplication because the desirable thing is to have only one source of truth.

So while there's some remote similarity (the copying of data), the goals are completely different.

A good RAID controller exactly fits your included quote. It will speed up reads by distributing to all disks, but writes will be slower because they have to be written to all disks before the controller will signal the write as completed.

What you're really after is materialized/indexed views, or having the NoSQL DB be a layer after your primary DB...

If you excuse the buzzword, data warehouses are built for this very reason. So you are right in the sense that cache invalidation and denormalization are problems in sql based large scale systems, but they are largely solved problems whereas in mongodb you would need to do this work manually.

> [...] data warehouses are built for this very reason. [...] [cache invalidation and denormalization] are largely solved problems [...]

Uhm, no. Data warehouse is not a database with operational (i.e. current) data, it's a database with historical data. If it is out of sync with production for a day or a week, it's totally OK, depending on its design.

So no, cache invalidation is not a solved problem there.

dozzie, could you remove the "Uhm, no." from that? It only adds snark to an otherwise good comment.

I've worked with data warehouses which were replicated like a multimaster, YMMV.

Reply on news.ycombinator.com