Comment by jacques_chester | Hacker News Reader

Comment by jacques_chester | Hacker News Reader

jacques_chester

Here's my theory.

NoSQL came about because smart people were exposed to relational technology in this order:

1. A university course or book that mostly focused on SQL and then normalised design

followed by

2. Using MySQL in production.

What's missing from this picture is learning the other halves:

1. Normalisation matters to transaction processing. Fast queries is another matter entirely and usually only gets airily waved at in many books and university courses. I went through an entire semester without seeing "OLAP". Techniques I learned on the job were kept to the "Advanced Databases" course which was only taught sporadically.

The idea that OLAP is some high mystery is just silly. It's join-beating, denormalising stuff, like NoSQL, just with decades of literature and code to back it up.

2. It also matters that MySQL is not the benchmark of relational technology performance or features.

My day job is working with Oracle databases. The price, the odd absence of useful features because It's Never Been Done That Way (I'm looking at you, primary key triggers and booleans-stored-as-char(1)) ... sometimes it's amazing that people pay so much for it. Then you see the manuals, the supporting tools[1] and the performance a half-decent DBA can massage and you get that this stuff isn't as bad as the sticker price says.

For my own work, postgresql is where it's at. But for a big site I'd look to DB2, Oracle RAC, Teradata, NonStop, Greenplum and on and on and on before betting on 5-year old technology reinventing a 50-year old paradigm that didn't work real well the first time around.

[1] Except SQL Developer. What a dog.

> Normalisation matters to transaction processing.

If you mean to mean that normalization matters only for transactional processing, I dare disagree. Normalization matters for data sanity. Building and maintaining a complex and agile application is much easier on properly normalized data. Making sure one bit of data is stored only in one place and is properly decoupled from other bits of data's existence is still the Right Thing to do. True, it is sometime in direct contradiction with data access performance, but this is an optimization issue, which can be solved with denormalization, materialization or the use of some "NoSQL" storage.

Anyway, in these NoSQL discussions I always wonder: As far as I know, Wikipedia is still using a purely relational data-store, and it's main data (an enormous dict of blobs) seem to be the perfect candidate for a non-relational storage. So then, if NoSQL is so good at this task, how come they don't have moved yet? Should they move? (Genuine question)

That's a complicated topic.

For the social web, people typically denormalize data because consistency is less important than personalization and/or read performance. At some point, it's just easier to write the same data in a redundant way than to query from a canonical source and transform it on the fly. Wikipedia's raison d'etre is to show the same data to everybody. In fact, it goes to great lengths to ensure that everybody sees the very last updated version, no matter what. So there's no great pressure to denormalize core services; in fact, quite the opposite.

The next part of your question is whether a document-oriented database would be better. I think it would be possible to write a wiki on top of a document store. You'd gain a lot from simplicity, although you'd lose certain kinds of flexibility.

But, this is not practical for Wikipedia at this point. For everything else that goes into rebuilding a page, or administration, there are plenty of traditional joins.

The software is very much married to SQL. MediaWiki, the software that powers Wikipedia, is open source and database agnostic. There are people running MediaWiki sites on pretty much every RDBMS you can name. While queries and updates are all abstracted away, the core concepts are all obviously SQL. The abstraction layer just gets rid of syntactical quirks and handles escaping.

That said, a typical MediaWiki installation is not well normalized either. MediaWiki is capable of hosting a lot of extensions that extend the behavior of the wiki. If the extension needs to persist data that's associated with existing tables, a typical strategy is for the plugin to maintain its own parallel tables of data, which reuse the same primary key.

Like a lot of successful websites, the MediaWiki culture is pragmatic above all else. SQL databases are used to persist data and the best you could say is that it's a hybrid strategy.

> If you mean to mean that normalization matters only for transactional processing, I dare disagree.

I think we are, actually, in agreement. The benefits of ACID are essential to OLTP. It also so happens that normalised data can be written fasted as it only needs to be written in once per datum.

As for Wikipedia, I think there's two reasons. First would be path dependence, second would be that their data is (I'm guessing) easily partitionable.

I don't really buy that. I have built applications using BDB on the back-end. It works really well for some things. It just has other shortcomings development-wise that usually makes up for the benefits and then some.

The thing is that object-oriented and relational design are fundamentally different. It is very rare to find people who are very good (or even equally good) at both. They are entirely different design disciplines aimed at very different problems. What non-relational db's do is they free the developer up from this second, very different discipine using tools that usually perform very well for standard OO operations.

That's a powerful thing.

Of course it's also powerful to have a math engine which can take your stored data, digest it, and spit out a report based on criteria you hadn't thought of until 30 seconds ago. And iterating through a collection of objects? Not a good way to do that.

> That's a powerful thing. I have been working with people who know relational databases and OO. And I have been working with people who could only do one of these things.

I know with which group I prefer to work.

If a technology is used as a "don't worry: low barrier to entry" to keep the challenged on board, it will become a great measure for lack of competence if you know it.

Seriously, until now nosql advocates would use specific technical merits of the different options we have now, but "developers don't grok rdbms's": its not an argument. According to some, developers also don't grok fizzbuzz.

If you can't wrap your mind around set theory you should also not iterate through a collection of objects.

True enough, but there is a reason why OpenLDAP's performance is atrocious when run with an SQL back-end.

My own viewpoint is that there are cases where the relational model really offers very little of benefit and a lot in terms of cost, but that these are far more narrower than the NoSQL guys suggest. I see NoSQL at its best being a niche tool.

Remember with NoSQL, there's one answer when your customer say "can I tweak this report?" NoSQL? Answer is "No!" of course!

Reply on news.ycombinator.com