Database heresies (The "right" and "wrong" way to do ORM)
b-list.org
b-list.org
Your bad taste for relational databases is more likely a result of using a crappy ORM pattern (such as Active Record). Go try a proper implementation of Data Mapper (I can't recommend SQLAlchemy highly enough), then come back and tell me how you feel.
All that relational goodness does you no good if you want to architect your application using an OO data model in a web application server. Maybe you're trying to argue that that's the wrong way to do web applications? But if you accept that people want to use Rails/Django/etc... then you have to also admit the argument that RDBMS's are the wrong tool for the job. This is true for the same reason that a Dremel is the wrong tool for framing a house, despite the greater expressive power of the Dremel's cutting tools.
"SQL databases behave less like object collections the more size and performance start to matter; object collections behave less like tables and rows the more abstraction starts to matter."
This false dichotomy that's always presented is distracting, and leads to people naively taking religious positions about what should in fact be a non-issue.
(I see you develop with Rails though, so probably yes.)
I'm afraid I have to side with the author against the "relational set theory purists." I actually don't care much about styles of ORMs, and I don't even care that much about having to write SQL, either. I can do, and have done, both. For me, both work equally well, or rather equally poorly.
I loathe relational databases because, every time I have used one, it has eaten a vast amount of time in making the schema and queries fast enough. Yes, I know what third normal form is, and I start my database designs with it. I know how and when to denormalize to improve join performance. I know how to look through a query plan and create necessary indices. I've even looked through low-level IO statistics to figure out where a slow query wastes time doing unnecessary IO. Guess what: that work sucks. I've used Sybase, considered a "real" RDBMS, and I used MySQL, considered a "crappy" RDBMS. Both have sucked about equally, just in slightly different ways.
I am an application and system-level programmer. I want to be able to tell my app: "store this data for future retrieval." I want the store to be instantaneous. I want the future retrieval to be instantaneous. Period. As a major bonus, I want to be able to distribute the store between multiple machines, but do so as transparently to my application as possible. I also don't want to risk losing my data when one server loses its disks, so I want at least semi-transparent replication. ACID semantics are, obviously, a big plus. No RDBMS gives me this flexibility in designing an app. Oracle RAC, supposedly the cat's meow of scalable RDBMS, has RW database load balancing, but it still relies on having a single centralized SAN store on its back end.
To add insult to injury, the promise of arbitrary queries against the data just doesn't pan out. Yes, I can run an arbitrary query. Overnight. Adding this query to an application with a non-trivial schema and a large dataset usually requires so much indexing and query optimization work, that I might as well write the equivalent retrieval code for a Berkeley DB store by hand. I hate going through my queries and using trial-and-error to figure out where a subselect will outperform a join, and I can't afford to hire a bunch of DBAs, who also happen to know set and relational theory, to do this work for me.
Basically, you just define data structures in your language like you would any other piece of data, and it is silently in the background serialized into an efficient (I would hope) ACID datastore.
I work in java, and so I have the benefit of a stack trace where ever I want. So in the class that deals with sql queries I keep a hashmap with stacktraces as keys and total execution times as values. Whenever I feel an application sluggish, I just optimize the first 3-5 queries, which usually it's just "explain" and "alter table add key", and using a cache when it's serious. This allows no other thought whatsoever about optimisation. I don't even use indexes by default, on the grounds that they always slow inserts but not always speed things up.
It turned out that relational databases were in the right place at the right time, and the “good enough” implementations and uses took over the world.
Yeah, that's what I've thought for a long time too. It's nice to see that in writing.
But I guess the rest of the essay leaves me a little cold. If the relational model is fundamentally flawed (rather: fundamentally mismatched to the overwhelmingly popular application design paradigm) then isn't the solution just to chuck the SQL database in the trash and start over?
So what has us tied to SQL and afraid of using the filesystem for storage directly? What are the other things we get from a database that we want? As far as I can see, the real need is for atomic transactions and consistent backups (or replication, which is the flip side of those features). No one cares about the "relational" stuff at all, really, as proven by all the awful ORMs out there.
And yet, if that were true, we'd all be happily using Berkeley DB. So what else am I missing? Some part of me thinks that there's a great startup idea in this space, but I don't quite see it yet. Tools like Google's App Engine seem to be attacking part of the problem by at least tweaking around with the SQL assumptions, but they're still fundamentally relational at their core (and they're still services, and my gut tells me that most of the world will never trust their data to someone else's service, no matter how well-justified it might seem).
Since the relational model predates OOP, to say nothing of flashes in the pan like Rails, it's backwards to say that the model is "fundamentally flawed" because it is mismatched to OOP. The mismatch isn't even real: OOP is a valid way to describe applications but it is not intended to describe databases. The relational model can enforce consistency and data integrity at the whole database level, which is a lot different than the object or single application level.
Programmers who developed large systems pre-Oracle have a very different view of the benefits of RDBMSs than programmers who first tried to use a database while developing a toy program with Rails. A little experience will teach you that for pretty much all real applications in the world the data is the only thing that has value; application code comes and goes. Any business that puts OOP purity before data integrity and consistency will not last. Would you feel better knowing that your bank uses Oracle, or that they use Rails and a half-baked ORM?
You might think "no one cares about the relational stuff at all" based on ORMS and Rails, but that is a peephole view of the world. The sooner you realize that Rails et al. are the Hollywood celebrities of the IT world the sooner you'll figure out how important those things really are.
I had more to say about this subject a while back in my article "Why Programmers Don’t Like Relational Databases" at http://typicalprogrammer.com/?p=14
I remember reading about the way Microsoft did things in office: saving was just dumping the program memory in a file. It's obvious it was bad, but what's interesting is that the most important reason it was so horrible is that it forced you to carry the old code with you forever, just to be able to load and save. Having another abstraction layer in between would have made it much more manageable, even if it saving would still mean dumping the memory somewhere.
Of course, in their particular case that may have been an advantage.