Clojurize the Data, Not the Database | Hacker News Reader

Clojurize the Data, Not the Database

anvil.io

58 pointsmcav17 comments

I disagree with the author. I've yet to see an abstraction layer that doesn't end up as the lowest common denominator of all databases supported or becomes a leaky abstraction.

Databases are different — accept it. I use PostgreSQL and Redis — there is really little they have in common and I use them for completely different purposes. I can't see how you could stuff all that into a single abstraction. I mean, in my Clojure code I use SQL transactions in one part of the program and Redis' optimistic locking (WATCH/MULTI/EXEC) in the other. They are _different_ and they are supposed to be different.

I also believe that the whole story about how difficult it is to "switch databases" is a red herring. If your code isn't a pile of spaghetti, then your database access functions are grouped together, not spread all over the place. This usually means that instead of abstracting the database with a generic database API, you abstract the database with your API, which only performs the operations you need.

If you do that, then a) switching databases isn't that difficult, b) your "database DSL" becomes your DSL, not somebody else's.

As a practical example, my code has a namespace called "distributed" which abstracts distributed synchronization using Redis. I can call (distributed/register-server), or (distributed/dequeue-job). The functions are very short, and really consist of redis-clojure calls with minimal database logic. Now that is a DSL I like.

If only there were a mapping to make a relation in my data store out of an object in my language.

ActiveRecord seems like a local maximum, and I'd be very curious to see what comes up in the next decade.

I've written an ORM for clojure (unreleased). It's 1k LoC, which is huge in clojure.

Let me tell you, I'm sick of writing ORMs, I'm sick of using buggy, abstraction-leaking ORMs. I'm sick of the need for ORMs. Relational DBs should provide ORM as a native, first-class abstraction. The fact that every programming language has ORMs, and that people are starting to adopt Mongo over say, postgres purely because it doesn't need ORM and has a nicer query interface is telling.

Relational DBs need to get with the times.

You mean a local maximum for clojure or programming in general? Because ORMs that use the Data Mapping pattern (like SQLAlchemy or Hibernate) are better in some cases where ActiveRecord is a bit simplistic.

This is a brillant idea and I'm interested that it hasn't been implemented for other languages yet...

EDIT: It seems like a simple idea, but I'm wondering if the execution has failed or if it was adoption or if this is truly an original idea.

It's one of those ideas that sounds great when you hear about it in college, but doesn't really work when you try it in practice. "Universal" data abstraction layers like ODBC and object-relational mappers like Hibernate, Django, SQLAlchemy, etc. have been around at least since the mid-90s and perhaps before.

The problem is that all those inconsequential differences between data stores are really important. The article explicitly says "Scalability and performance are outside the scope of this", but in real-world software, scalability and performance matter a lot. When we'd tried out Hibernate for a past employer, performance dropped by 80% - data operations were now taking 5x as long to complete. You can recover much of that by tuning your mappings, but by then, you're pretty much writing SQL anyway, and your app wouldn't work with any database other than the one you tuned for.

Also, nearly every app needs its own data abstraction layer, one that works with the particular operations and domain objects of the app rather than tuples and records and tables. At this point, there's little reason to go through another abstraction layer, adding complexity and slowness to your program, when your calls to the data store are already localized to one module.

The Turing Tarpit: Anything language powerful enough to express programs in is in fact a programming language, and the flip side, any problem complicated enough to irreducibly require a programming language to solve is intrinsically complicated. It's all fine and dandy for the first 20%, but then it starts exploding in your face.

It turns out we have general-purpose libraries for dealing with data already. We call them "programming languages". If there is some way to simplify the problem in the general case in such a way that data manipulation no longer requires a full programming language, nobody's really proved it. If your do create an "ORM" or some other mapper that is somehow simpler than using a full conventional language, than ipso facto your mapper or whatever is too simple to handle the full complexity of the problem.

I'm not 100% convinced this nut is completely impossible to crack, but I am sure it would take accommodations on both the data and the programming side (much of the pain of this process is self-inflicted with bad paradigms and bad programming, but probably not 100%), and it sure as heck isn't as easy as it looks when you just look at it. One of the ideas in my head is to take a crack at this problem myself, but I've been telling that one to shut the hell up and go back to hiding in the corner.

I don't think the idea is original (still a great idea, though).

ODBC, for one, would seem to have had the same aims:

https://secure.wikimedia.org/wikipedia/en/wiki/Open_Database...

It's a bit old, but the Database Template Library is worth checking out. The DTL presents ODBC records as STL objects, enabling the use of all sorts of yummy STL algorithms!

Ruby ORMs like DataMapper and ActiveRecord are datastore-independent and provide similar functionality. I'm sure similar ORMs exist for other languages.

For right now, I find Ruby's datamapper to be state of the art for providing a uniform interface to many back end data stores. Definitely take a very good look at datamapper to collect ideas for a Clojure library.

Different databases/formats/etc have different strengths. While it's nice to say your program to store to any kind of datastore, it's much more practical to have it optimized for the kind that's best for it.

> While it's nice to say your program to store to any kind of datastore, it's much more practical to have it optimized for the kind that's best for it.

I think this is very arguable. For the cases where you want to do some query optimization, thats what the Clojurize protocol's prompt function is intended for.

  (prompt
    [adapter expression]
    "Execute a native database expression")

But a more specific library would be better able to cope with the unique features of a particular database. Why send commands via 'prompt' when a more specialised library will have functions to do that job?

I think there's definitely a place for libraries that add things like validation to models. But the more functionality you include, the more specific your library needs to be.

Correct me if I'm wrong here but isn't that the purpose of the adapter implementations to also hold the different dialect and/or datastore specific optimizations?

[edit] OT to this thread: didn't clojure-dbm have some similar ideas to Clojurize? I remember forking it from your github repository early last year some time and doing some cool stuff with it.

All of the implementations you mention sacrifice specific functionality in order to support a general interface. For instance, clojure-dbm was an interface to key/value stores, so it supported a small subset of common functionality: fetch, put and delete. It didn't support functions that are specific to a certain type of database, such as setnx in Redis.

I agree to some extent, but most of the time all you want is generic data. What is used to get that generic data and turn it into some format that you can consume (be those objects of data structures) is irrelevant.

Reply on news.ycombinator.com