Django ORM vs. SQLAlchemy
reddit.com
reddit.com
For every project, there exists a point where an ORM stops being an asset and starts being a liability. With Django ORM it happens pretty early on, once certain types of queries cannot be run with it since it is very limited in scope. With SQLA this point comes when performance drops (by my metrics it adds quite a bit of CPU time on instantiating SQLA object wrappers around rows). This usually happens late in the game, after you've done all your development, deployed and have been running for a little while. In other words: at the most inconvenient time.
On top of that, consider the following piece of code:
print zoo.location
print len(zoo.animals)
print zoo.name
Edit: I'd like to add that if you have a small project with well-defined scope, it is possible to use an ORM efficiently. However, if performance is of any consideration or you anticipate that the project will grow (most projects do), then consider creating an API to access your data instead. In the short term it will be almost as quick to develop. In the long term, it will save you many headaches.
SQLAlchemy is tailored for performance above all, in that it allows the database to be used in its natural mode of set-based thinking. You can write real queries, and keep decisions the database is capable of making on the database side. The naive system of "row by agonizing row" programming, where you load everything into memory in order to reason about your data, is not really how relational databases are supposed to be used. You need a tool that works along with the relational paradigm in order to work that way, or else it's back to hand-written SQL strings. And it is absolutely false that a large scale application written out as hundreds of hardcoded SQL strings is easier to manage or expand upon than one that makes good use of an abstraction layer.
Also your example about len(zoo.animals) applies to any ORM. Django, Storm, Hibernate, Active Record, all have lazy evaluated collections. At least with SQLAlchemy and Hibernate, "zoo.animals" is only evaluated once, and not on each access.
I agree that loading rows one by one and hard-coding their relationships into you business logic is not the most convenient thing. However, the point of SQL is not really to let you say "zoo.animals". The point is: you ask it questions and it returns answers. Where ORMs break down is where you start talking about aggregate rows or complex computed properties. What do you mean, sometimes animal has property potential_mates_with_offspring_count and sometimes it does not?
Yes any ORM that uses lazy evaluation will suffer from the issue above. That is why I am not a fan of any ORM. Instead, a carefully written and thought through data access layer is much more scalable and may provide true abstraction of your data storage mechanism.
Lastly, no zoo.animals is not guaranteed to be cached, since it may be a declared attribute. The session is not a cache so you cannot rely on it for that.
For the loading full objects use case, you can stretch it much more by using query caching and hopefully Pypy will help a lot here as well. But you can ditch most of the object loading overhead by just requesting individual columns back, then using just the abstraction layer as the next level, then finally feeding your query into a raw cursor. But these are optimizations that you apply only in those spots that it's needed. You certainly don't have to ditch the automation of query rendering, the rendered form of a query can be cached too.
I also certainly agree that data access or service layers are a good thing, and perhaps you have a notion that "using an ORM" means "the ORM is your public data layer" - I don't actually think that's the case. But in my view you still want to make full use of automation in order to implement such a layer.
The way I try to think of it is, Django's ORM was designed for web applications, and SQLAlchemy was designed for databases.
Having just read through that thread, and zzzeek's earlier post (http://www.reddit.com/r/Python/comments/olech/is_django_cons...) I still don't see that any of this has changed from the last time I read a similar thread.
My take:
Yes, SQLAlchemy can do more queries than Django's ORM.
No, you are unlikely to need these in practice for most web apps (zzzeek's example (for a generic User<->Address schema) is "Give me all households in New York with exactly two occupants where neither occupant has any residences outside of the city." Really? When was the last time you had a query like this in a web app?)
Yes, SQLAlchemy is a PITA to learn and use, unless you use the Django/Rails style declarative base interface.
Yes, you will lose out on basically all of the Django infrastructure if you use SQLAlchemy: django-admin, fixtures, introspecting the schema to generate models, django-south, etc.
If you need complex queries or like writing web frameworks from scratch, SQLAlchemy is the way to go. If you just want to use Django and get your user data out of a table, then just use Django's ORM.
I write web applications in the financial analysis sector, and the general technique of writing queries where subqueries must be composed together to combine groups of coarse-grained sets into an ever more specifically narrowed down set is very common - we do reporting of a fairly complex temporally-oriented schema and we can throw up bar charts and excel spreadsheets right up on the client, straight from the DB, without loading intermediary objects into memory or anything like that. The queries are highly composed, we're using aggregates to flatten out sets, then joining to that, we're using window functions in some cases, and we can do it all straight in the ORM with not a literal scrap of SQL anywhere. When the dataset grows from 1000 rows to 1000000 we'll be able to scale since we're using the relational database as it was meant to be used.
I learned how to write those kinds of queries when I worked at MLB.com. There, we wrote scripts that generated all kinds of elements of the homepage and article pages, reams of baseball stats, game wrapups, schedules - again all pulled straight from a really, really vast schema spreading over three databases. Before I worked at MLB I barely knew what a join was.
Looking through my source here, I just counted nine such queries just in two source files related to reporting alone. If you come out to Pycon this year, come find me and I can show you some of what I do.
Obviously the example in my post/talk had to be contrived since I couldn't exactly put up code from work. Perhaps I'll try to think of a more real world example for my Pycon talk this year.
and if you're a guru who wants to use django with orm, but with more control, then you can also use sqlalchemy; but for most django users it's overkill.
more interesting is the question: "what is sqlalchemy for java?". the closest i could find a while back was empiredb - they were great guys, very helpful, but it felt like a very small project. is there anything else?
my understanding of hibernate is that philosophically hibernate is more like django-orm than sqlalchemy, in that it doesn't embed sql in an dsl (you either map objects directly or use hql in strings). the kind of think i was meaning by "sqlalchemy in java" was something more like c#'s linq, or the scala equivalent(s?).
but i may be wrong, as i haven't used hibernate much. in which case please educate me...
[some context for java people: sqlalchemy will let you define an sql schema in python, then create the database from that, and then let you construct - again, in python (objects and chained method calls) - sql queries against that database. so it's more like "sql in python" than an orm, although it does also support orm if you want (and i suspect because of that sqlalchemy's orm is likely closer to ibatis than hibernate).]
But an ORM is about a lot more than just building SELECT queries. Hibernate has a Session (SQLAlchemy copied this name from Hibernate), built on top of similar unit of work/identity map patterns - it also uses pretty much the same notion of transaction integration (that is, flush() any number of times in a transaction, including before queries, then commit() at the end). SQLAlchemy originally copied the name for their "save_or_update()" method, until taking a cue from Storm and renaming it to "add()". The merge() method in SQLAlchemy is directly from Hibernate, to the point where I read its source to determine its exact behavior in some edge cases.
The notion of "cascades" and the options it accepts are almost identical to Hibernate. There's the fact that collections are represented persistently in memory once loaded, rather than doing a "load on every iteration" kind of pattern that it appears all the other Python ORMs do, as this was the pattern that seemed more familiar to me as well as a lot more efficient in a majority of cases.
Collection loading via joins or a second query that loads for a full set of parent objects at once is pretty much inspired from Hibernate, though I sought to dramatically improve Hibernate's behavior here by getting "joined eager loading" to work totally transparently, including if the parent query had LIMIT/OFFSET on it.
Originally we had also copied this Hibernate concept called "entity name mappings" until realizing it was a terrible idea not at all relevant to Python. The horizontal sharding extension, which I've always viewed as just a proof of concept/experimental thing, was designed pretty much verbatim from Hibernate's similarly little-used API...people were really bugging me for a transparent, horizontal sharding system for awhile, even though they still seem unwise to me, so I figured copying their idea was better than just totally guessing how to do it.
Overall, SQLAlchemy can be seen as a product largely derived from Hibernate in Java and SQLObject in Python, rounded out with various patterns in "Patterns of Enterprise Architecture", and later on from the Storm ORM as well.
http://pypi.python.org/pypi/peewee/0.8.1
http://charlesleifer.com/blog/peewee-a-lightweight-python-or...
This is really cool thanks!
Is something like that already out there? Something that works well and can easily be integrated into a project?