I wrote the open source alternative to major vendors (ETS, Pearson, etc.) that competed in the Hewlett Foundation's ASAP competition last year. I started a company around that technology earlier this year (http://lightsidelabs.com/) and have been eagerly awaiting the release of EdX source code.
(as disclaimer, I wrote some preliminary thoughts on the announcement several weeks ago at http://bit.ly/ZJTt8E which are probably useful for gauging my pre-existing bias)
Looking through this codebase, it's roughly the level of machine learning work that I'd expect from a first-year graduate student; it gets the job done but is really only tailored to the very narrow task of essay grading and doesn't offer much extensibility. My guess is that for that narrow task, it does a pretty good job, but in my experience teachers are rarely satisfied with just getting a machine learning-generated score and nothing else. The frontend API is a nice addition and the Django service around it is sure to be beneficial, but it's not really ready for primetime yet.
I'll be eager to see if they continue to develop it or leave this as their open source contribution and continue on to other flashy announcements.