Open-source wise there's Moodle (PHP based) and Sakai (Java based).
But from my POV within higher ed IT, my impression is that many of the big guys (Michigan, Indiana, Cal, NYU, Yale, MIT, Harvard, Stanford) are desperately searching for what's next. edX (Harvard+MIT along with contributions from Stanford) has open-sourced their software, I have no idea if it comes close.
The problem with starting up is that to make anyone happy, a new Learning Management System needs to have two things working spectacularly well:
1. An online testing engine supporting both automated grading and human grading with a wide variety of question types and an easy to use test builder.
2. A gradebook tool that's easy to set up, can import and export spreadsheets, and can support any wild scoring system a faculty member can devise (and they can come up with some doozies). It also needs to integrate with any and every other tool in the system, as some faculty somewhere will want to grade every single type of activity students can perform in the system.
In addition to those two baseline elements it needs a world-class collaboration system for file-sharing, plus messaging and forum tools and online chat. It needs to support multiple class sections with unified resources, ad-hoc class groups, and student-initiated study groups. It needs to support fine-grained permission schemes based on classes, instructors, grad assistants, sections, individual students, and groups.
Every file, test, assignment, forum, and chat room all need to be able to be gated access based on student activity so that, for example, a quiz must be passed before getting access to a document which leads into an assignment followed by a forum-based discussion.
Research is also critical to support, and your system will need to be accessible in a secure way to collaborators, guest instructors, and remotely-enrolled students around the globe, in a way that doesn't make the faculty wait more than a few minutes to grant those people access.
All of this will need to be provided in a fully responsive user interface that scales well from 4k screens down to tablets and smartphones, including the testing and gradebook engines. You'll also need a free mobile app that provides all the same funcionality in a reliable native interface on iOS, Android, Windows Phone, Blackberry OS, and more.
You also need to integrate with the University's student information system, and update enrollments in courses and sections on the fly as students add/drop and courses change, and send grades to the SIS in return. You need to integrate with external tools at the university such as wikis, message boards, and eTextbooks; and outside including cloud-based calendars and document sharing tools including Google Apps for Education and Office365.
You'll also want a instructor evaluation system, electronic portfolio functionality, program-level assurance of learning awareness, institutional assessment reporting, user activity monitoring.
Oh and all of this needs to be customizable to match the traditional way of doing business at each of thousands of different universities. Business processes are not up for compromise.
Notice I've left out concerns of usability, security, performance, stability, ease of deployment and upgrading, etc. That's because those are at the bottom of the list of priorities for the people with the money.
So... good luck.
That being said, Moodle has some serious issues with dataloss if a user's session times out (and for a long time, the default timeout delay was just 24 minutes). If someone hasn't specifically clicked "Save" or "Submit" before time runs out (even if they were typing constantly the whole time), then those buttons invisibly turn into "Throw away all my work and make me log back in and start from scratch". It's horrific. https://tracker.moodle.org/browse/MDL-11972
We landed one major university and several private training organizations,however ultimately got squeezed out and decided to move onto bigger and better things.
The issues people entering this market face range from Patents, that in my opinion should never have been issued, (http://www.google.com/patents/US6988138) to their insane license agreements that cost education providers a lot to walk away from.
We use it at my school and it's great. I also use their api as a developer at EmeraldExam.com and find it to be terrific. They're an open-source rails app: https://github.com/instructure/canvas-lms
EDIT: should also mention lore (lore.com), which was founded by some friends of mine and fairly recently purchased by a John Katzman company.
It is hard.
Paper comparison?
I have no further comment to offer on this.
The group within Microsoft that produced Encarta had a lot of teachers in it. After Encarta was retired, that group took it on themselves to create a lean competitor to Blackboard. The effort lasted about 2.5 years. It suffered from a lot of internal political resistance, because at the executive level, competing with Blackboard was viewed as being neither a smart battle, nor 'synergizing' enough with the Windows/Office prime directive. However, what eventually killed the project was the recognition/fear of how much bureaucratic resistance there would be from schools themselves, and perhaps to a lesser degree the tangle of privacy laws. That is, overall it was decided the risk/reward didn't make sense.