I've heard of the success stories, MySql, Redhat, Jboss, are there others I should pay attention to? Thanks in advance for any advice, links to blogs, book recommendations, etc.
In the case of MySQL and JBoss, they have a free (beer and speech) offering that met a huge need in the developer community, which got them their necessary ubiquity. Once they had market share, they could start charging for services and other things (didn't JBoss charge at one point for documentation?, I forget)
Redhat is interesting in that in my opinion, they just showed the power of business development. Customers like their software to be "guaranteed to work on X", and software vendors like to keep X to a small number to reduce testing cycles. If you look around at large software vendors that "certify" their product on Linux, it's almost always on Redhat enterprise linux. This is a business relationship, not a technical one. Redhat builds a rigorous release process and helps companies certify, sure, but anybody could do that. They had the vision to build relationships where they could pay off.
I'd argue that Github is an open source startup. They use, advocate, host and inspire the open source community across every language and platform. They also happen to be very successful. Tom's recent post on "Optimize for Happiness" is a fantastic read, and would be something I'd read over and over again if I were to start up a company anytime soon: http://tom.preston-werner.com/2010/10/18/optimize-for-happin...
Canonical Ltd with Ubuntu is another one.
The business model for open source business solutions is to give away the software and build a service business around it as the domain expert. A company like Engineyard makes money by selling services around Rails which is open source. Some of the key Rails/Ruby contributors work at Engineyard.
Automattic has a very effective business model and a good stream of revenue from several sources. Millions of bloggers, developers, designers and consultants make living through the WordPress ecosystem. Many large blogs and theme developers even make millions with WordPress.
In a presentation, the CEO of activeCOLLAB(Ilija Studen) mentioned that they were pulling in 50k Euros a month, this was back in 2007 *
----
* Video Lectures.net(unfortunately it's on Croatian): http://videolectures.net/webstart08_studen_sks/