Often the RTOS is not exactly free, but not entirely closed either. A while back, i used to work on Nucleus RTOS by Mentor Graphics with a pretty impressive global foot print http://en.wikipedia.org/wiki/Nucleus_RTOS. It used to be sold as an api (with source code given to customers) who developed applications based upon it. I have written portions (IPsec/IKE, SNMP, Ipv6) of its networking stack and at least all of its customers have access to source code. It is pretty well written with very decent coding conventions and can be compared to any good well known open source project (VLC, even Linux kernel). Then there are others such as Wind River's VxWorks among the more popular ones. Though i am not very sure of its licensing model, but it is pretty well recognized and established in the embedded world. Just that these are not as well known in the over all software community but rather more restricted towards those in the embedded industry.