The baseband chip.. it runs its own RTOS (separate from Android/ios/etc): http://en.wikipedia.org/wiki/Baseband_processor
Rather than explaining, I think this link is a pretty good example: a chip from qualcomm that was sold at least through 2012:
http://www.osnews.com/story/27416/The_second_operating_syste...
It was demonstrated at BlackHat in 2011 on an iPhone 4 and HTC Dream (android):
https://www.blackhat.com/html/bh-dc-11/bh-dc-11-briefings.ht...
http://www.infoworld.com/article/2625180/smartphones/coming-...
Edit: Slides from a talk at DeepSec:
http://2010.hack.lu/archive/2010/Weinmann-All-Your-Baseband-...
Edit: video: https://www.youtube.com/watch?v=fQqv0v14KKY
edit: I think this is the paper: https://www.usenix.org/system/files/conference/woot12/woot12...