Each app could register as a search provider - responding with content relevant to the app itself. The search system would aggregate the content from search providers. If there were multiples then it would even track the historic popularity of each provider in order to prioritise the results. (For example there are providers for your music, contacts, text messages, google search, mint etc)
There is a settings page where you can turn individual providers on and off. I added this functionality to the apps I developed. Here is the API/usage documentation: http://developer.android.com/guide/topics/search/index.html
Note how it starts out saying it is a core feature. So core that the search button no longer exists with software buttons, although to be fair search can be shown in the action bar. And I never met a single Android user who actually used the search functionality! Many people did have a dedicated search widget on their device, but that was a Google search, rather than the phone search. The current Android 4.2 homescreen search puts google search results at the top of the screen and device search below which is mostly covered by the keyboard and requires additional clicks to see it all anyway.
This is a long winded way of saying that it seems to me that search turned out to be a dud on the platform by the king of search. I just tried search on my iOS device and it turned out to be very laggy, poorly performing and useless (eg clicking on an email search result opened the email app but not the relevant message!) This leads me to believe search isn't used much on iOS either!
As you say, in-app search does seem to work well across all platforms and apps. It looks like a system based generic search just fails to work on current generations of hardware, operating systems and user interfaces.