I am using a manifest in my new app and never witnessed a user prompt. I am testing on Android 2.2 native browser, chrome and safari.
It is my understanding that desktop browsers do present the prompt, but mobile browsers do not. This feature was mostly designed for mobile, since the browser cache in mobile tends to be constrained. I have a feeling as things move along, that prompt will go away entirely.