When uninstalling an application, you usually do want to remove all of the application's components. How often do you say, "You know, I'd like to uninstall 25% of this application, even though the remaining 75% will just be dead weight without it"?
I can understand not deleting things out of ~/Documents, but a lot of stuff that goes in the Library folders is not what users think of as data that should outlive the application.
However, I think that there are basically three categories of application data
1) Documents -- These should never be deleted and are not invisible 2) Settings & other small data not worth deleting, probably nice to keep around in case you ever re-install. Most stuff. 3) Large semi-temporary files, like samples and other downloaded add ons that are optional parts of the application
I think OSX handles 1 & 2 well, but you're right, it needs a way to handle #3 too. However, I think that #2 is a much better default than #3.