I think I agree that the default behavior should be to leave all that data on the user's system. But I also think that apps should have some kind of associated file manifest, which you can use to automatically clean up after the app, or at minimum get a definitive list that you can go through manually or with a script.
It turns out that Apple installer packages do include a manifest (cutely called a BOM) of package contents, but users don't typically keep package installers around after completing installation successfully, and it says nothing about files that may be written by the app itself while in use.
I don't see how you could possibly claim that the user library "ruined" anything. Most apps need to save some kind of settings and/or transient data files. Where else would they go?