It works out of the box with most browsers and like you said there are polyfills for older browsers. BTW the syntax with svg 2.0 is just href.
About the cross-origin issue, you mean it lacks something like a crossorigin attribute? That's only an issue if you plan to load svgs from other websites, I don't think it's a very common usecase. If you use them at multiples places in your website, you should probably have them served from your sever anyway, if not then I guess caching is less of an issue.