http://stackoverflow.com/a/5381905
http://www.w3schools.com/svg/tryit.asp?filename=trysvg_rect2 (This is inline CSS, but it can have a separate style element as well.)
So far as why it's not commonly used elsewhere, unless you've programatically constructed the whole image already, it's rather a pain to animate SVG. The markup spit out by authoring tools for any non-trivial image tends to lack the structure you'd want for animation. I suspect that's why we see JavaScript APIs like d3 used instead of embedded JS.
Authoring tool could spit out JS themselves, of course, but in all the contexts where JS executes you can basically substitute a full HTML document, and that will be more flexible and performant (SVG is not well optimized in most browsers, as the OP notes.)