It's either just using SVG at all that's the problem or something in the SVG that's a problem. SVGs are generally not GPU rasterized since GPUs are really not built for paths. That's a whole research area in and of itself.
There are things you can attempt depending on your browser, though, such as avoiding non-convex paths in the SVG.
If you want good performance everywhere though the only real option is just don't use SVG if you need to animate it.