If you think about it, it's no more brittle than ajax calls that return HTML. Having used this technique successfully, I know it works and can work well.
I don't recall any handicap at all.
Yes, this kind of solution requires more resources.
Yes, if zoom, filtering, panning and other interaction is required, then the server would have to output SVG and some javascript to manipulate it.
These objections are all worth raising but do not negate the power of this kind of solution. For certain categories of highly interaction apps with a large amount of image or data processing, doing all the work on the client side is a really bad idea, given the wide varieties of client capabilities and available bandwidth.