Now logically an SVG "image" is a webpage, and you could easily just embed an iframe or such like (I _think_ <object> also worked? long time ago now) and it would just work because the browser engine understands that an iframe means "time for a whole new world". A big problem for SVG images in image tags is that it could just arbitrarily make any <img> tag also now have the same essential "I'm a full web page now" semantic, which was really something webkit wasn't really ready for at the time. The original implementation was not super great and I hope/assume it's somewhat better now, but at the time we essentially made it so we had a hacked in a image element subclass that would copy the render image from a pseudo iframe. There are a whole bunch of other issues that had to be addressed, and also at the time weren't covered by any specification, like what happens with event handlers in the SVG vs the Image element.
Now you run into performance problems. The overwhelming majority of SVGs you hit are entirely static, a small subset have declarative animation, so now we have a question: do we really want to burn the cost of instantiating creating essentially a full iframe for every SVG image? that impacts load time and memory use so you want to avoid it if possible, and in principle the engine can detect whether the SVG image contains any scripts or event handlers, etc. But if you do try to do this, then you hit issues if the embedding page then tries to interact with the image, and trying to handle that kind of change dynamically would involve that kind of complexity that leads to errors (oh for safer memory handling in C++, but at least these days there are better options everyone is working on).
IIRC the end solution that every browser went for was to just say "screw this" and drop most SVG features from SVGs loaded through image elements.
One thing that people don't understand when looking at SVG, is that they _think_ it's an image format, but the reality is that was never the actual intent of the specification, and that's why SVG is so insanely complex.
Many aeons ago people were like "what we really need is a standard format for vector graphics", and then some people said "it should support animation". If they stopped there you'd just have an image format. But the commercial groups that worked on the standard, wanted a more, for a few reasons. The big party I saw referenced a lot in the past was Adobe (or Macromedia at the time?). They didn't seem meaningfully involved in the SVG-WG when I was a member so I don't know how accurate the claim is, but essentially what people have said is that Adobe/MM was actually interested in SVG as essentially a future for Flash so wanted the specification to be feature compatible, but also because capitalism wanted to sell their software and so pushed for a specification that was "human writable" but fundamentally you would need well funded commercial software to actually develop content. That seems plausible in the "capitalism is evil" sense, but also the design space for "technical standard", "supports everything flash can do", and "uses xml" is super awful and complicated so I don't think that the complexity and weirdness of SVG is direct evidence of that.
The _much_ _much_ more interesting party that was involved in the SVG WG were japanese makers of mapping software for phones (from prior to and overlapping iPhone era browsers), or such companies targeting the japanese market. I do not understand the exact specifics of the legal (or maybe just contractual?) constraints that they were operating under, but the general gist was that they needed their mapping software to be entirely in terms of actual standards. And this is where the feature additions in SVG go insane. The constraints they were under meant that everything they did had to be a standard, but any implementation had to support the _entire_ standard. At the time SVG started, and for a long time after I had become involved in the SVG WG and then escaped, the html specification was a single standard, so the SVG spec could not just (for example) reference XHR even when it was standardized. As a result for these manufacturers the SVG spec could reference the XML and ECMAScript specs, and then essentially everything else had to be entirely self contained. So if it needed network APIs in JS, they needed to specify them internally, and long story short, that's how the SVG specification got raw socket access (no browser ever supported this :D).
It's also how SVG got profiles: these manufacturers had to be able to say "we fully implement this standard" but at the same time this is meant to be an image format for actual art and such, and so it needed complex blends that these manufacturers could not afford (in terms of hardware perf). Unfortunately these profiles were often mutually incompatible, even for basics like blending, because the profiles were selected on the basis of cost with no consideration of interaction.
Anyway, that's a very abbreviated tale of how the "image format" SVG became the gigantic and complex specification it is.
[1] This came from importing ksvg2 into webkit, and the constant refrain of "webkit just being a fork of khtml" could much more legitimately have applied here. In terms of compatibility with content that actually existed ksvg2 was far more complete than khtml ever was, and the general dearth of non-static or generally complex SVG content meant that for years very little work actually needed to be done on the merged ksvg2 logic beyond tracking changes in the rest of webkit. By the first release of Safari a huge amount of basic compatibility and perf work had been done, by Safari 2 large rearchitecting had also occurred, and around the time I started contributing (while avoiding my thesis) KJS was also being significantly rearchitected. By the Safari for Windows release I would say in essence khtml had been rewritten, and I think JSCs only real remaining architectural link to KJS was that it was still an AST interpreter, but I think that was changed by Safari 4? I could be a little off on timing vs releases because this was a long time ago, but even by that point in history the claims of webkit being khtml were simply not accurate let alone today. Bringing it to modern terms by the point of Safari 1's release WebKit was to KHTML as Blink today is to WebKit.
[2] Apple's webkit used to also support pdfs as images because it just through all "image" data at CoreImage or ImageIO that just sniffed the image type and would work out how to decode a much larger set of image formats than anyone would ever think reasonable. I'm not sure how much that has been cut down now, but this is how things like that OpenEXR vulnerability happened - ImageIO supported OpenEXR a format that wasn't generally considered a "web" image format, but if you throw it at ImageIO it would parse it. ImageIO also supports a wide variety of RAW image formats, and you could have those directly as well.