It's a bit nitpicky to go this direction, but in general having the display server implement all of the things X does is not necessary at all, especially because most apps, certainly browsers, essentially ignore those features of the display server, because they need more functionality and better performance.
That said, I would not be surprised if XFree86 was, at some point, significantly larger than contemporary web browsers, given the enormous amount of functionality that was loaded into it. Of course, modern web browsers definitely trump any old X server, so is this really relevant? I guess it depends. I get a huge sense that what people are longing for is a return to the "good ol days" when software was simple and fast, and X is a damning counter example, though it certainly doesn't mean that there's no point to be made at all, just that it is perhaps less black and white than people suggest.
I am not trying to say that feh is as complex or as slow as a web browser, only that in general, people treat the division of "native code" and "browser code" such that code running in a browser is always more bloated and slow. Somehow though, this is demonstrably not true. Looking at only very simple software with intentionally very limited scope is not going to give the whole picture. When you look at fairly complex software, the additional burden of having a browser underneath can fade away compared to algorithmic complexity issues and poorly optimized code. Your text layout and font shaping routines are probably going to struggle to compete with a browser if you have a very heavy layout to handle, and if you're doing accidentally quadratic things it's going to matter more than JIT or GC overhead.