In principle yes, but it is hard to predict which code will be at some point performance critical and which code will not. In the worst case you chain a lot of alleged "non performance critical" tasks together and in the sum (actually product, performance percentage is multiplicative) your problem becomes a performance problem.
> Or might want to mmap() large files
This is a fringe application that is not covered by my definition of "application that is unlikely to use more than 2GB of memory". Most programs read linear from files.
> So all the hassle of maintaining a separate ABI (toolchains, kernel, libraries etc.), for a modest performance increase for some subset of applications? Starts to smell pretty niche to me.
As far as I see it there is not enough awareness of this feature. So instead of declaring it "niche" that causes "hassles" maybe advertise it more since the hard work (toolchains, kernel, libraries etc.) has already been done. (My personal use case is derivatives market simulations and performance the gain is in the double digits percentage. I'm pretty sure other use cases exists, they just need to found.)
In the end, who defines "hassle", how big is it really, and who defines "niche"? Potentially accelerating 90% + x of code running on your CPU is hardly a niche.