It makes it easier to shove into a library or middleware.
Most of the issues you list are UI problems that could have been solved with a non-modal login prompt, a "remember this login" option, and a "stop sending the Authentication header" button.
My point isn't that existing HTTP auth schemes are great, because they're not (even if they weren't saddled with a terrible UI). It's that migrating from those mediocre schemes would have been easier than migrating from the mediocre schemes we're using now will be.