If JavaScript is disabled I just show a message saying the site needs it.
I also only support modern browsers and haven't had a complaint for 3 years.
About 3 months ago I completely dropped support for any browser that doesn't support CSS grid (no more IE 11!). Again, no complaints. This one is more "brave" but I use Modernizer and show a clear message outlining all the required features the user's browser doesn't support and why they would want that feature in a browser. There's a link to a website to help them find a modern browser on their platform.
Edit: I realised that based on what I said you could argue that users are annoyed but I'm just not aware. I doubt this is the case since the users are keen and vocal. I get messages almost daily from them telling me the features they want to see. The last time this had to do with browser support was one message 3 years ago about IE9 not supporting the TLS certificate or something. They were forced to use IE9 at work.