I personally have dropped it for most of the apps I make (especially mobile web apps) so I can drastically reduce the final JavaScript bundle size by not having to polyfill a bunch of crap that all modern browsers already have built-in (Promises, fetch, fat arrow functions, let, const, etc.).
Our official SLA now is to support the current version and one prior for Google Chrome, Firefox, Safari, and Edge. We also don't support any version that is no longer receiving functionality updates.
The most important thing though is communication. If your site isn't going to work well or at all on certain browsers, alert your users to this with a popup. There are lots of libraries for doing user agent detection that makes this trivial.
Locked windows workstations, OS are frozen on Win7 For as long as possible.
JQuery is a great example of how you can create a library with some core functionality and know with 100% certainty that if you use it correctly it will work across all browsers that it was intended to support.
Whenever we have issues with browser support where I work it's almost always because a dev has gone rogue and decided to build something with a heap of vanilla JS instead of using or extending a library with cross browser support in mind.
Perhaps you need to ask why you're having so many issues when so many libraries and frameworks will support IE11 out-of-the box if you use them correctly.
Maybe 5 years ago. You cannot use modern JS and CSS in IE11, you need to use transpilers, polyfills and pray to god that all your dependencies are compatible. Testing and debugging is IE is slow and painful.
Is anyone begging for it? Your users problems are the problems to focus on. If it's not a problem for them, then it's not a problem for you
That's your clue. Go for it.