We're a big eCommerce and we're still supporting IE11. We're close to dropping it but it still shows up at about 4% to 5% of our users. And it's not that hard to support IE11, just need the right Webpack configs and polyfills and etc.
It would be interesting to see if traffic drops by that much, or a significant number of those users keep using your site but with a different browser.