If your analytics show almost no usage and you're not required to by any SLAs or similar, I'd say go for it. We did recently and it's definitely reduced the mental overhead in our development process.
Our official SLA now is to support the current version and one prior for Google Chrome, Firefox, Safari, and Edge. We also don't support any version that is no longer receiving functionality updates.
The most important thing though is communication. If your site isn't going to work well or at all on certain browsers, alert your users to this with a popup. There are lots of libraries for doing user agent detection that makes this trivial.