Of course, I realize that implementing this would be a lot of effort and would require change in your release flow.

But just in case, there's more demand for what I suggest, here's an inspiration from FontAwesome: https://fontawesome.com/get-started/svg-with-js. Note how they provide a fontawesome-all.js having all the icons. But they also provide separate fa-*.js for different icon families for the people who don't care about downloading the whole thing.