If you are mainly using jquery for its DOM manipulation¹ rather than for browser compatibility² or things that didn't exist consistently in older browsers³ then there are much smaller libraries that do that job which may be worth looking into.
https://github.com/fabiospampinato/cash or
https://github.com/franciscop/umbrella to give a couple of examples. Some explicitly support IE11 so you are not dropping as much support for legacy browsers as you might otherwise.
Though if jQuery works for you and isn't a performance issue, then by all means keep with it. It may not be ideal, but good enough and does the job. Let the naysayers spend their time debating whether you should or not, and just get on with making things!
---
[1] selection engine, chained selections, chained modifications, …
[2] not the issue it once was, if you can abandon IE and old Android browsers from your supported UAs or can deal with any issues that crop up individually
[3] again, if you can afford to drop support for legacy UAs