Been there, done that. And it's a pain in the *
You'll find yourself needing a method to add or remove classes eficiently. So you'll rewrite .addClass() and .removeClass()
Then you'll need a clean way to make Ajax and callbacks. You'll rewrite .ajax() and .when()
Then you'll need to fade something in all browser. So you'll rewrite .animate()
Ad lib.
JQuery Minified and GZipped is 35k (not 90 as the op said, ignoring the gziping.) JQuery is cached in most browsers. The performance hit is not worth the pain of re-inventing the wheel.
The problem is not JQuery, it's bad code. Bad code won't magically disappear in vanilla JS.