I haven't found that to be a major issue. And I don't think the term "refactor" is the best choice here.
jQuery is library that provides DOM shorthand - it isn’t a true application framework.
Of course not. I don't want a third-party application framework. I discovered years ago that they're not worth the extra effort or the technical debt one incurs with them. (At least not to me. I'm not telling anyone what they should think or that they should design their applications like I do.)
My approach for me and the developers on my team is to develop a deep level of understanding of HTML, CSS and JS. Once you have that, the prospect of doing things for yourself that frameworks would otherwise provide does not seem all that daunting. To me, frameworks like React, Angular, etc. don't reduce complexity, they increase it.
And I hasten to note that I realize that many bright, capable people hold a different view , and I'm not saying they're wrong.