3rd party libraries / frameworks written with non-JavaScript languages is worth another blog post and discussion. I agree completely with you that libraries / -most- frameworks should be written in plain old JS.
The reason is simple, contributing and reading the project source code should not start by learning a programming language. It is a different scenario when a team in a company has decided to learn and write their project with TypeScript/CoffeeScript/etc. language.
About tooling, I think writing modern JavaScript (ES2015, ES2016) is beneficial and therefore, you need a tool like Babel. When you go to that path, why not use TypeScript instead and get ES2015/ES2016 + type system + great compiler?