One aspect that's disappointing to me is that the AST they use seems fairly idiosyncratic; Babel's is quite close to the ESTree spec, and even easier to work with.
Relatedly, it also doesn't look possible to swap out the parser, which Babel makes easy. I'd love to use the TS typechecker for a compile-to-js lang I'm working on ([0]), but would need to use my own parser.
I'll also note that incredible folks like James Henry have already written tools to convert TypeScript ASTs to the ESTree format[1]. I've heard it's been working well for using ESLint on TypeScript code.
Personally I want to go the other way – ESTree to TS – so while the prior art is helpful, very little of the code could be reused.
If TypeScript (or a fork thereof) exposed an option to use your own parser, an ESTree -> "TSTree" library could be used to map the output of any babel plugin to TypeScript. I'm sure you've discussed/considered this; presumably speed is a top concern.
[0] https://github.com/prettier/prettier/issues/13 and https://github.com/prettier/prettier/issues/1306