The downside of an approach like this is that it limits you to a single programming language on both the frontend and backend. This is exactly the reason RPC frameworks like gRPC use an IDL.
I agree that it limits the usage of the tool in a polygot environment. I disagree it's a downside.
The absence of an intermediary language does give some benefits to the first class citizen (in this case, Typescript).
However, there are some other developments [1] which attempt to make the Typescript type system an IDL to allow for better interop.
If your IDL isn't a first-class citizen then what do you expect the derivatives to be?