The downside of defining things like pointer provenance in the IL is that it ties the semantics of your IL to C. If you are writing another language and want to have very different symantics - say a flat memory model where pointers are integer indicies into memory, or definied signed overflow - then you will have to be very careful and constantly have to fight or work around the C semantics. Basically the same downside as when you use C as a compilation target (like for example Nim does/did).