That would make interacting with OS slower, and probably even more important are things like .DLLs/.SOs. If the dynamic library or heck even a static library your linking against used message passing imagine all the cases where that would be a bad idea. All that overhead just so you can call a function not written in whatever language you happen to be using. ABI's like we have now are a much better for this reason. Also just because one uses message passing does not mean there are no compatibility issues.
I think the main issue here is if we reference the article linked. "You Can’t Actually Parse A C Header". To know the size of types and therefore figure out memory layout you need to be able to read a C header. With that information you can construct a lot the ABI for a C function. Aside from some OS specific things like stack layout requirements and hardware/OS specific things like what registers are used for certain parameters of a function ect...