Yeah it seems super weird to me that cargo doesn't cache built artifacts based on crate name + version + enabled features + rustc version (and anything else that might make it unique). It seems silly that I have many built copies of the same dependency scattered across different projects on my machine.
Then again, maybe it's actually fairly rare that the things that make a crate build unique are all the same all that often, and the gains vs. complexities of caching just aren't worth it.