If references were guarenteed sane and consistent values like in Rust, sure - but it's trivial to get a bogus reference in C++, so I don't buy the type safety thing. An assert actually catches errors, a reference just declares intent.

As for smart pointers there's a lot of domains with tight performance requirements where you can't afford them - and to be quite frank, if you can afford to use shared_ptr you can also afford GC, so just use a sane language