That's a great question, and I'd love to know the answer.
Even so, it is more code, and somewhat more risk. Lack of safety elsewhere might end up using code that is otherwise safe in order to build an exploit (by sending it something invalid that breaks an invariant, or building gadgets out of it, etc.).
You also need potentially to audit all the crates and keep them up to date and so on… without crates you can't do so much.