Django auth was designed around the typical use case ca. 2005. The world's moved on a bit since then.
What I end up doing nowadays is roll my own (and re-use bits of contrib.auth where appropriate, such as password hashing) using a custom auth backend and keep a separate site for the Django admin which uses contrib.auth models. As long as I don't have to use 3rd party apps that rely on auth.User I'm fine.