Django (and pretty much any other framework) is usually "fast enough"; typically you'll hit the limits of your datastore first, and even if you reach the point where the framework becomes an issue it's trivial to scale horizontally due to its stateless nature.