Sure thing, but most real-world setups have multiple heavy packages installed.
> If you then write your init file(s) in a certain way (use autoloads) you can maintain this performance while retainng all your customization.
Worth noting that autoloading is useful up to a point - after a threshold, the gains are only theoretical (your reported startup was 0.1s, but immediately after Emacs startup, it froze 2 seconds because all the autoloads are now doing its work).
In the end when I open Emacs I want a bunch of stuff to happen (open files, color them with syntax, etc) - some work has to be performed sooner or later.
> the workflow with Emacs for those who use it is not to be restarting it all the time.
I start it a few times a day, for not accumulating state, particularly that related with Clojure nREPL connections. Not a costly operation.