I suppose if you've already memorized the arcane syntax of base R's plotting functions, along with a giant laundry list of highly customized visualizations available via libraries, then perhaps ggplot2 really is slower for exploratory plotting. But for anyone coming to R fresh, the expressiveness and coherence of ggplot2's grammar absolutely blows away base R for fast exploratory graphics.
For working data scientists in industry, there is an extremely important middle ground between exploratory plots that only I will ever see, and publication quality plots. The most important plots I make are those that I'll share with a small audience within my company. These need to be clear and aesthetically pleasing, but not polished to an extreme degree. Again, ggplot2 excels in this regime.