Hmm... if you were a horrible person you could declare a `char[n]` member instead of a pointer. Then you could placement-new the impl in the constructor, and static-assert that `sizeof(impl)>=n`... No more cache misses :-).
This doesn't take into account the alignment of the type though (you'd want to use std::aligned_storage<sizeof(T), alignof(T)>), but that requires knowing enough about T to be able to use sizeof() and alignof(), which means no incomplete types, bringing us back to where we started.