This isn't really apples-to-apples comparing with FFTW.
1. It's been my experience that distros don't configure AVX properly for it, and
2. PhastFT takes its inputs de-interleaved in separate real/imaginary arrays which is generally not how complex data is provided, so that overhead doesn't appear in PhastFT.