Does the toolchain on x86 still build an x86 and ARM executable(s)? Perhaps that contributes to the 2x build time?
This applies to incremental and clean builds running locally with default Xcode settings
When you archive and build for the store then your compilation will include all architectures