As a developer, it's our responsibility to provide users with at least 2 options. While Sign in with Apple is convenient and seems to be fairly popular, users should be in control of their data.
Some stats from my app where users have the option to either Sign in with Apple or use their phone number:
73% use Sign in with Apple and the rest using their phone number.
If it was a privacy issue, wouldn't people rather trust Apple than an indie developer?