We ask phone number for the same reason as Uber does - our masters need to contact the user to be able to deliver the service. Explaining this to Apple took a week of back and forth, and only lasted for one submission.
Would you recommend to make it more clear in some way?
In that sentence, "contact" can mean many different things. Will you contact me by text for non-urgent things like billing or promotions? As a user, I don't want to give up my phone number for promotions. And since the request is right after the sign-in flow, I think it's reasonable Apple thinks "contact" can also mean 2FA.
I would put more precise language, such as: "Our masters may need to contact you in order to arrange delivery of X service."
Also, the fact that Apple did approve your app already is further evidence that this is just a misunderstanding, likely because your copy is not 100% clear.
(Sidenote: This is not important, but personally, the term "master" makes me uncomfortable. I personally would not call people "masters," but I also don't know your target market, so the term may be more appropriate in that context.)