>My new boss says it’s not really a big deal to ask me to join these calls before my start date.

Then it's not a big deal if you tell him you are not joining the calls as you will have a medical appointment at the time.