Is he expecting you to answer at 3am?

If he expects you to answer at 3am, it's time to find a new job.

If he's willing to wait to get an answer, then don't worry about it.

I turn of the sound and vibrate on my phone, so I don't get interrupted while I'm sleeping.

He's probably just trying to show off about how dedicated he is. Don't get tricked into overworking yourself to match him.