I doubt the mere presence of the phone itself is distracting, but rather it's the fact that putting it somewhere far (or allowing it to be taken from you) requires a conscious intention and is sufficiently absurd that it triggers the novelty feeling, which helps your focus.

I just put my phone away in the closet. Perhaps this will become part of my new "focus" ritual.