Once the baby starts understanding more about the world, he will not be able to understand why daddy is locked in that room. He knows you're there, he wants to play with you. He doesn't understand the concept of work. He will bang the door very frequently. They love doing it while you're on phone meetings because they can hear your voice, so they remember you and go banging the door right when your microphone is open. They also learn to open doors pretty quickly when it interests them.
If you're working close to where there's a person taking care of the baby and the person is not his mom or someone you absolutely trust 100%, every time the baby makes a single noise your primate brain will focus on it. Maybe a cougar caught him! I had my mother-in-law taking care of him during the mornings, and although I trusted her, I knew she did not have the reflexes to properly ensure he wouldn't get hurt while he was learning to walk properly (~1 year). He would occasionally fall and hurt himself and cry. Absolutely EVERY SINGLE noise he made (even when it was not falling) would completely put me in "alert mode" and I would focus on him, not my work. I could only be able to better focus on work once my wife arrived home and started taking care of him. But it was still not 100% attention on work: I would still hear all his sounds.
The fact that my mother-in-law was taking care of him while I was at home working meant I had a lot of interactions with her. And let me say that: it felt like I was married with her, and not with my wife. The big problem is that I really would never ever marry someone like her! This was a major source of stress. And she did a lot of thing I didn't agree she should, and my wife didn't have the balls to challenge her because she was already helping us so much, so this ensured everybody felt uncomfortable (although, of course, I am thankful she indeed helped us so much).
In the end my wife quit her job and now I'm working in the office. This is not only because of the reasons above, there are other external reasons. While I'm at the office I can fully focus on work while knowing that my wife is doing everything she can to take care of him. And I won't hear noises that my primate brain will interpret as "your son is dying". Still, I attend early morning meetings from home and he occasionally visits the office while he shouldn't (the office door can't be locked, it's a rented house and I can't change it).
On the other hand, he cries every single day that I'm abandoning him while I'm going to work and it doesn't matter how many times I try to explain it, he doesn't understand why I won't play with him all day like his cool mom does.
No matter what you do, you're wrong.