With a large pay increase and a promotion to “guy who remembers stuff so nobody else has to”, I would gladly come into the office and not feel like it was rude to be constantly interrupted.
But as it is, my job is “finish the work you asked me to do”, and constant interruption is quite rude, in-person or otherwise.