It's perfectly acceptable for a manager, and not remotely 'lacking in manner or courtesy' for a manager to say 'Can we chat later at 1?'.
I've been working for 25 years in 4 countries, I don't know a person who would think otherwise.
The individual here is perfectly within their rights to 'feel anxiety' in any given situation, bu they lack the emotional maturity to recognize their special condition.
It is not a normal human emotion to have anxiety because your manager asks to chat, which should be frankly a very common occurrence.
I completely and fundamentally disagree with your 'if you ask around people will agree with this'.
If you ask the question as 'do you find it problematic or anxiety inducing or lacking in courtesy when your manager asks to chat later without context' - then you've 'led' the answer and might likely create some kind of intellectualized rhetorical response.
But - if you actually observe people in any workplace, collect myriads of messages, and then ask them after the fact to identify when their managers were 'not courteous' without highlighting this specific issue, it will not pop up unsolicited.
It's total bullshit rhetorical nonsense.
If your manager says 'Hey can we chat after lunch?' - you say 'sure' (unless you can't) and then that's that. It's utterly normal.