When you say you "remove" the token, does it continue to be valid? If the token accidentally leaks (logs?), or if someone willfully intercepts it, will they have access to the calendar?
Microsoft does not, so it will remain valid. We don't store it or log it anywhere, and the system that retrieves and processes data is not publicly accessible.
I know you have good intentions here, but I hope you are aware that you may unintentionally be storing it somewhere [1]. Not much you can do about it unless you keep actively searching for it everywhere.
[1] I strongly remember reading an article here on HN recently about some security breach stemming from writing production secrets in log files, but I can't find the link :/ If anyone else has it, can you post it?