Prime minister of tiny Norway is about as powerful and important as a small city mayor in the USA.

Do retired small city mayors travel in first class on diplomatic passports and expect to be free from customs and immigration rules that apply to everyone else? They do not.

He isn't a fairytale prince. When he loses his post, he can travel on legitimate documents like the rest of us.