Do you really think that former Prime Ministers travel on regular passports? Don't be dense.
Do retired small city mayors travel in first class on diplomatic passports and expect to be free from customs and immigration rules that apply to everyone else? They do not.
He isn't a fairytale prince. When he loses his post, he can travel on legitimate documents like the rest of us.