- Have a refresh token which is only usable with the (right) auth server and can only be used to generate new access tokens. As it's only usable with the auth server it's easy to revoke (it's kinda like a session cookie).
- Keep validity of access tokens short.
So by revoking the refresh token the logout will be done in at most highest_refresh_token_start_time + refresh_token_validity.
Through sadly any "faster" logout method on a distributed system is indeed quite complex. (Like propagating bloom filter headed blacklists of early revoked access tokens).