The bloom filter allows you to largely distribute the work of the server, and serves as a reasonable proxy during a failure off the server. More importantly though, there's no reason it need be a centralized server. Invalidated tokens could be broadcast to a wide number of servers that each maintain the invalidated token list (it's a great case for a CRDT, since it is append only with a TTL). Normally it'd be a pretty compact list, and if it isn't, you probably want to take a more defensive posture anyway.

...and beyond that, if an attacker can take out your invalidation server, which needn't be directly accessible to the public, you've already had a pretty serious security breech. I think the least of your problems would be the invalidation server.