Some critique of the crypto bits, since I'm a crypto person.
1. Do you actually need asymmetric cryptography for this? It seems like at some point the proxy has full authority, and it could just encrypt the token for you symmetrically? (This is valuable because symmetric crypto is a lot less precarious than asymmetric crypto, see next point)
2. Please don't use PKCS1v15 padding for encryption in new systems. It's been known to be busted for about 20 years now. We have workarounds, and they may well be deployed in the exact context you're using it. But they keep breaking, because we just have them to keep the infinite amount of already-deployed software running, not because we think it's fundamentally fixable. This is also the textbook example of a vulnerable service: one that takes ciphertexts and decrypts them on demand. With PKCSv15, I can modify the ciphertext so that _the way you treat the modified ciphertext_ tells me something about how to make private key operations. And in this setup, that means I get the real token, so that sounds bad. The good news is that you've successfully designed around it by adding a signature, so I don't think it's mountable... but. Please, no more PKCS1v15 :-(
3. It feels a little awkward to use JWT for the outside signature but not the inside encryption. But less JWT is a good thing :)
Concrete suggestion: use Fernet (you're already using pyca/cryptography) or libsodium's secretbox and then all of the crypto problems go away. You keep the security engineering dilemma of stateful v stateless proxy (do I want the real token in the proxy at all?) -- but that's another argument.