If you can just pass around the JWT you can save a network call. I would say the size of the JWT wouldn't matter as much as that call.
I guess it also depends on use case. If you are in domains such as banking with elevated security requirements, then probably you want to hit userinfo endpoint else you can continue with token validation with cached or stored keys.