I’d say that it forces you to maybe parse and understand the headers that vary header.
Accept-Language is supposed to be parsed and matched according to RFC4647 in one of 3 ways. That part is rather complex, but it’s not Vary’s fault.
I'm not sure if it is possible to solve this nicely in a generic way, but it does make it a bit ugly.
If you segregate content-type/language by end-point then the variability goes away. But now you have a plethora of end-points. The complexity can get moved around, and in this particular case the complexity for the caching layer can be removed, but the complexity isn't gone.