Generally, the performance of a video codec is held back by available hardware performance more than anything else.
The specifications of a video codec generally specify the decoder, but not the encoder. The decoder specification is intended to be implementable on hardware or software, so it can be played back in real-time, either now or in the immediate future. If some compression technique is available, but can't be done real-time, it can't be used. If a compression technique could be done real-time but only if limits are placed on it, then limits will be placed on it.
The encoder has to generate a bitstream that the decoder will decode, so it's limited by the decoder's capabilities. To an extent, it's limited by available algorithms (see how much better a modern h.264 encoder is compared to the original ones), and by encoding performance (particularly if you need to encode in real time, but even offline encoders can only be so slow before they become useless), and the quality / compression performance tends to get better with time. It's still limited mostly by the decoder, though.
This example is for audio codecs, but LAME actually generated higher-quality MP3s than all of the first generation of AAC encoders, entirely because it had smarter encoding algorithms. For that matter, XVID tended to produce videos that rivaled, and occasionally surpassed, the first generations of h.264 encoders. Despite that early lead, both LAME and XVID were overtaken once the AAC and h.264 encoders matured, because they were still limited by the decoders they were targetting.
There's probably still a long way to go just adding complexity to the decoder, even if we don't invent any new techniques. Eventually, it'll get to the point where you could increase the limits on the decoder, but you'll end up using far more power, and getting no noticeable improvement.
So, the limit would be the point when nobody can come up with any new techniques, encoders can perfectly measure perceptual quality (and therefore make perfect decisions about what techniques to use to get the best combination of bitrate vs quality), and improving existing techniques has a high cost and very low return.
I don't think we're anywhere near there, yet.
Lossless codecs, on the other hand, definitely do have a hard theoretical limit, because they can't discard data like lossy codecs can.