Depends on the use-case.

If you are streaming a live sports event you don't want the video frames to fall behind from the real clock too far, but you also don't want the video to stutter when there is packet loss. I believe you can use the UDP trick I mentioned above to amortize loss of high quality frame data, with "double sent" low quality frame data.