Ten-year software video compression engineer here:
TL;DR: it's partly because we're using higher video resolutions. A non-negligible part of the improvement stems from adapting existing algorithms to the now-doubled-resolution.
Almost all video compression standards split the input frame into fixed-size square blocks, aka "macroblocks". To put it simply, the macroblock is the coarsest granularity level at which compression happens.
- H.264 and MPEG-2 Video use 16x16 macroblocks (ignoring MBAFF).
- H.265 use configurable quad-tree-like macroblocks, with a frame-level configurable size up to 64x64.
- AV1 makes this block-size configurable up to 128x128.
Which means:
Compression to H.264 a SD video (720x576, used by DVDs) results in 1620 macroblocks/frame.
Compressing to H.265 a HD video (1920x1080) results in at least 506 macroblocks/frame.
Compressing to AV1 a 4K video (3840x2160) results in at least 506 macroblocks/frame.
But compressing to H.264 a 4K video (3840x2160) will result in 32400 macroblocks/frame.
The problem is, there are constant bitcosts per-macroblock ((mostly) regardless of the input picture). So using H.264 to compress 4K video will be inefficient.
When you take an old compression standard to encode recent-resolution content, you're using the compression standard outside of the resolution domain for which it was optimized.
> Is this continued improvement related to the improvement of technology? Or just coincidental?
Of course, there also "real" improvements (in the sense "qualitative improvements that would have benefited to compression old video resolutions, if only we had invented them sooner").
For example:
- the context-adaptive arithmetic coding from H.264, which is a net improvement over classic variable-length huffman coding used by MPEG-2 (and H.264 baseline profile).
- the entropy coding used by AV1, which is a net improvement over H.264's CABAC.
- integer DCT (introduced by H.264), which allow bit-accuracy checking and way lot easier and smaller hardware implementations (compared to floating point DCT that is used by MPEG2).
- loop filters: H.264 pioneered the idea of a normative post-processing step, whose output could be used to predict next frames. H.264 had 1 loop filter ("deblocking"). HEVC had 2 loop filters: "deblocking" and "SAO". AV1 has 4 loop filters.
All of these a real improvements, brought to us by time, and extremely clever and dedicated people. However, the compression gains of these improvements are nowhere near the "50% less bitrate" that is used to sell each new advanced-high-efficiency-versatile-nextgen video codec. Without increasing - a lot - the frame resolution, selling a new video compression standard will be a lot harder.
Besides, now that the resolutions seems to have settled up around 4K/8K (and that "high definition" has become the lowest resolution we might have to deal with :D), things are going to get interesting ... provided that we don't start playing the same game with framerates!