Please do elaborate.
Case in point, a 40 second 1080p clip would take over 8 hours to encode on an i7-4800 - that's less than 2 frames per minute. You need a lot of horsepower to cut that down to 40s / 60 FPS.
[1] https://bitmovin.com/bitmovin-supports-av1-encoding-vod-live...
When measured in, say, total sum of all CPU cycles and hence total energy spent, then yes: the decoder is the bigger deal. So your argument holds for non-live videos.