I will seek when the option is presented. It’s either tiny or nonexistent though (e.g. for gifs, or if the author didn’t enable the video controls, which default to disabled in many browsers). I’m happiest when a video loads paused, and has a giant play button on it.
How are you scrolling? When I’m browsing and reading my hand lives on the mouse. On mobile it’s even easier to click to play. In either case, clicking on a video takes less than 1 second.
I do agree that extremely short loops can be a different story, but 20 seconds is by far over that time for me. 5 seconds is around the threshold.
I’m sure this preference varies by person.