Interesting idea. I used to watch Berkeley webcast videos at 1.5x to shave off ~30 minutes from a 90 minute lecture. Any faster wouldn't be intelligible.
I do the same. I usually watch all lecture videos at 1.5x or 2x speed. Saves me tons of time which is good because usually to videos are of random interesting topics that are distracting me from work.
It is the case that you can teach yourself to understand faster and faster speech. The blind often have human interface devices which speak at absurdly high vocal acceleration.
This also depends heavily on information density. I can listen to some types of content at 2x without issue; other content I can't accelerate more than 10-20%.
Some content I have to scrub and rewatch even at 1x speed.
Some content I can watch at 4x.
Blind people regularly teach themselves to listen at 4-5x speed.