Audiobooks my friend. Listening at +1.5x (playback speed) helps
This happens both when I'm listening to audiobooks and doing something else, or when just sitting and doing nothing when listening. I don't have the same problem with reading, I can do that with high focus for hours without any problem.
I think the key here is to listen while doing something undistracting and boring -- so not like driving (which requires your attention often), but rather e.g. on the treadmill or commuting in some longer distance, where you can passively zone one (eg. long train commute, not 3 subway stops).