It's not, for me at least, the distraction potential. I think its more-so the tertiary sensory experiences physical books carry: the smell of the binding; the dimples & knots in the paper as you trace your finger over lines; the visual clarity of your position in the book relative to what is left; there's a lot that even eink sacrifices. People often say that these qualities are what they enjoy about books, but I suspect it runs even deeper than that; that these qualities add landmarks to your memory, and aid in recall.
So, if you read on an electronic device, it's probably beneficial to configure it to not use "continuous scrolling" but rather use something "fixed".
If you read articles on your phone you may notice that you read in a fundamentally different way then you read printed text. You generally keep your visual field located in exactly one place relative to the phone and then scroll content with your finger up into that space. It's as if you moved the book around while keeping your eyes in one place. On the printed medium your eye stays locked to the corners of the page, the page doesn't scroll out from underneath you.
But of course you are also phrasing an interesting point, which is that screens are usually way brighter than ambient light. To this we can add that a mobile phone is so much smaller than a printed page, you think about the cognitive tasks involved in opening a paper where at first you would open it up wide and be surrounded by word and then you would fold it down to the part that you wanted to read and focus on... Hard to imagine a similar experience on a phone; a set of desktop screens has the right size, but maybe not the ability to effectively expand/condense...
The act of turning a page may also serve some function: the visual content is "refreshed", the patterns on the page are brand new and stimulating. Perhaps the memory for each page is "chunked" in a way that a write to long term storage occurs on pageflip. If so, then infinite scrolling would lead to "buffer overflows".
Another thought about scrolling: I also wonder if the movement of the eyes "across" a page is more stimulating and ergonomic than keeping them in one place, which in my experience gets rather tiring.
I find this to be very much true personally. The spatial organization of words on the page seems to provide a trellis that helps structure how I recall information. So much so that I intentionally convert epubs and other long-form documents to PDF these days before reading. I find this becomes even more important on material where I'm writing comments in the margins; the spatial relationship between the comments and the text helps me to recall both.
I also tend to favour software tools that try to preserve the spatial relationship between pages. On Sony and Fujitsu e-readers, for example, when you pinch out, the display transitions to a 16-page contact sheet view that doesn't shift depending on what page you were just reading. At first blush, it seems lazy, why not show the current page and the prior 7 and subsequent 8 pages instead? But after a while you start to realize that it's to help structure your spatial position within the document, a little like how in a book the left hand pages are always to the right of the right hand pages.
You'll have just as hard of a time concentrating on your 5000 dpi perfectly lambertian eink display in 2030 as you do now on your phone, simply because it's a multimedia device built around capturing and redirecting your attention.