The constraint here is the e-ink refresh rate. That constraint does not exist on regular screens, which is why I'm surprised neither mobile reader apps or even desktop software takes purposeful advantage of it: fast scrolling back with fine, per-page control, keeping page in-place and zero perceptible latency rendering full page. With a finger drag or click&drag movement, this can easily be even faster than with paper books.

I know this works because I've seen it with some PDF readers on desktop in the past, when I lucked into a combination of lightweight document and right "fit page to display" scaling. And, of course, this works natively in browsers with PgUp/PgDown, but only on the few remaining websites that haven't been thoroughly shittified (no "en-" there on purpose) with JavaScript and excessive styling.

When you go back in a physical book, you also put a bookmark (either an actual bookmark or one of your finger) to be able to resume reading where you were, no?

To each their own, it has never really bothered me honestly.