Yup, ignoring the costs of memory/storage access (cache, main memory, disk) can hurt an algorithm's real-world performance. Laying your data out in chunks that match the memory/storage blocks sizes (cache line size, disk block size) can yield much better performance.

And while having the algorithm be aware of the chunk sizes in every layer of your memory hierarchy is optimal, it can be inconvenient to get all that set up. But what's cool is that you can often lay your data out in a fractal-like pattern, and get pretty good performance no matter what the various chunk sizes are: https://en.wikipedia.org/wiki/Cache-oblivious_algorithm