I think the question should be why don't we have filesystems handle this implicitly using a tree structure behind the scenes. Are we so many layers above our hardware that it simply doesn't occur to us as the right solution?
Nor does it let you start iterating through a directory with getdents(), then pause, then come back to iterate the rest of the files later. You have to receive every file name in the order the OS wants to give them to you.
That prevents applications really being able to make use of clever storage mechanisms - you can never find the most recent 3 files in a big directory with anything cheaper than a linear scan.
(But we have impressively speedy tools like Wiztree or Everything thanks to it.)