This seems to be a known issue, but nobody knows how to fix it. (It's some UAC thing, apparently.) I imagine this is what WSL 1 used, and it was similarly slow.
From the original NT microkernel design, filesystem access is all asynchronous messaging, and it can be intercepted and modified by drivers at many points. A lot of software takes advantage of that. But it's also why caching isn't as effective as it could be, since you can't rely on all those unknown layers returning the same result each time.
Plus a lot of Windows systems are running an AV which likes to check to see if you should be allowed to read any particular file in case it might be dangerous.
In relation to the above conversation about WSL1 performance characteristics, my impression is that ReFS would be equal or worse to NTFS for what Linux/POSIX file operations expect. (ReFS is still very much on the "SQL Database" side of the analogy, like NTFS as it was of course designed to support much the same transactional semantics, and some additional ones as well at the "virtual storage space" level above the physical pool of drives.)
In danger of getting further aside into tangential spaces, Ubuntu's current experimentation with ZFS by default is probably one to watch to see if the user experience of complicated multi-physical disk file systems like ReFS can be made to fit and is useful to a broader user base than just power users. (Though arguably compared to the broader spectrum of casual Windows users, even some of the more casual Ubuntu users might be considered power users in comparison.)