There are several ways of accomplishing this in user mode.

The Detours library lets you attach a custom DLL during the launch of any process, intercepting Win32 calls to file system APIs by writing a custom hook.

There's the built-in "compatibility shims", which are borderline undocumented, but they work similarly. They can toggle compatibility flags and redirect file and registry paths. Conveniently, these are designed to trigger only for specific EXEs based on heuristics.

To support technology like App-V and later Docker containers, Windows has an entire API surface for "virtualising" the filesystem and the registry, as well as any other NT kernel namespace.

Then there's the User Mode Filesystem and probably more approaches I'm not aware of or that I forgot about...