If you follow the MS docs, they do not mention anything about install CUDA on the Linux VM, you do not get a functional setup to start using CUDA and if you follow Nvidia's documentation, they push you down the docker container route. I personally prefer to setup Tensorflow using Anaconda rather than pip.
A slight correct on the OP, the special driver NVIDIA mentions is for CUDA only (no display driver included), Nvidia does not "support" installing of their full display driver in WSL2 so YMMV in terms of reliability and performance.
I personally gave up on WSL2, not because of WSL2, but because it is so hard to use remotely (I do not have frequent local access to my GPU workstation). I tried various instructions and WSL2 itself was fine when used locally (via remote desktop), but networking scripts I found were unreliable, or if I used Windows openssh server it kept crashing and bash.exe is too limited to be useful, just not worth the hassle and effort when I can install Linux native.