It has little to do with the filesystem. Windows has OS level locks. In the case of a running executable, the mapped memory holds a lock on the exe file to prevent deleting it. This is intentional. If it didn't hold the lock then it would be possible to delete the exe file on modern versions of Windows.

Edit: since there still seems to be confusion I'll try to be clearer. On NTFS you can delete an open file. This is a solved problem. The DeleteFile API even does this by default now. The thing that prevents deletion is an OS lock. This lock only prevents deletion. Renames are still allowed.