Linux also does not allow you to overwrite a running program; you'll get ETXTBSY.
The difference to Windows is that Linux allows you to delete (unlink) files that are in use. The inode actually owning the data continues to exist until the program is dead.