It does. But this issue arises because of file locks. Running an executable holds a lock that prevents deletions (but not renames).

Many OSes have file locks, though they often don't use them as liberally as Windows.