But if the file you are removing is the only reference (i.e. hard link) to that inode, you're not just modifying the directory's contents; you're also removing data from the filesystem as a whole.
Any other hardlinks in any other directories are completely unaffected.