An important piece here is that there's still a training signal making it to the makes weights. See SimSiam for a similar example.