I mean if we move away from it, yes technically there will be 0 harm possible because we will have to kill all the diary cows, and prevent their offsprings from ever again being born. But thats kind of argument like killing all humans apart from me will end all of human suffering, technically true but not what we want.
Plant based 'milk' aint no milk, just drink with some flavors. Not judging or arguing for/against, but its not equivalent nor replacement in ie nutrition nor taste.