If person A doesn't want to have sex with person B and person B would be inordinately happy to have sex with person A, just about all ethical systems would say they shouldn't have sex.
If person A has money in savings and is clinically depressed and person B wants to take that money and go on a vacation that would make them very happy, just about all ethical systems would say, no, that's stealing.
One of the things functional societies do is to limit the amount that certain people can make themselves happy as a mechanism of limiting the amount to which other people get hurt. Banning workplace relationships because of their serious potential for hurt seems like a straightforward case of this.
Nobody the debate is not between "some restrictions" and "no restriction", the latter is very much a straw man. We are debating where in particular the line should be drawn. For any given behavior X there are trade offs across the entire spectrum from full restriction of X to no restriction of X and the latter extreme is almost always far from ideal but much better than the former extreme.