Children. If there aren't any children, then sure, call it victimless.
But if there are children, an adulterer is directly harming them[1].
TBH, we often forget about children when dealing in family law. Another example that comes to mind is paternity fraud, where the argument is that there's no need to find the real father if the alleged father can be violently forced into supporting the child.
Everyone forgets that the child who is mislead about their father in that situation is the biggest victim. All the systems focus on the emotional well-being of the mother at the cost of the child's rights.
[1] Because, at this point in time, every single piece of evidence we have ever seen supports the assertion that children living through their parents divorce have worse outcomes.