Self-plagiarism is a pretty boiler-plate clause in most mature codes of conduct (e.g. when submitting a paper for publication). It doesn't mean "I use the same turn of phrase every once in a while". It means "I reproduced substantial portions of prior research or other text of my own and passed it off as new research."

As for who the victim is, typically the direct victim would be the funder: if you received a grant to pursue novel research and it turns out you were just rehashing prior work, then you have stolen funding. Similarly, if you are under the employ of a university, then you have stolen funds by reproducing old research and passing it off as new...instead of actually producing new research. These are of course only the monetary victims, the broader victims are those robbed of scholarly advancement by having your job title filled by someone who recycles material versus advancing the field.