How can Google detect if a spammy link has been created by the site's owner or by a competitor?

I don't see any way to detect that - apart from asking webmasters to report spammy backlinks. But that really isn't sustainable.