You need to pin the match to the start and end of the string with ^ and $ respectively otherwise the negation just matches an empty string or other irrelevant string.
I start with
(.)(.)\2\1
Now to invert I change it to
(?!(.)(.)\3\2)
As you say, the negation matches an empty string. So I put on anchors:
^(?!(.)(.)\3\2)$
But now it matches nothing. I'm not even sure what that regexp says. The entire string is a negation? Would anything match that regexp?
I see elsewhere on this page that the answer involves putting in an extra dummy character, putting that new negation-and-dummy-character in parens, and then requiring that, between the anchors, there be 0-or-more of negation-and-dummy-character.
^((?!(.)(.)\3\2).)∗$
Two questions:
1. Why are my backrefs still \3 and \2? I added another pair of parens. (I thought ?! might not count, but it counted in my second example above.
2. Why does abba no longer match? It has no matches to the negation-and-dummy character construct, which ∗ should match, right?
NB: I used ∗ as my asterisk to avoid bb-code.
^(?!.*(.)(.)\2\1)
BTW a double space indent then formats as code on HN I think.
I went with:
^(?!.(.)(.)\2\1).$
with the thought that if I really wanted those matches I would want the whole strings.
Fun game!
^(?!.*(.)(.)\2\1).*$
(?!....)
^(?!....)$
^(?!....$)
> ^(?!(.)(.)\3\2)$
> But now it matches nothing.
So in this case, your regex matches an empty string (since it doesn't match any character at all between ^ and $, only spaces.)
> ^((?!(.)(.)\3\2).)∗$
> 1. Why are my backrefs still \3 and \2?
> but it counted in my second example above
By the way, my solution for abba is
^(?!.*(.)(.)\2\1)