I think first and foremost it was incredibly inappropriate for the ref to be talking about the play with the player. But here's a description of what I saw:
-Ref inquisitively looks at Fabiano, pointing to his arm
-Fabiano jokingy shakes his head, smiling, waving his hands in front of himself
-Ref laughs, points to his chest incredulously as if indicating that is in fact what made contact with the ball
For me, given the deliberate pointing, this interchange was pretty clear. Even if he did mean something along the lines of what you said, it's terribly inappropriate to do something that could even be interpreted that way.
A 2-1 score is VERY different than a 3-1 game, and nobody would deny that the Ivory Coast was attacking strongly through the finish. I know if I were on CIV, I'd be livid the ref was fraternizing with the other team, perhaps because they're brazilian superstars.