Using the phrase "my sweet summer child" violates like half of the comment guidelines and is a boring canned phrase to-boot. If you want to insult someone, at least be creative about it.
(There were some sporadic uses before Game of Thrones, but we're talking like 100 years before Game of Thrones, and the meaning was a bit different.)