It does sometimes spit out recognizable chunks of its training data rather verbatim. While half of their compiled evidence is entirely unconvincing to me or I disagree with on other grounds (mostly moral and the difference between a human or computer mimicking a style) - there is some that is pretty undeniable. Search for "the dead" or scroll down to near the end of the article and notice the lyrics "generated" for the punk song. Then compare the GPT-3 "generated" lyrics to the second link's [Verse 2].
https://www.twilio.com/blog/building-a-punk-lyrics-bot-in-py...
https://genius.com/Smash-mouth-all-star-lyrics