Nothing stops anyone from writing a mediocre song. Hell, it's one of the easiest things in creation to do. Hum a melody, make some words fit. Write it down, make up your own notation for the melody if you want; you don't even have to perform it. Congratulations! You're in the same field as Colin Meloy.
ChatGPT isn't even doing this much. It's read all the tabs and lyrics of every song within its reach until 2021, and then spits out an average response based on average similar requests. It's a songwriter in the same way a 7-year-old telling you a story about an alligator who ate a city is an author.