The case law on what constitutes a new, derived work is not at all well developed in the context of software. But looking at analogues from other media—and that's what the law does—it is easy to imagine that a very substantial refactor, touching basically every line, or a less substantial refactor with substantial additions, might qualify as a new, separately copyrightable work.
The claims of copyleft are almost entirely untested in court, and it's very likely that many of them will prove empty if they're ever challenged. Indeed, many of copyleft's claims, those which place obligations on users and contributors, are more TOS-style claims than claims that look like ordinary "reserved rights" under copyright law, and therefore presumably have a legal basis of enforcement in contract law, not in copyright law.
Explicit agreement by contributors to be bound by a license, however, covers most bases, whether the license is interpreted as a license per se, pursuant to copyright law, or whether it's ibstead interpreted as a contract under contract law (or both).