I don't think it's a derivative work just because one of the inputs is copyrighted. I think it's more descriptive than derivative. Content producers don't generally own copyright in critics' descriptions of their movies or of plot summaries, even though their copyrighted material is a necessary input to the description's creation.

Except the 'derivative' has nothing in common with the original.

I don't think you could argue this is a derivative work and not be sanctioned by the courts for bringing frivolous litigation.

I was debating with a friend about just this: whether a text-to-speech model is a derivative work of audio recordings by a given speaker, such that they'd then have claim on ownership of it. (You could almost certainly create an [overfit] model that could re-generate the original performance of a text from said text.)

Moot if it was a work-for-hire, of course; but if I, say, created a Samuel L. Jackson speech model by training on samples from his movies, and sold it as one of those car-navigation voices, could I be sued? By Mr. Jackson? By the copyright-holders of the movies?

And if I could, what does that imply about impersonators, who do the same thing, but with their brains?

It would fall under "transformative" fair use.

It would not be considered derivative work, because what is produced is nothing like the original. There is nothing recognizable in the work produced for a court to rule on.

This is like saying the hash of the text of a book is derivative. If it were ruled that this is the case (that a hash is a derivative work) then suddenly every single number in existence is a derivative of every single other number (since there will always exist some function that will transform X into Y.)