If they are, and the license allows creating finetuned models but not using the output to improve the model, then the derived model is not a violation, but it might be a derivative work.
So, no matter what they TOS says, it's not an infringing work.
> Downstream users can't just integrate the derivative work into a product without abiding by the GPL terms
You absolutely could do this if the original work is not protected by copyright, or if you use it in a way that is transformative and fair use.
Similarly, if someone creates a fair use/transformative work then the license can also be ignored.
Your customers bought that product under license A. Afterwards it turned out that you pirated some artwork from disney. Then your customer can sue you (not disney) to make things right. The specific license of the original work seems quite irrelevant here.
edit: you seem to see the customer as the primary victim here instead of Disney, but if Disney weren't a victim the customer wouldn't have a case.