It's not clear at all whether the model is a derivative work from a copyright standpoint. Maybe they are, may be they are not - it's definitely not settled, the law isn't very explicit and as far as I know, there is no reasonable precedent yet - and arguably
that would be one of the key issues decided (and set as precedent) in these first court battles. I also wouldn't be surprised if it eventually doesn't matter what current law says as the major tech companies may lobby passing a law to explicitly define the rules of the game; I mean if Disney could lobby multiple copyright laws to protect their interests, then the ML-heavy tech companies, being much larger and more wealthy than Disney, can do it as well.
But currently, first, there is a reasonable argument that the model weights may be not copyrightable at all - it doesn't really fit the criteria of what copyright law protects, no creativity was used in making them, etc, in which case it can't be a derivative work and is effectively outside the scope of copyright law. Second, there is a reasonable argument that the model is a collection of facts about copyrighted works, equivalent to early (pre-computer) statistical ngram language models of copyrighted books used in e.g. lexicography - for which we have solid old legal precedent that creating such models are not derivative works (again, as a collection of facts isn't copyrightable) and thus can be done against the wishes of the authors.
Fair use criteria comes into play as conditions when it is permissible to violate the exclusive rights of the authors. However, if the model is not legally considered a derivative work according to copyright law criteria, then fair use conditions don't matter because in that case copyright law does not assert that making them is somehow restricted.
Note that in this case the resulting image might still be considered derivative work of an original image, even if the "tool-in-the-middle" is not derivative work.