Yes, you have to trust F-Droid to sign apps.

But that doesn't weaken the security model — since August2021, (new) Android devs can't sign their own apps anymore, Google holds the keys instead.

The security model already assumes a third party is signing the apps.