People are not concerned because it can, because as you say it needs to in order to operate if you opt in.

The concerns implied are that:

* You are automatically opted in, or you are nagged to opt-in in a manner that makes it easy to accidentally do so, as is not uncommon with these things.

* That the use of your data for training isn't just for you, it is also training any other models Google wishes to use it for.

* That once they have started, opting out does nothing: your data is already theirs. Not that wasn't already anyway, control of how your stuff is used is apparently a right reserved only for corporations.

You could instead use a trustworthy LLM API provider instead of an American one bound by legislation like the US CLOUD act.

Confidential compute is another option.