No, a Gadget does not just get automatic access to your data. It only gets access to the things you grant it explicitly.
And if you want real life examples, every single LLM provider has gotten insane amounts of data shared to them, even if you have to approve.