Doing inference locally is a significant improvement, but other issues remain, like prompt injection and data exfiltration.

Especially if the whole point is to have it search through a giant pile of arbitrary un-vetted stuff, like all the emails you ever received from anybody.