As long as your Google account remains secured and no one manages to sneak a 2nd Google Account on your phone, that is. Find My Phone is a real time tracker that can be abused e.g. by spouses or stalkers. IIRC only triggering the "audible ring" leaves a trace on the phone (a notification + the obvious sound), to check if someone used FMP for tracking one has to dive deep into the Google Activity Logs.
Only way to protect yourself is, of course, to disable GPS on the phone - but what if you want to use Maps or another navigation at the same time?