> I often tell my friends I run an app on my phone that captures my location 24/7 [...] But then I tell them all my location data is not sent to anywhere on the Internet
Your phone is on the Internet.
It takes only one attack (for instance, someone sends you an image which exploits an RCE on the image decoder and then chains into a privilege escalation exploit), or a careless mistake (like marking the wrong folder to be synchronized), or even an automatic update of the app (which adds a helpful "sync across your devices through the cloud" feature or similar), to have all that saved location data copied elsewhere.
You can't leak what you don't have; if you never saved your location history, there's no risk of it being leaked after the fact.