You are being tracked, and your network is identifying you (tagging, mentioning you, posting photos that include you etc) whether or not you use Facebook.
This is like telling people who have privacy concerns about 23andMe / Ancestry DNA "just don't use them." Meanwhile their parents or siblings happily open up the floodgates.