You may not be on Facebook but you can bet your friends or family have shared your contact info with them. And even if that exposure is limited, you can bet they’ve got a shadow profile of you.
And Google has been monetizing that profile for a lot longer than Facebook has.
If Facebook is building a data portfolio of you, which is highly likely, then you can't avoid them, which means the problems that arise from the practice still impact you, whether or not you want them to.