Lately I've been wondering whether I should withdraw most of my money from the bank. Any serious drawback to that approach? For one, I don't like my transactions being tracked and two, find it outrageous I have to pay various fees to use/access my money while it is used to enrich others.
You should be quite able to find a bank or credit union that will hold your money without charging a fee. Where are you?
You can still use cash to avoid tracking, and you can even thwart forensics by making regular uniform withdrawals. The neat thing is that patterning like this makes you less susceptible to analysis but far more susceptible to getting mugged because now we know that you pull $400 every Tuesday at 3pm. Pick your poison.
Where are you banking? My bank pays me interest, even on my checking account, and I don't need to worry about a burglar taking all my savings. There's no reason you can't keep your money at the bank, stop using credit cards, and pay everything in cash; all the bank will know is that you withdraw $200 every week.