A few millions are sufficient for you to be able to say "fuck you" to any employer or person you theoretically depend on, while still being able to live a comfortable live and - possibly - defend your right in court, even if it does not make economical sense to do so. That's what most people mean.
Having a few billion gives you some additional fuck you privileges - police officers, world leaders, other billionaires -, but you probably won't get on a level where you can say that to anyone and not fear repercussions. Also, if you can live happily on your own, you can already tell them that if you have a few million to your name.