This is basically correct. If your airport is small and you never get in trouble with ATC or have an emergency situation, you will probably not get caught.
However, if you do get caught, it is a criminal charge with (IIRC) up to $250,000 fine and 6 years prison time.
Or worse, you overestimate your ability and die/kill people.
I wouldn't take that risk.
If you want to fly a lot, get a license. If you want to fly a little, make friends with a pilot.
You can even get a student license with relatively few requirements, and fly solo (no passengers) during the day.
---
That said, I do sympathize with people wanting to opt out of the FAA's draconian rules. (E.g. If you have been professionally diagnosed with ADHD, the FAA bans you from any pilot's license ever, even a student license. :/ Lovely)