This is a common refrain, but I've never seen more than one-or-two anecdotal accounts of this happening. Do you have citations or articles that support common CPS intervention in cases like this? In my experience CPS is primarily focused on abuse and malnutrition, and tries everything possible to keep families together.
Parents act like the world outside is scary and dangerous, despite declining crime rates. I think that many younger kids are in dual-income families, and are at after-school care from 3-6 every day. Those that aren't are often in after-school activities, and aren't home, bored with time to burn outside. If the kids are home, they'd rather play Fortnite or Call of Duty with their online friends than build a fort in the riverbed.