We let our children roam and got a visit from the police because of it. Someone reported "unsupervised children". The police came to our door and started questioning us.
My wife just kept saying, "did we do anything illegal?" and "were they unsafe?".
The police ultimately admitted that they were perfectly safe and we did nothing illegal, but warned us that, "it just looks bad and you should be more careful".
We've also been scolded by other parents at the park when we let our kids go to the other side of the park 100 yards away. "How could you let them go so far? What if they get hurt or need you?"
It makes us not want to let them roam, just to avoid the hassle.