I would say it's pollution, crime and lack of avenues for small kids.
I was living on a skyscraper and the amount of black dust I was removing from next to the windows was mind-blowing. I still remember moving 1hr out of London and feeling like I was breathing again. Not what I wanted for my kids.
The amount of stabbing and theft going on was pretty crazy. It was all around you. Some parks are ok and touristy, some parks are straight up dodgy and you run into drugged up, drunk people, (homeless?) people harassing you.
There are almost no kids around so not super easy to find friends for your kids. Nurseries are small and crowded because real estate is at a premium.
Again, moving out from the city fixed all the problems. Town life in the UK is actually pretty great. Sure you will miss on the main pros of London, events and networking.
I'd recommend finding a nice town commuting distance from London to reap the most benefits.