1) Terrorism. I know you mentioned property destruction, but intentional terrorism targeting humans is a real risk. Cars are already lethal weapons. Terrorists have used them to mow down dozens or hundreds of people at a time. Flying cars could become kamikaze weapons, or could be hacked into becoming them, with far more lethal potential. 9/11 was just a bigger version of this, so the precedent exists. Yet, there would be no TSA to protect you. Would we need civilian air defense turrets?
2) Noise. This would be huge, because rich neighborhoods that have frequent helicopter traffic are already flooded with complaints. If flying cars became commonplace, the noise issue would be dramatically worse than that.
3) Light pollution. We already have an issue of light pollution that makes stargazing and enjoying the night sky difficult. More importantly, it disrupts sleep patterns, which has all kinds of knock-on effects for society in terms of mental health, childhood development, accidents, and so on. Constant lights in the sky at night would be a nuisance that even high fences wouldn't block.
4) Inequality. Those who can afford the new vehicles, which will surely start out pricey, get to perform geo-arbitrage. They can buy cheaper land that hasn't yet factored in the new commuting realities, and yet make the same wages. Would be one extra hurdle for those trying to become upwardly mobile. One more huge debt burden to take on, while striving in the rat race.