It's not just inconvenience, although that is part of it. The transportation department would have a huge added burden by needing a way to determine which houses have children, a lot more signage to mark the designated slowdown areas, extra workers to make the adjustments as families move around, a way to update various mapping/GPS systems, etc. The number of passing zones would be drastically reduced. Jake braking would be a nuisance for those houses with children. And there are probably more side effects that I can't think of immediately.

I don't disagree with any intention of improving safety, but there are pragmatic reasons we shouldn't unconditionally implement reduced speed limits around children.