As a kid I did this every day for years and see absolutely no issue with it (-5 to +25c instead of 40F), all year long in any weather. My parents either brought me by bike or I went on my own when I was old enough.
And that's even before you consider what a game-changer e-bikes are.
Weather is a non-issue. The main problem is fear of cars. It's a classic tragedy of the commons / race to the bottom for "safety" with giant SUVs. Maybe ADAS systems are one way out, but real safety ultimately requires safe bicycle infrastructure and scaling back the number of cars on the road.
Weather is a big issue. It is impossible to put kid into buggy/trolley in rain without it getting wet. 1 year old with pneumonia is a horrible thing. Add shopping in rain etc...
Bikes have horrible safety record on their own, even in "safe" bicycle infrastructure. Crashing into pole on bike and in car is a big difference. Common bike-buggies have zero protection for kids.
Crashing into a pole with your kid on board? A) don't do that? B) you'll all be fine as the speeds you should be riding with children along.
Not sure what you mean by a "bike buggy", but bike-mounted seats do provide protection against falls, and bike-towed trailers are essentially immune to them. Longtail cargo bikes put the children in the rear and have a bar that also protects during a fall. Box bikes are very difficult to overturn completely and also provide protection.
If you want to talk safety records, cars are by far the worst. They're extremely dangerous for the occupants and everyone around them.