We found the same—using OSM data that just had points was problematic, both because of accuracy and because it tended to often be the front door, not the centroid of the object.

I believe we limited our model generation to selecting places that had outlines, and computed the centroid of that outline. One of the benefits of our technique is that we didn't need to be comprehensive—we can throw out lots of places and still have enough to be useful for the model.