Most buildings in OSM aren't actually tagged with numbers. (A lot of houses aren't even marked as existing buildings in my experience.) It should be have their street at least, but adding building numbers is just too time-consuming for outside of cities it seems.