To be fair, I checked my boyhood home in very rural Maine and it was correct for the size and shape of the multiple farm buildings.
OpenStreetMap and such have pretty good building footprints, sometimes enriched with building heights. Maybe that can be used for some correction
Looking at where I live and where I grew up the building heights are quite badly estimated.
- Some groups of houses around here that are more or less identically built but on sloped terrain are reported to have widly differing heights
- My neighbour building is reported to be half the height of this building (they're more or less equally high at 5 stories)
- A small office shack behind the neighbour building is reported to be taller than it (it's a single-story building, the neighbour building is 5 stories)
- The freestanding buildings on the farm where I grew up are like you said, badly combined, much of the estimation there seems to be dependent on shadows,etc.