Yeah, used to live in Japan and it was quite annoying.
The 'building number' having no relation to location is fairly useless. I generally had to explain through landmarks/obvious buildings where I wanted to go.
The whole system is pretty useless for giving directions.
It's better now with Smart phones (wasn't a thing when I first moved there), but still bad. Korea uses the same system afaik.