On a recent visit to SF, I was waiting for a friend to finish with work. He invited me to stay and hang out at his company cafeteria. When I declined, saying I'd rather go wandering, he and his friends looked at me like I was some kind of partially deranged maniac. I had a great time!
When I was in college I'd go into New York City and just wander around. I walked from the Empire State Building to the World Trade Center (the original was still standing). A cool place to walk in (though overwhelming and a bit dangerous if you don't know where not to go).
I hear you on vacation walking. I never understood guided trips when there is so much fun to be had with random adventure. Similar to you I enjoyed both Japan and Australia quite a bit.
I love hiking in the mountains, but when I lived in the country I often wound up replacing aimless walking with aimless driving.