I'd say that as famous buildings go, the Pentagon is up near the top of my passive vocabulary - instantly recognizable and nameable. Washington, D.C. is rich in those.
But actively? No. If tasked with naming ten or twenty world famous buildings, I simply wouldn't think of it. Eiffel Tower, Sydney Opera House, the Pantheon (oh yes), and somehow, Word Trade Center, even if it is no more, but no Pentagon would occur to me.
[Edit: The usual typos]
I'm curious as to what other instantly recognizable and nameable buildings are in Washington D.C. to a non-American. I can think of the Pentagon and the White House. I can't really think of any others I would think a non-American would immediately recognize and be able to name. The Capitol, maybe? The Washington Monument?
I'm also wondering, with time, if the rebuilt World Trade Center will become just as recognizable. It certainly is striking in the Manhattan skyline, but it just doesn't seem "the same" to me for some reason I can't put my finger on.
The Washington Monument (the big obelisk thing?), the White House, and the Pentagon -- the last as much because it's used in the news in the same way that "10 Downing Street" is, but unlike a London street, the Pentagon is a unique, distinctive structure.
I do know the difference :-)