Anecdotally, I've only ever heard the name Fermi in the phrase Fermi Paradox. Oppenheimer I know much more about, and I only know there's a movie about him because it's all over HN this month.
I'm sure that the red scare did tarnish his name for decades, but it looks to me like those decades had ended by 2004 when Google's trends pick up.
[0] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
My context is from being outside of the US though.
Case closed.
Plenty of folks who aren't physics geeks have heard of the Fermi Paradox.
Source: Am not a physics geek.
google trends seems to agree: https://trends.google.com/trends/explore?date=all&q=world%20...
Whereas Fermi (in the context of the Fermi paradox) is kinda foundational, if you're in the extraterrestrial life field.
Oh also Leo fucking Szilard! From wikipedia ...
> He conceived the nuclear chain reaction in 1933, patented the idea in 1936, and in late 1939 wrote the letter for Albert Einstein's signature that resulted in the Manhattan Project that built the atomic bomb.
Russians, Sting, 1985
-- so I would say Oppenheimer being the "inventor" of the atomic bomb was (literally) pop culture