I don't think this would be particulary hard either. For IE, the NSA can just get MSFT to do it. For Firefox, they can compile from source, and for Chrome, well, they can probably compile from source too, because they probably have access to the build source of Chrome, with or without GOOG mgmt knowledge.
Can anyone come up with a (technical) reason the NSA could not be doing this?
But if they did that to everyone? Surely it would be noticed. Probably very quickly. There are a LOT of smart security researchers scouring browsers for bugs and running them in carefully controlled environments every day. Someone would also eventually notice that the production binary doesn't match the version built from source, especially for open-source browsers.