The reason is super simple: Google demanded that if you index content on a page, that content must be freely accessible. You can't index what you don't show. EE made the hack to the layout. It was not always like that. I remember searching google help for ways to block domains (ee) from their results. Apparently that was the only reason anyone wanted to block domains :P