I think (and I don't live in the valley although I did work there for a short time) there is truth to what you are thinking. In the end of course the work that you do and who you are matters. But I do buy into the idea that it shows how serious you are in the same way that people in finance flock to NYC or people in entertainment go to Hollywood to seek fame or fortune. Of course you need to be somewhere that you will be happy and can live as well. So not a good idea to push yourself to somewhere that isn't right for you.
I think the previous response is correct in that there is definitely a halo effect that makes one seem more credible. Example was an article I was just reading entitled something like "Ex-Googler Gives Life Lessons" and it turned out the person was 25.