Does google pay significantly more than, say, any old rails job? Perks don't really impress me. 20% time sounds interesting, but I hear it's gone. Why would I want to deal with the interview process?
If you want to know the best thing about working for Google (I work for Google), it's the peer group. Sure, the interview process misses a lot of great candidate, but it's really hard for a poor candidate to get through the doors. There's no one I work with who I think is dumb, or who I think couldn't handle anything I needed them to handle.
I think eventually you stop feeling impostor syndrome but it hasn't happened yet for me :)