Yes, I do this exact same thing. I really don't want to store password on my side and I'm of the impression that sites like Google, Facebook, and Twitter do this better than I ever will and that most of the users visiting the site will have some kind of account on these sites. Ignoring man-in-the-middle attacks, if one of the above thinks the email is valid, I have no reason not to.

As a disclaimer, though, I've worked on highly specialized apps where the users belong to a certain organization bound by one of the above. For example, I made a website for my school and every student there has an email address that is tied to that school's Google Apps for Edu account, so I'm guaranteed that all valid users (students) will be able to log in via Google.