I usually use my google account to login to websites I care about (most sensitive websites I know provide this facility) and moderately strong but easy to remember passwords for non-critical websites that don't provide OpenID (e.g. a forum I never consult).
Of course I have 2FA on the google account and I feel rather safe.
Should I be worried?
The basic overview of how OpenID works is here: http://openid.net/pres/protocolflow-1.1.png
So if I understand this diagram correctly, you connect to the server that you want to log into. It determines whether or not you are authenticated (probably through cookies). If so, it redirects you to the OpenID server. The OpenID server checks your credentials and returns back to the original server. The diagram does not specify, but I'm guessing the OpenID server has a session cookie for your credentials so that you don't have to log in over and over again.
So what are the downsides?
1) The server you want to log in to knows about your OpenID. In other words they can potentially identify you and track you (possibly with the help of Google). 2) The OpenID server knows which sites you are going to and when you are logging into them. It can track all of your movements. I think it would be naive in the extreme to assume that Google does not do this ;-) 3) Without knowing more details I couldn't say, but I suppose it might open you up to Cross Site Request Forgery style attacks. My experience with Google code in this respect is that they are usually extremely thorough about this sort of thing, so I think this is unlikely on the OpenID side. In my sleepy state, I can't really think clearly enough to imagine whether or not there are ways to screw up the code in the server that uses OpenID, though.
My employer loves Google and we use a ton of Google services. I am lucky enough to have incredibly technically competent management and they understand the tradeoffs of having Google track everything we do. Because of that, I use OpenID for any other external service that my employer mandates/approves of. I'm pretty careful not to go to OpenID authenticated sites by clicking on links and open them directly, so I feel confident that I can avoid CSRF attacks even if they could be done.
I do not use OpenID for personal use. My employer may be happy to have me daily work activities tracked by Google, but as a private individual, I don't really want that. I use an offline password manager: http://www.passwordstore.org/