It's disappointing to see this show up on HN without contact myself or the other rubygems.org admins. We've put rubygems.org into maintenance mode until the issue is fixed.
You can never be too sure. There's a whole contest (http://underhanded.xcott.com/) whose goal is "make something that looks legit, but actually does something evil / has an exploit).
Nuke from orbit.
It is better to be paranoid about it and get the word out before someone actually gets hurt. If the users get to it before the maintainer gets to it, at least someone got the word out.