I think it would help if a user's first few karma points could only be obtained through upvoted comments, and not submissions. Submissions should be disallowed until you reach a certain karma value, ensuring that you've contributed meaningfully to discussions for awhile before you can add your own. (This would avoid the all-too-common 1-karma-user posts of spam or stuff from Reddit that isn't even related to us.)
I don't think invite-only is a good thing. It probably would have prevented me from ever being here.