Regarding stories, I think off-topic ones should be downvoted/deleted. Off-topic being defined as "Most stories about politics, or crime, or sports, unless they're evidence of some interesting new phenomenon. Videos of pratfalls or disasters, or cute animal pictures. If they'd cover it on TV news, it's probably off-topic." Having this text or a link to the guidelines on the submit page would probably be nice. (see http://ycombinator.com/newsguidelines.html)
Regarding comments, there are also some guidelines and not following those guidelines clearly deserve a downvote. I personally think that downvotes shouldn't be used to "say that you disagree". I prefer that it be used to filter out useless comments.