My usage pattern is almost the exact opposite. If I like a story and find it relevant, I upvote it in part so I can easily reference it later.