But probably you get stuck in local maxima. E.g. if I am on Hacker News I want to find new and exciting stuff and not things I already know ...
There is no reason you have to select content purely based on the website it's from, or the words it contains, but it's a really good filter for getting rid of stuff you definitely don't like. I.e. if you filtered reddit this way, most images and memes would get ranked way lower for me. Someone else might have articles or news stories ranked lower.
But mostly I'm looking to automatically find people who have similar interests or voting patterns, and then selecting based on what those people voted for to see what you might like.