I've been kicking a design around in my head for the past few weeks which retains most of the outward form of something like Reddit, on the grounds that it has proved to be a model that can attract users, but redefines the upvote and downvote process such that an upvote now means "I want to see more of this poster's comments, and transitively (and weaker as the links go out) what that user sees" and a downvote now means "I'd like to see less of this poster's comments and transitively what they upvoted". The difference here being that upvote no longer means "I think everyone should see more of this comment", but now "I think I should see more of this commenter".

I think it ought to be unidirectional to prevent obvious attacks. I'm not too worried about the "filter bubble" because I think you are inevitably in a filter bubble, and it's on you to manage it, not expect random sites on the internet to somehow pop you out of it (they basically can't, by definition).

I think I'd still want categorizations for subject, just to give things some sort of focus. The result is that instead of /r/politics or /r/$ANYTHING being captured by some sort of single monolithic gestalt, multiple simultaneous gestalts could be running side-by-side. You'd want some deliberate mixing, both to keep things interesting and because I think you'd want to ensure some "heat" in the simulated annealing sense to ensure that things don't get too isolated and things keep moving. (Communities themselves keep moving, after all.)

The net effect of this is that while trolls would still exist, anyone not interested in trolling should naturally filter them out of their view relatively quickly, while the trolls get each other to troll at. Since they can not be eliminated, all you can really do is fence them off. One of the other interesting side effects is that you ought to get some other interesting communities in there, too; the community of people who thoughtfully comment, the community of "doers", etc. One interesting possibility is that this may even allow "celebrities" to have a "social network" that nominally anyone can join, but requires sufficient social proof that it isn't just "anybody with a Twitter account and an itchy trigger finger can get the first reply below $CELEBRITY's tweet.", wrecking the utility of social media for the celebrity. I haven't quite solved the question of how to initialize the communities once the system gets going, but it's probably feasible.

What stops me from coding this up is that whenever I start really fleshing this out into an implementable system, I realize it violates one of my life rules: "Never engage in an endeavor in which the worst case scenario is complete success." I don't want to run a community site like that. So for me personally, the outcome matrix boils down to "Failure -> waste of time, Success -> oh shit". Because even if this did miraculously fix all social issues, which I am hardly naive enough to suppose, there's still legal compliance, DMCA compliance, etc., all sorts of things I personally have no desire or drive to do.

However, by all means, feel free to "steal" my idea. While I am by no means an expert in this field (the aforementioned paragraphs are pretty much the sum total of my thinking at this time), I'll even spot you some free "consulting" in the form of playing intelligent-rubber-duck to bounce ideas off of if you'd like. And let me know if you get to deployment.