IMHO sharding traffic by subreddit doesn't pass the smell-test, though. Different subreddits have very different traffic patterns, so the system would likely end up with hotspots continuously popping up, and it'd probably be toilsome to constantly babysit hot shards and rebalance. (Consider some of the more academic subreddits vs. some of the more meme-driven subreddits — and then consider what happens when e.g. a particular subreddit takes off, or goes cold.)
Sharding on a dimension that has a more random/uniform distribution is usually the way to go. Off the top of my head (still speculating wildly and basically doing a 5-minute version of the system-design question just for fun), I'd be curious to shard by a hash of the post ID, or something like that. The trick is always to have a hashing algorithm that's stable when it's time to grow the number of shards (otherwise you're kicking off a whole re-balancing every time), and of course I'm too lazy to sort that out in this comment. I vaguely remember the Instagram team had a really cool sharding approach that they blogged about in this vein. (This would've been pre-acquisition, so ancient history by Silicon Valley standards.)
As for subreddit metadata (public, private, whatever), I'd really expect all of that to be in a global cache at this point. It's read-often/write-rarely data, and close-to-realtime cache-invalidation when it does change is a straightforward and solved problem.
The founding Reddit team was non-technical (Even Spez. I've been to UVA; it's not an engineering school; and spez had never done any real engineering before coming onto Reddit). They ran a skeleton crew of some 10ish people for a long time (none from any exceptional backgrounds. One of them was from Sun & Oracle, after their prime).
Same group that started with a node front-end, python back-end monolith, with a document-oriented database structure (i.e., they had two un-normalized tables in Postgres to hold everything). Later they switched to Cassandra and kept that monstrosity -- partly, because back then no one knew anything about databases except sysadmins.
Back then they were running a cache for listing retrieval. Every "reddit" (subreddit and the various pages, like front-page, hot, top, controversial, etc.) listing is kept in a memcache. Inside, you have your "top-level" listing information (title, link, comments, id, etc.). The asinine thing is that cache invalidation has always been a problem. They originally handled it using RabbitMQ queues: votes come in, they're processed, and then the cache is updated. Those things always got backed up, because no one thought about batching updates on a timer (or how to use lock-free) (and no one knew how to do vertical scaling, and when they tried, it made things even harder to reason about). You know what genius plan they had next to solve this? Make more queues. Fix throughput issues by making more queues, instead of fixing the root cause of the back-pressure. Later, they did "shard"/partition things more cleanly (and tried lock-free) -- but they never did any real research into fixing the aforementioned problem (how to handle millions of simple "events" a day... which is laughable thinking back to it now).
That's just for listings. The comment trees are another big bad piece. Again, stored un-normalized -- but this time actually batched (not properly, but it is a step up). One great thing about un-normalized databases and trees, is that there are no constraints on vertices. So a common issue was that you could back-up your queue (again) for computing the comment trees (because they would never get processed properly) (and you could slow the entire site to a crawl because your message broker was wasting its time on erroneous messages).
Later, they had the bright idea to move from a data center to AWS -- break everything up into microservices. Autoscaling there has always been bungled.
There was no strong tech talent, and no strong engineering culture -- ever.
-------
My 2 cents: it's the listing caches. The architecture around it was never designed to handle checking so many "isHidden" subreddits (despite that they're still getting updates to their internal listings) -- and it's coming undone.
I've been parts of 2 different engineering teams that wrote crap that I would cuss out but were super succesfuly, and most recently I joined a team that was super anal about every little detail. I think business success only gets hindered on the extremes now. If you're somewhere in the middle, it's fine. I'd rather have a buggy product that people use than a perfect codebase that only exists on github.
I could easily be wrong though, I haven't done web development for years.
You can’t treat the scaling as a binary feature, that it does or doesn’t
You start getting a lot more DB queries than usual, bypassing cache, the DB's remote, it's clustered or whatever, now you've got a ton more internal network traffic and open sockets, latency goes up, resources start to get exhausted...
Speculation, but having major subs private change the load profile which may result in the outage. Reddit certainly wasn't optimized for this.
If you search for r/news for example, it's nowhere on that page.
This is likely shifting load in very unpredictable ways... I'm sure a sibling comment is right that it's probably less load overall in general, but it'll be going down codepaths that aren't normally being exercised by >95% of requests and aren't working on the assumption that virtually all content is being hidden.
There's probably some microservice instances that are currently melting themselves into a puddle until they can deploy additional instances, additional DB shards, or roll out patches to fix dumb shit that wasn't dumb until the usage assumptions were suddenly inverted. Meanwhile there's tons of other instances that are probably sitting idle.
Reddit goes wrong often so I expect this outage could have any number of causes.
That's not necessarily true. Perhaps the status of subreddits is cached (because there's no reason to hit the DB 100 times/second to check if r/funny is private or public). But for a given request to a private sub, it would need to check each user.
EDIT: to clarify -- a big stretch to connect editing comments (which was well-known to have happened) with other actions.
https://www.theverge.com/2016/11/23/13739026/reddit-ceo-stev...
Not true! He did not openly admit to doing this. Instead:
* he did it, and told no one at all
* was caught doing it, after people figured it out
* then, admitted he did it
This is absolutely not the same as "openly admitting it".
He did secretly edit comments.
Still misleading, I agree.
> was a mod of r/jailbait in 2008
not the comment editing
In either case, it’s easily as speculative as the parent comment above, maybe slightly more so, since the parent came from Twitter.
He did not admit to it until evidence was compiled and hit the front page, and let it appear for part of the week to participants and onlookers as if there was massive internal strife.
Steve Huffman has reportedly told employees that Selig threatened Reddit. Selig posted a (perfectly legally recorded and disclosed) call recording showing the alleged “threat” was a misunderstanding over which the Reddit employee on the call apologized immediately.
Huffman serves in an official capacity at the Anti-Defamation League. People are (rightly, I think) critical of his handling partly in light of that.
Fabricating? Or do you have some reason to believe this is true.
The wayback machine doesn't list him as a mod (SFW): https://web.archive.org/web/20100420183707/http://nl.reddit....
PS. Wow, the sidebar makes that sub seem even scummier than I assumed it was.
That said, that meme is propaganda and defamation. Anyone could add anyone as a mod anywhere for a time, and violentacrez added spez "for the lulz" or however people like that think. I'd be beside myself it there's evidence that he was an active moderator working on that sub. Although, given the prevailing winds of the modern left, in a few years he'll probably be trying to claim that he was.
Obviously I can't reference the comment in question right now, but I'll try to remember to circle back and add a reference when(/if?) Reddit comes back up.
Edit: /u/andrewsmith1986's comment can be found at https://old.reddit.com/r/dankmemes/comments/1477psa/all_3_ar...
Consider: spez was the voluntary "mod" of the entire platform as CEO, and he maintained that subreddit.
Excusing him for the unsolicited mod invite is just optics management. It would be like saying lowtax had no responsibility as co-signer for the existence of subforums that his paid or unpaid staff maintained.
This would likely be like the use of underage subjects in nudist art, painted, drawn, or photographed. Such art is generally not considered pornographic and are legally protected, and some even displayed in museums and the like, yet websites will still ban the material to not have any relationship to it and to not have users trying to push the boundary. I'm speaking to the extent that rules are enforced, many websites have issues with enforcing rules in general due to the amount of user generated content being much larger than the amount of moderation available, so there will be something slipping through moderation from time to time.
Most states in the US could use a photo of a teenager in a bikini as enough 'evidence' to bring charges of Possession of Pornography involving a Juvenile, depending on what the actual photo depicts. Whether the contents of the image could be found to substantiate a conviction for the charge would be a trial/appeals issue. Nudity is not required for an image to be considered CSAM in most US states (or at the federal level), there are also Federal precedents that make cartoon depictions of under-aged characters count as CSAM.
It may actually be a defense.
https://en.wikipedia.org/wiki/Shouting_fire_in_a_crowded_the...
I think it helps strengthen the argument rhetorically, since people can't as easily use the existence of such "exceptions" as an argument for adding more, or bypass the principle entirely with slogans like "freedom of speech isn't freedom of reach". (Suppressing the "reach" of certain ideas obviously does inhibit their free exchange.)
This approach hasn't been "battle tested" yet though so we'll see how it goes in practice.
Nudity... depends on the purpose but probably not. I agree that's unfortunate if your stance is that there should be no restrictions on porn, but I'm not sure the arguments for why freedom of speech is a good idea really apply to porn in the first place. I think it'd be better to make that argument on its own merits rather than try to conflate the two.
That seems very open to interpretation to me; it seems to me these kind of things express a "feeling" much more than an "idea", and they might also be considered "antisocial" that you mentioned in your previous comment.
I meant "nudity" only in the sense of "nudity", nothing more. e.g. "I want to make a nudist TV cooking programme", or stuff like that. No concrete "ideas" are being exchanged with that as such.
I'm not a free speech absolution by any means, but I have generally favoured the exact opposite: "free expression" instead of "free speech" as that covers so much more. I think we can have an expansive "free expression" which includes many things while also having reasonable limits on that based on e.g. "does this reasonably harm people in a significant way?"
I suppose there may be room for both. "The freedom to exchange ideas" is after all a subset of "freedom of expression" (though not necessarily of "freedom of expression, with a bunch of exceptions").
That some people will try to abuse this seems inescapable no matter what; we'll still be argueing the details 200 or 2,000 years from now because there is no way to capture any of this in clear neat rules. The best we can do is come up with some decent set of ground rules which convey the intent and purpose as best as possible. This is why we have judges to, well, judge, and "reasonably harm people in a significant way" seems like a lot clearer of a guideline for this than a much more vague "ideas".
Flag burning wasn't protected as free speech in the US until 1989. I have a list of stuff that was banned or censored in the past that would be considered unobjectionable by almost everyone today, and I suspect things would have been better if we had "freedom of expression" instead of "freedom of speech" (or "free exchange of ideas", for that matter).
The issue I see developing is that any attempt to carve out what is not desired by some group is going to create standards that will let other groups carve out what wasn't intended in the first place. Look at how encryption is under attack and one common way it is attacked is by claims of how it promotes the spread of CSAM. So government asks for reasonable backdoors that will only ever be used to stop such material, yet tech circles realize that any such backdoor will allow for arbitrary power to block any material.
It also makes it clearer what's going on when people are waving the "free speech" banner over things like harassment and abuse. Allowing abuse not only doesn't increase the free exchange of ideas. It also often directly decreases it because it drives off people who get targeted by racism, sexism, et cetera, ad nauseam.
It just starts from porn and ends in gas chambers. And in the middle is freedom limited to "meaningful" contents and activities. If you looked up and compared state of freedom of speech, CSAM/CP regulation, internet censorship, especially authority intervention, crime rate, and standpoint on democracy-totalitarian axis for various random nations, it should all line up well against each others.
But if you try, please start with something more related to what I actually said, thanks.
It does infuriate me when some people may use "freedom of speech" as their excuse for "You must let me have a place to speak", when that isn't even guaranteed in the first place.
What do you need in order to have a free exchange of ideas? Oh that's right, free speech. Free exchange of ideas is one of the benefits of free speech. But it isn't free speech. Also what about speech to entertain? What about speech to criticize? What about speech to just mindlessly express yourself?
> like spam or antisocial behavior
What counts as antisocial behavior? Rap music? Heavy metal? Is protesting government antisocial behavior? What about criticizing politicians? And more importantly, who decides?
It's amazing how little people know about free speech. It should be mandatory to have a class on civics and of course free speech. People have such a childish and superficial understanding of free speech. And of course these people always tend to be for censorship.
But there's a lot of "speech" that should be and is already prohibited, both legally (CP being an obvious example), or through social standards of conduct (screaming expletives at random passers by is liable to get you kicked out of just about any venue).
In my view, focusing on the goal of freedom of speech (which is to say, the free exchange of ideas) rather than on speech itself communicates much more clearly on why the principle is important and where the line is. It makes it obvious why CP is not legally protected but Nazism is. Both are despicable, but one needs to be protected in order to preserve the free exchange of ideas while the other doesn't.
And again, I'm not saying other forms of expression that don't convey ideas shouldn't be allowed; just that the reasons why they should or shouldn't be allowed are separate from the reasons why the free exchange of ideas is important.
Yes, this wording does narrow my argument somewhat. But I think it largely narrows it in ways that make it easier to defend rhetorically, especially when trying to apply the principle to non-government entities such as the large social media conglomerates that own our modern town square.
Absolutely not. I'd argue that anyone should be free to talk with others about their opinion, but that doesn't mean I agree with that opinion. And letting then speak without shutting them down doesn't mean I agree either, just means I agree that they should be able to speak freely.
What kind of dystopian viewpoint is that? You go around stopping everyone from saying stuff you disagree with?
Sure, when “talking” is done via sound waves or someone else’s medium it’s different
But if you have free-speech parties at your place and people spray paint messages on your house then you are now “hosting”
You decide which words you leave or paint over
A better analogy would be, imagine you rent your house to someone else. You make a rule that tenants may display political messages in their windows, but only for one political party.
That would be illegal. You can prohibit all signs if you want, but specifically choosing what signs someone gets to display violates their first amendment rights and could trigger a fair housing lawsuit. It doesn't matter that you aren't the government and that you own the property.
I _think_ I agree with that. Don't hold me to it, but it feels right.
> but that doesn't mean I agree with that opinion.
Yup, sure, agreed.
> And letting then speak without shutting them down doesn't mean I agree either, just means I agree that they should be able to speak freely.
There is a world of difference between "not actively preventing someone from speaking" and "setting up a system whereby someone's speech is enabled and broadcast". Casting this to the real-world - if someone's yelling their opinions on a street corner, and I simply walk by without stopping them, then no, that's not an endorsement. But if I notice them yelling, and walk up and hand them a microphone - or (more closely mirroring social media setups) I install a public-access microphone, and stand there observing who uses it without trying to control it - then yes, through inaction I have endorsed what they choose to do with it.
> What kind of dystopian viewpoint is that? You go around stopping everyone from saying stuff you disagree with?
In areas I control and am responsible for, yes. If a guest in my home started spewing (what I consider to be) unacceptable speech, then (depending on my history with and pre-existing respect for them), I'd either take them aside and ask them to reconsider their choices, or jump straight to asking them to leave.
You can be a proponent of free speech and not allow people to stand on your porch yelling heinous things, but that's not what Reddit was doing. They knowingly profited from that speech.
That's an excewptionally broad definition and would sound pretty scary with no context.
That's an exceptionally broad definition and would sound pretty scary with no context.
That's nonsense. The Sears debacle showed that reddits leadership team was fine with deleting posts if it was going to cost them money to not delete them.
That 'political commitment to free speech' sure disappeared quickly when r/jailbait and u/violetacrez hit the main stream media.
spez was fine with hosting a community of child predators because it was one of the most popular subs. It was the top recommended result when you searched for reddit on google.
You can support free speech without actively providing a community for predators
reddit used to be owned by Conde Nast. Sears got upset about a post and complained to Conde Nast, who then told spez to take it down. If you have a political commitment to 'free speech' that folds if you might have to face some consequences for defending it, you don't have that strong of a commitment in my opinion. Certainly not strong enough to justify hosting a community of child predators
https://www.techdirt.com/2009/08/21/reddit-sears-grills-that...
They end up conforming or losing money. There’s no one reason for this. You try to run a website visible to the world, you’re gonna be subject to a world full of reasons.
Surely this is a meme by now? Any CEO that has ever said this about a website they control is just pandering to the crowd. Musk's Twitter has complied with more government takedown requests than the previous regime had.
If anyone would say it and mean it, it was probably this guy. He was quite the activist before he committed suicide.
It is though. 230 be damned. These were not small or hidden communities. They were frequently on the front page. Generally, and especially in this case silence is violence. The optics of that sub and the frequenters thereof are terrible. Do you want to try and justify their inaction further or concede this point? It should have never been allowed in the first place. Spez/Reddit at al should continue to be shamed for their long-standing tacit approval of these communities. Earning respectability requires public contrition for bad decisions that affect the public and non participants. As is typical, the communities were only shuttered when the victims cries grew loud enough to affect their brand image. Cf fph, wpd, fappening, t_d, Boston bomber fiasco, all the racist subs, and countless other controversies that spez/Reddit fumbled. Reddit deserves to close. The management team is evidently not competent or mature enough for the task and has repeatedly proven that their inability to learn from their mistakes and failure to become the proactive stewards needed will result in preventable harm to people who do not even use the platform.
Heh. So what did they moderate? things besides jailbait? How free!
I get that it might be different in the heads of the people who have worked very hard to create those platforms. I'm just not seeing any different in its effects on the world or on the targets of the racial harassment.
The difference is intent. Intent matters. Intent is the difference between murder and manslaughter, or between a conspiracy and mere speech.
big win? whomever votes to silence the guy gets to judge.
Also, it seems wild to me that you think a small company means you somehow have less ability to supervise your employees. What's your plan if you hire a racist who wasn't dumb enough to post openly? Just let him go to it?
Now where I'm from (Germany), not just "hate speech" is against the law, but it's also unlawful to insult another person. It's complicated, but for the latter it's mostly sufficient that they feel insulted by what you said to them personally.
Now while I don't go around insulting people in person or on the Internet, I personally think - for instance - that it should be allowed to call a person an asshole, if they behave like an asshole. Yet, if I did that here, or even online to another German person, they could go to the police and press charges. If the public prosecutor is sufficiently bored, this very low barrier could also be used to dox me in an otherwise reasonably anonymous setting, since the resulting lawsuit could result in my data getting subpoenaed from, say, Twitter and my ISP. This has happened to other people here in the past.
Now while I'm neither in favor of either hate speech nor randomly and viciously insulting people online, I consider the law in Germany as outlined unreasonable in an online setting. I think freedom of speech is more important fundamentally than another person's right to not feel hurt, or for some powers that be to silence or punish me because I said something inconvenient that they merely claim to meet some of the criteria for speech that is restricted here.
Mind you, this is the case all the while freedom of speech is enshrined in the German constitution as well. But I think it is a pretty good example of why I think freedom of speech should not be curtailed just in the name of another person's feelings about said speech. Even if a person, as you do, doesn't see a direct and tangible benefit in allowing that kind of speech, I would argue that a larger fraction of people are against disallowing it, because of the indirect consequences and where that line of lawmaking leads.
Another thing to consider is this: Say you're modestly happy with the current government wherever you live, and you'd be happy for them to have an "easy" way to curtail freedom of speech. Would you also be happy for the opposing political side to do the same thing? What if some extremists came to power?
This kind of reasoning is why free speech absolutists are so staunchly defending freedom of speech, even if it may be inconvenient or insulting to themselves or others.
You're also shading over exactly who gets free speech. If digital Klansman get to freely harass black people, many of those black people will not participate in public spaces, silencing them. Indeed, that sort of ethnic cleansing is often the goal of racial abuse. See, e.g., Loewen's "Sundown Towns". So whatever "free speech absolutists" think they're up to, in practice the result is often a diminishing of the free exchange of ideas that the Founding Fathers were clearly pursuing.
There is probably a line. But you don't know where it is and neither do I. You and I might agree that X is to one side of that line, but if we ban that behavior, then we have initiated a process that we might call line-discovery -- the search for the line that X was to one side of -- and line-discovery is highly prone to outcomes that result in bans on content from the other side of that line. So we don't want to engage in line-discovery, even though there are obvious examples of things to one or the other side of the line.
You may think you can ban the obvious things without ultimately engaging in line-discovery, but, the argument goes, you are mistaken. You will ultimately find yourself doing line-discovery.
You start out with obvious-sounding prohibitions on racism and hate speech, but eventually you're arguing about, say, whether it's racist to report on polling showing that violent protests are unpopular. [0]
And that's because banning any speech always leads to line-discovery.
So it comes down to a question of which scenario is worse:
A. You ban obviously bad stuff while accepting some risk of banning things that aren't actually over the line.
B. You privilege all content to avoid that outcome.
Some people are outraged by this framing and think it's obvious that you would want to risk banning some behavior to the right side of the line if it means eliminating the most obnoxious speech. But, basically, that is not obvious to everyone, no matter how many times they are reminded that there is some really bad stuff out there. [1]
[0] https://reason.com/2020/08/27/protests-violence-david-shor-k...
[1] Interestingly, this is really not so different from the argument about evidentiary standards for punishing criminal behavior, except in that case the politics are flipped. There conservatives would rather risk punishing some innocent people if it means the absolute worst actors are guaranteed harsh punishment, but liberals think it's worth risking some amount of literal rape and murder in order to prevent punishing the innocent. So I think, actually, both sides are entirely capable of seeing this from the other side; they just don't want to.
Let's grant that the harms of the kind of speech you're worried about are exactly the same in either case. [0] Platforming "racist harassment" because of a political commitment to free speech implies that other forms of controversial speech will get the same treatment, preventing the kind of line-discovery I described in my previous comment.
"Platforming racist harassment because you just kinda like racist harassment" leads to who knows what. All we know about that person is that they like racist harassment. Maybe other stuff gets banned. Maybe not. Either way, it's unlikely to be in service of avoiding harmful second-order effects.
So that's an enormous difference between the two motivations. In the first case the position is in defense of an ethic of open dialogue and an attempt to prevent second-order effects that are harmful to that dialogue.
In the second case -- who knows.
It seems to me that the first motivation is much more likely to prevent the kinds of second-order effects I'm worried about and that distinguishes it from the second one.
The notion that reddit ever was a bastion of free speech is absurd. They didn't "light touch" on upskirt, revenge, and kiddy porn because of "philosophical and political commitment to free speech", they did it because they didn't want to accept any more responsibility for content than they absolutely had to, and that's because it is not financially viable to moderate large communities using paid labor. That is why you see so many social media companies pushing against rules for online content; not because they're champions of free speech.
If it were about "a commitment to free speech", they wouldn't allow completely unaccountable and anonymous members to delete content, silently mute users, and ban users....including employing automatic tools that would ban people preemptively based on subreddits they posted in, or automated tools for powermods to ban someone across all the subreddits they moderated.
If you pissed off a powermod, your account could end up getting banned from nearly all the major, common subreddits - not just from theirs, but they'd communicate in private channels to other powermods that they wanted someone to be banned elsewhere.
Oh, and they were happy to moderate, severely, anyone who revealed any personal details about a reddit user. Which conveniently helps protect people doing stuff like upskirting and posting revenge porn.
"philosophical and political commitment to free speech", my ass.
I'm not naive, user content is hard to moderate. But is it hard to say "any subreddits with these keywords go on a list for review"?
https://old.reddit.com/r/TrophyWiki/comments/mohrlg/reddit_t...
There is some whitewash in the comments there: "[violentacrez] received the trophy because all the work he did to moderate the site..." as if he got the award for keeping things clean, but consider that he contributed the vast majority of those subreddits' content himself by cruising social media for salacious pictures of minors to share while he was in his 40's, and the award is named "Pimp Daddy."
Remember when Phrack was added to the LOC ... and was over 7 years old already (http://phrack.org/issues/42/1.html)?
More seriously, you really start to feel "ancient" when your body goes from "new year, pretty much the same as old year" ... to ... "new year, who removed vital organs and bodily fluids while I slept last night?! The bastards!!!"
Of course, it is a classic trope nevertheless ... and I did start to feel ancient even 15+ years ago ... but once you start noticing real changes, then you REALLY get that feel (and you know more is coming, lol): https://youtu.be/MqBNSMbEzI0
>Back in the day, you used to be able to add anyone as a moderator and it auto accepted.
>People would make shitty subs and add people, take a screenshot, shut down the sun or make private, then use that screenshot to start a witch hunt. Violentacrez could have added you as a mod of the sub and you'd be in the same situation.
>TL:DR I used to mod a sub with Barack Obama and Snoop Dogg.
Hyperbolic I know, but that’s what it feels like.
Interestingly, HN is being really slow for me right now and also gave an error when I first loaded it. Maybe something more global is going on, like network or cdn issues?
I had to reload this page 5 page until I was allowed to post a reply. Something is definitely going on.
HN seems to be groaning under the load too and might go down with Reddit.
A place I once worked had a 404 handler that was extremely expensive, but nobody noticed for a while because 404s are relatively rare. One time a vulnerability scanner took down the site because it was just hitting known vulnerability paths which all 404'd. The code that executed during a 404 was n^2.
Maybe it is a regional thing?
The official Reddit Status Twitter account proves that it happens very, very regularly: https://twitter.com/redditstatus
The official Reddit Status Twitter confirms this is a pretty common occurrence: https://twitter.com/redditstatus
The "elevated error rates" always presents as an "oops, you broke reddit!" landing even on old.reddit. I imagine since it is an "elevated error rate" rather than a total outage that it might be localized to geo or some other kind of shard. I'm on the US West Coast, though, so I can't imagine I'm in a minority.
(Which is confirmed by the number of people responding to GP.)
You'd get timeouts, and I'd never see them - despite being West Coast (Canada) as well. Or at least, that's my best guess so far.
I imagine it probably has some to do with specific geography as well. Cloudflare will proxy back to nearest node and maybe some are better than others.
Edit: I wonder if it’s because all the subreddits I’m on are low or medium traffic. I’ve unsubscribed from the front page and /r/all and tend to only read niche subs.
But yeah clearly others have a different experience with this. I wonder why.
Anecdotally I think it's more and that the threshold for "errors above normal" is probably set pretty high. It feels like their infrastructure isn't very reliable and depending on which backend Cloudflare is routing you to, YMMV.
I always just assume the uptimes absolute dogshit.
Source: I'm finally breaking a 13 year Reddit addiction.
> According to Reddit, the blackout is responsible for the problems. “A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,” spokesperson Tim Rathschmidt tells The Verge.
https://www.theverge.com/2023/6/12/23758002/reddit-crashing-...
For example, maybe it causes reads from the database take a lot longer than they normally would, locking up the database or causing the process the crash (again, that’s just pure speculation).
What happens when a user has 99% of their posting in subs that are now hidden, and the API is programmed to produce a fixed 30 comments of history on the overview page? The answer is extremely deep database pulls... you might pull a year of comment history to get 30 comments that aren't hidden. And depending on how they do that, it may actually pull the whole comment history for that timespan, since most of the time posts aren't hidden like this.
I worked at a backend team at work with some very overburdened legacy tables in mongo, and this is the kind of thing we'd think about. Yeah you can use an index, but then you have to maintain the index for every record, and change it every time a sub goes private/public (and we literally were hitting practical limits on how many indexes we could keep, we finally instituted a 1-in-1-out rule). And how often does that happen? Even deleted comments are overall probably a minority such that indexes don't matter, but, this is relational data, you have to know which subreddits are closed before you can filter their results, and mongo sucks at joins. And the mongo instance can become a hotspot, so, just filter it in the application instead for those "rare" instances. Even if they are doing it in mongo, the index/collection they're joining may suddenly be 100x the size, which could blow stuff up anyway.
edit: for me, one overview page is now taking me back one month in comment history. And I comment a lot on subs that are currently closed, so it could easily be throwing away 5-10 comments for every comment it displays.
E.g. many Google searches lead into private communities now. Word is spreading.
If they see a "this sub is private because reddit is dumb and also evil" they may come back later, they may not.
As for the forced reopening, beside the conspiracy this is something that could happen. It's a private company, moderator on strike are a loss of business, they would be 100% in their right to remove all the "traitors" (I'm not saying this would be a smart move, simply that if they really plan to go down this self destructive path it's the best time to do this and prove potential investor they still have control).
Less if you add "competent" to the qualifications list :)
Legally, of course. Morally, it is completely unacceptable. This isn't "oh they're jerks"; this is "the system is broken".
A meatspace analogy:
You host a weekly gathering at a restaurant. You decide to temporarily boycott the restaurant to protest some behavior of theirs -- your actions are a loss of business, _so the restaurant decides to host your weekly gathering without you_.
We'd never accept that in the real world, but for some reason we do online -- we fall back to the legal argument that It's A Private Business (which is true) and completely ignore that Reddit doesn't own the community, that the community doesn't _belong_ to Reddit. They own the platform (the restaurant); they don't own the community.
My first thought this morning was "if I was reddit, I would re-open all closed subreddits and instate my own moderators"... I'm kind of surprised they didn't. I'd have to imagine that if they wanted to, the attempt wouldn't fail - this seems like it could be done almost trivially.
> I think it's extremely unlikely that this is deliberate. The way that Reddit builds "mixed" subreddit listings (where you see posts from multiple subreddits, like users' front pages) is inefficient and strange, and relies heavily on multiple layers of caches. Having so many subreddits private with their posts inaccessible has never happened before, and is probably causing a bunch of issues with this process.
https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...
however, they have stated that may do this if the protest extends beyond the 48h mark.
Others look like they got the hostile takeover and a new mod.