ArchiveTeam has saved over 11.2B Reddit links
old.reddit.com
old.reddit.com
I don't care about the reddit API but I am enjoying watching the this dumpster fire burn. Popcorn sure does taste good indeed.
I think my explanation is at least as plausible as yours with the added benefit of not denigrating an entire group.
Obviously there’s exceptions, but it is a really common phenomenon. They found communities or get modded simply because they spend the most time and hang out in the right chats, not because they’re objectively great neutral moderators. They are very frequently people with very strong personalities, and are often in the long-term have a more destructive influence on communities than any short-lived comment troll, especially wrt to the effect on newcomers as they tend to make communities increasingly insular.
>tend to get burnt out
People can also just overcommit to volunteer activities in general for a period. If people can just dial down their involvement, that can be OK. But if it's an activity that's all-in or nothing, it probably won't last for more than a while as people burn out or their priorities just change.
These same people, are just overstressed and take too much shit, and this explains the vast majority of all alleged abuses?
I've heard this before. I hear it every time someone is shot to death while crawling towards their attackers while being screamed at to not move and come closer simultaneously. Every time that someone dies with a knee on their neck for 10 minutes. Every time a grenade is thrown into a baby's crib.
That's about the most generous I can be. Reddit moderators have not actually thrown grenades into baby cribs. They're not quite as bad as those that do throw grenades into babies' cribs.
Is there a link to this? That's crazy!
If I were spez I would simply disable the ability to make new subreddits private.
This protest is not for the greater good, it's harming half a billion users for the benefit (actually, for no benefit since nothing will come out of it) of the 3% that want to not see ads and use the 3rd party apps.
It's not because a few terminally online Reddit addicts are vocally posing as the resistance that the majority of the community supports it.
Without comments, HN would be just a boring link aggregator and we'd get very little information if the article was BS or not. But because we have comments we get gems at times where 'the creator of X' discusses the merits of the article. That can be nearly priceless. Things like this draw people that don't upvote and don't comment, but they still get immense value from it.
Posts are what makes Reddit, so much so that Reddit created hundreds of fraudulent profiles in their early days to fake popularity.
https://arstechnica.com/information-technology/2012/06/reddi...
----
Of course this interests me what the future looks like for social media. At one time in the past you needed users to generate and post content. Could we end up with social media sites with 'good enough' bots faking humans that draw in the masses, but few biological commenters and posters would exist?
If Reddit does a mass replacement of mods, the illusion is broken. You're not running your own forum, you're doing free work for some website. So if you want to create a place to discuss X, then you dont think to make a subreddit for X, you go with something you actually control or far more likely just somewhere the illusion hasn't been broken; like create a discord server.
That illusion is what has made Reddit basically the forum. It's the whole value of the site. Destroying the thing that makes your site valuable is crazy.
Reddit's value proposition is not the ego stroking of the 0.0000001% that are moderators, it's the discoverability and interoperability between unrelated niches.
If little dictators don't get their kick from rulling lawlessly on a community anymore, I say good ridance.
What does this have to do with what I wrote?
>the discoverability and interoperability between unrelated niches.
Why are those niches on Reddit if Reddit isn't giving away faux forums?
>If little dictators don't get their kick from rulling lawlessly on a community anymore, I say good ridance.
Odd because I get the impression you prefer the communities those little dictators create to be on a google crawlable site over discord.
> What does this have to do with what I wrote?
You are suggesting that those users will go away from Reddit to form separate dedicated communities, and I am saying that they will try but fail to attract people.
Reddit allows the vast majority of the subs that participate in the blackout to survive simply because they are a part of Reddit and benefit from its infrastructure and features.
You really think a website dedicated to cute animals will attract 34M subscribers like /r/aww? And another one with the exact same theme will attract 4M subscribers (with a lot of overlap) like /r/Eyebleach?
Those communities exist and strive because the barrier to entry is literally non-existent. It takes one input to create them and one click to join them.
> Odd because I get the impression you prefer the communities those little dictators create to be on a google crawlable site over discord.
I would prefer if there were no little dictators, with elections every 3 months, showing detailed stats of the moderators actions, and most of the moderation to be in the style of StackOverflow, so community driven.
Sad.
This whole thing about third-party apps has been ridiculously mismanaged by the communities.
The only 2 reasons people want to keep third-party apps are 1) they prefer them to the official one, 2) they don't want ads. Both of those reasons are valid, but neither are even remotely close to justifying the actions that those Reddit nerds are taking.
[1] https://www.reddit.com/r/reddit/comments/145bram/addressing_...
I get not being sympathetic with petty kings of message boards, but let’s be real, Reddit is an awful company whose incompetence is legendary. They’ve failed to monetize, failed to maintain the user experience and now are failing to keep a vital aspect of their business going.
Ultimately their failure is a good thing. Reddit broke the internet by ending the phpbb era. Time for the next thing.
The notorious Digg collapse was in part because of their fight with a big poster who was dominating the rankings and in the process supplying a lot of the content. They won against him, and the rest is history. Similar with Vine.
Edit: not that I'm against paying mods, I support that. But replacing enthusiast mods with paid mods, I doubt that'll be adequate.
There's a bit in Predictably Irrational by Daniel Ariely where he talks about social compensation vs. market compensation (I could be remembering the terms wrong). He asks how poorly it would go if you went to Thanksgiving dinner with your grandmother, and after eating the amazing spread you handed her a $20 and said, "That's for the great food, gran, it was amazing!"
Gets rid of a nuisance relatively easy, but degrades the quality. But reddit doesn't seem to care much about quality for the last years so.
They don't need to. They do need to keep the stock price up just long enough to cash out and leave a sucker holding the bag.
I’ve never been an active Redditor, but there are people discussing my open source software on Reddit, so I’ve answered questions there on a good number of occasions.
That my informative and completely rule-abiding posts can be made unavailable at the whim of some community mods while the site is still alive feels like a betrayal. These mods don’t own my posts. No complaint if Reddit goes the way of Digg and takes everything with it.
My prior experience moderating other forums tells me the contribution of these community moderators are often way exaggerated and usually easily replaceable. Some form personality cults and periodic rotation would actually be a good thing.
The big subs that they would 100% do it on that are a few million plus, you would never really notice. /r/pics, /r/gaming, etc who even pays attention to who the mods are. The mods aren't the community.
If Reddit replaces volunteer mods with paid mods we would get a more consistent moderation and almost certainly a more professional experience. People getting banned for disagreeing with a mod would stop for example. You wouldn't have to guess what the mods mean by their rules. For example, on /r/startups replying to people giving them an answer and saying if you have any more questions I'm free is called an unauthorised ask me anything. Which is crazy. There are many subs where it's anyone's guess what the rules are. And it can literally depend on the mood of the moderator. There are some subreddits that automatically ban people who have posted in certain subreddits.
Moderation is an important job. It's needed. But I can't think of any other social site that has such a bad rep for moderation.
You can employ paid moderators, I don't think it's a terrible idea from a quality of user experience perspective. It's an awful idea from a "We're desperately trying to get this fucking company to IPO".
I can guarantee you the opposite. You'll have a site-wide abusable Scunthorpe-incompatible report-based automated moderation with no proper appeal mechanism in place because half a dozen underpaid interns/offshored employees will be responsible for taking over the work of hundreds of moderators.
It's already happening in some cases. There's ways to make reports go directly to the so-called Anti-Evil Operations team who will irrevocably override any moderation decision and enforce abusive reports.
It's easy to get people banned, post some hateful content, wait for reports, and then report the reports for report abuse.
> But I can't think of any other social site that has such a bad rep for moderation.
There are some legitimate cases of poorly moderated subreddits and mod abuse (and the whole powermod issue), but beware, most of the time people complaining about power-tripping mods and "not being able to say anything anymore" have been banned for very good reasons (those reasons being straight-up hate speech most of the time).
They can do whatever they want with their platform, but I can also not like mods who make decisions I don't agree with.
I also don't think mods should be allowed to ban people just for being subscribed to other subs or having posted there. The whole idea that I co-sign everything the sub stands for just because I read it tells you all you need to know: they expect and often demand that you do in fact co-sign everything the sub stands for.
And that's the fundamental problem with reddit, really. It's not just the keyword squatting mods: it's that it's a giant social experiment that distills the worst of mob behavior and anonymity.
Fixing reddit requires some kind of check on the mods and the elimination or complete overhaul of the karma system.
Did you mean "more advertisement friendly"?
Like no shitting on a big mobile publishers, no criticism of Blizzard, or EA, or other big corp.
Pretty much every other social site of note doesn't have a rep for moderation, on account of not having moderation at all. A solid 20% or higher of the YouTube comments I see are straight-up phishing scams, Facebook and Twitter are complete cesspits where only content that's literally illegal to post ever gets removed, and the less said about imageboards the better.
Wikipedia is the only even remotely comparably large site I can think of that actually has anything resembling moderation, and you'll find the exact same crowd criticising them for enforcing their rules as well.
I think you may be making a mistake....
Years ago I had to manage SMTP servers. Of course a huge part of that is dealing with spam. Users were mad about how much spam they got and asked if "I was even doing my job". In one particular users case, they did get a lot of spam, and it was a pain in the ass to deal with and they always had lots of complaints. So I showed them for ever 3 messages they received I blocked somewhere near 1000 messages to their address.
If you've never been on that side of the system you don't know what 'not having moderation at all' looks like, but I can tell you it looks far, far, far worse than YT psts.
In theory channel owners have the ability to handle that for YouTube specifically, but in practice the tools and incentives aren't there to make it actually happen.
That’s why they use the volunteer mod model, and why they keep us at arm’s-length, and mandate that we cannot receive any compensation of any kind from anyone for moderating.
That said —
The mod code of conduct gives them avenues for removing mods that violate it; there’s also neutral admin-developed tools that identify users who are already active in helping the community out as potential moderator recruits.
So mods that close subreddits maliciously — with an intent to damage Reddit or to demand that they disburse money to a third party — could be removed from mod privileges, and replacements found.
Now, reddit may SAY that this is the case as an excuse, but it isn't the truth.
I've been contributing to this project for ~2 years now and I've never seen it running so fast
https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior
It appears to make you into a sort of node that backups up a share of reddit and sends a copy to a group that's archiving all of reddit.
> Can I run the Warrior on ARM or some other unusual architecture?
> No, currently we do not allow ARM (used on Raspberry Pi and M1 Macs) or other non-x86 architectures. This is because we have previously discovered questionable practices in the Wget archive-creating components and are not confident it runs under different endiannesses etc. If you still want to run it apparently Docker can emulate x86_64.
https://dustinrue.com/2022/10/colima-cpu-architecture-emulat...
colima start --profile amd64 -a x86_64 -c 4 -m 6
docker run --name archiveteam --label=com.centurylinklabs.watchtower.enable=true --restart=unless-stopped atdr.meo.ws/archiveteam/reddit-grab --concurrent 1 [username]
Reddit's value is in the discussions (the links are usually shared on all social platforms, so that's the only differentiator).
So what value is there in a collection of links going back years (meaning most of them are likely already broken)?
> By Reddit links I mean posts/comments/images, I should’ve been a bit clearer.
I also use Reddit for tech updates and discussions from subreddits like r/rust and r/ProgrammingLanguages, but most of these already have alternative sites (e.g. discourse, Discord, SE), and I’m more hopeful most of the new posts migrate to one of these than other subreddits or Reddit migrating in general.
I've seen Discord used as a platform for project-based discussion, and I just can't get into it. It just seems so fundamentally not built for long-term threads and the aggregation of information (like a wiki, for instance), and it kind of frustrates me to see some projects that are hell-bent on keeping all related discussion on Discord.
Granted, Reddit is not the greatest place for what I'm talking about either, since it also prioritizes newer self-posts/links, but I feel the clunkyness is even worse in Discord.
SELECT * FROM fh-bigquery.reddit.subreddits
There are torrents available it seems for comments that cover later times than that.
The rest of the data can be found on AcademicTorrents.
(Also, is this including Reddit-hosted images/video?)
The first rule of social networks is you can't touch them or they will die.
Proof: Aol MySpace Twitter Facebook (Zuckerberg won't touch it now) Reddit
This is obvious except to billionaires.
Now for the news: the only content you see on social networks is designed to reinforce your supposedly persistent self and sell ads.
There is no other purpose for social media.
All social media content on the internet since the internet began will be deleted, lost and forgotten as there will be no profit motive to do otherwise.
"icanhazcheezburger.com"
These things and by things I mean the dumpsterfication of social media content or internet content because it has no value.
Those same things don't "influence elections" and traumatized children and supposedly destroy society.
Social media is a fancy dumpster fire and that's all it will ever be. It burns out and starts anew from some other source.
Saving content from forums is the same idea. Reddit is just a collection of forums.
Do you think that, at no point in the future, it would be valuable to know how regular people thought and what they talked about today?
There were TONS of comments on HN about how reddit is still safe place for product reviews (or just opinions).
Plus they are kind of forums where niche knowledge can be found.
I was merely saying the data set is valuable or they wouldn't have trained on it. GP was saying the data was worthless.
Ill use a simple example to illustrate.
If you (yourself) were to write a song that sounded just like a Beatles song and was in effect a lift or "copy" of it and it went to #1 on the charts. You would expect a Beatles lawyer to contact you pretty quickly.
Thats called "one to one" copyright infringement. Lawyers and court systems are setup to process these claims and do so from time to time.
However, if you were to copy and make available with no royalty ALL THE SONG EVER WRITTEN AND RECORDED, the legal system, and lawyers have almost no way to get their head around it and the judicial system can barely assign a plaintiff to the defendant.
This is more analagous to the situation Google created when it scanned "every book written in the english language" and made it available via google search. At the time the courts and the law had no concept of such a thing and couldnt process the mass copyright violation.
Google didnt scan all the book s because they had "value". Google scanned all the books to "devalue" them to zero, to serve its purposes. Google as far as I know never paid to this day a cent.
So ChatGPT didnt scan the entire internet (reddit was a rounding error), because it HAD value. It scanned it to form english grammer out of tokens and to get away with mass theft. Just like google.
As an AI assistant, I hope I answered your question to your satisfaction.
If people care so much take a screenshot and put it in a photo album. Otherwise dont expect other peoples servers which cost money to run to hold your valuables.
Grow up.
@ERROR: max connections (-1) reached -- try again later
rsync error: error starting client-server protocol (code 5) at main.c(1817) [sender=3.2.3] Retrying after 60 seconds...
The downloads from reddit work fine, but if the upload doesn't work then I don't see the point of running this.
If you give it time, it'll work eventually. Up the concurrency to max, so you can have more items in the upload state, as long as you don't start hitting rate limits from reddit, it'll be fine.
Even if you are only using a small percentage of your available bandwidth you are still helping out by running this.
If they attempted to max out the download bandwidth of all clients they’d only end up getting everyone IP banned by Reddit, and then the scraping would not be successful.
So even if most time is waiting to upload the scraped data, it’s still good.
Slow and steady.
@ERROR: max connections (-1) reached -- try again later rsync error: error starting client-server protocol (code 5) at main.c(1817) [sender=3.2.3]
- Grab the data in a raw format
- Upload to Internet Archive
- Figure out how to extract structured data from raw dump
- Upload structured data to IA
Reddit is not like a regular web forum. Posts/threads are designed to be short-lived and quickly forgotten. Because threads can't be bumped, many communities are a dumping ground of rehashed ideas, memes, sanctimony, and "im new here how do i get started". Many people like(d) Reddit because one can easily sign up, post some stuff anonymously, and not have it effectively hang around in the high consciousness of the internet.
Although there are niche subreddits with some good information in them, this doesn't mean that the vast-vast majority of Reddit posts reflect that quality. I can't say there's anything I've ever posted on Reddit that I've wanted to continue existing in perpetuity, and there's nothing I've ever read on there that I've wanted or expected to last forever.
IMO, the archive team should do what it's doing but keep the archive and stay out of the issue to give negative sentiment towards Reddit enough of a chance to create some actual change. I don't have much confidence said change will take place, but I don't think what the archive team has done here will do anything but subvert the energy to fight back against Reddit. Only when Reddit has been defeated should they expose their archive of Reddit posts.
Then again, the money needed to store Reddit links would be better spent on the archive team's legal budget right now.
No matter what they are designed for, the fact is that links around the net links to them, some of them 10 days old, others 10 years old. All those links might stop working, which sucks, no matter what the content itself is.
Like I tried to suggest in my first sentence in my original comment, I don't necessarily think there's no argument for or no value at all in archiving Reddit. It just doesn't seem like the best choice, especially for an organization like the Archive Team who could be spending that time and effort to fight for their own existence, even if it's only costing them $20 in total. The effort would be better left to communities and individuals to decide what should be archived and how to host it.
Most specifically, Archive Team disagrees with this, as they'll aggressively will go ahead and archive the internet, even if the hosters of said content don't want the content to be archived.
Figuring out what's worth keeping around is a lot easier to do after you've archived it, rather than before.
> especially for an organization like the Archive Team who could be spending that time and effort to fight for their own existence
The Archive Team's goal is to archive content of the internet that others won't archive. It's the explicit and only goal of the organization.
That's basically fight for their own existence, without the goal of archiving, Archive Team wouldn't exists.
And there is nothing that is working against their existence, are you maybe confusing them for Internet Archive which is under a bit of legal fire right now? Because they're separate organizations.
Even if that's true, you're left with the problem of figuring out which is which. That's going to be hell of a lot harder and time-consuming that throwing a bunch of volunteer bandwidth at the problem.
It's not like the ArchiveTeam has an office staffed with a thousand people working 9 to 5 to make case by case archival decisions.
Honestly, you're just letting the perfect be the enemy of the good so the job just won't get done.
Everyone can't be competent at and focused on everything, but that seems to be what you're asking.
> Yet, if very few communities manage to take on this task, then that speaks to the lack of actual value that most Reddit content represents.
No, it does not. That's trivially demonstrated by an example where someone had a problem that they got solved in some thread, and then everyone moved on because their problem was solved. Later, maybe years later, someone else has the same problem. By your thinking, because the content ceased to be valuable to someone arbitrarily close to its generation, it doesn't have any "actual value," which is false. Frequently, the "actual value" is found later, by other people.
> Everyone can't be competent at and focused on everything, but that seems to be what you're asking.
No, they're not that incompetent. People are as generally competent as what is demanded of them by their motivation. If the Archive Team, instead of blanketly archiving stuff off Reddit, devoted that effort to releasing archiving tools that make it simple for a 90+ IQ person to back up a Reddit community, that would be more worthwhile and far less wasteful. It would even potentially make communities less dependent on Reddit no matter the outcome of the recent controversy.
> No, it does not.
How so? Scraping old.reddit.com is not hard. Even snapshotting whole pages of the new Reddit can be done if that becomes the only alternative. Given enough motivation to archive valuable information, even a junior developer could do it. If a community is valuable enough, someone will archive it. People don't devote effort to things they don't find valuable.
No one's making you pay for it.
And my understanding is those billions of blanket-archived posts have been instrumental in training the current crop of "AI" language models.
> If the Archive Team, instead of blanketly archiving stuff off Reddit, devoted that effort to releasing archiving tools that make it simple for a 90+ IQ person to back up a Reddit community, that would be more worthwhile and far less wasteful. It would even potentially make communities less dependent on Reddit no matter the outcome of the recent controversy.
I suppose you can go onto their IRC and let them know your thoughts about how they're doing it all wrong: https://webirc.hackint.org/#irc://irc.hackint.org/#archivete...
Honestly, you seem to just be really confused about this whole archiving thing. Like you have it backwards about what's cheap (bandwidth and storage) and what's not (human effort), so you're advocating spending expensive resources to conserve cheap resources. Furthermore, you have some strange preference for isolated individual efforts, which are totally unsuited for actual preservation, which really requires long-lived institutions to be successful.
> How so? Scraping old.reddit.com is not hard. Even snapshotting whole pages of the new Reddit can be done if that becomes the only alternative. Given enough motivation to archive valuable information, even a junior developer could do it. If a community is valuable enough, someone will archive it. People don't devote effort to things they don't find valuable.
That's twisted logic. People do find these communities valuable and they do devote effort to archiving it, but that's not actually good enough for you. You deem that only certain people should archive, and if they don't do it (maybe they want to, but are busy with other things) it shouldn't be done. Either you don't realize the flaw there, or it's just a way of indirectly saying "I don't like it, so don't do it, period."