Reddit's Robots.txt Changed
old.reddit.com
old.reddit.com
But at the same time, blocking search engines from indexing your social media site is a dangerous game. Any search engine that respects this is gonna effectively de-list Reddit. That's no good for views, and views is what makes Reddit money. Presumably they have negotiated private deals with Google and probably Microsoft for this and are trying to sell their data to ML companies, because otherwise this would seem suicidal.
Kind of a shame. The information is still going to get shared around to all the giant corporations, but Reddit will presumably make it harder to access for all the little guys. And the more they tie the content to dollars, the more managers on the inside will start doing stupid things to try and generate more of whatever the most valuable kinds of content are.
Yeah see the API changes from last year and their pre-IPO investor roadshow. This ground is well trodden.
So maybe they can afford to be delisted from search engines, because the site is addictive enough to bring back its users – and new ones because it's mentioned everywhere. Like Discord or Facebook, where most content is also unavailable to search engines (although they probably rely much more on the social aspects).
Since the moderator revolt, I've noticed an increase in claims that there is an overwhelming volume of AI-generated content on Reddit coupled with boosting. Either that started with reddit's strike-busting efforts or was made more obvious by it.
Since then, this is becoming far worse. Reddit's front page is dominated by random reposts by random low-karma user accounts that all of a sudden create the top post in the whole Reddit universe. Then there are reposts by other random low-karma user accounts on other subreddit that benefit from the same exact treatment.
I'm convinced that the reddit revolt motivated reddit to take a more direct and involved approach to generating and boosting content, and now the effort targets specific segments. This also includes the creation and boosting of subreddits dedicated to mainstream topics. Regardless of this coming from reddit itself or marketing/PR companies, I would not be surprised if the bulk of this content was machine-generated fluff.
For any sub where people ask a lot of questions it's very annoying. I've already unsubbed from a bunch of subs because I didn't want to see the inane/clueless questions being asked.
I have to assume they're promoting these dumbass questions into peoples feeds because they're precisely the type of content AI will love. People asking questions and humans giving answers, but boy it sure does make the experience as a user worse.
The biggest issue for me personally is that Reddit's built-in search is about as good as it's built-in mobile video player.
Which is to say it is one of the worst pieces of software ever foisted upon humanity despite having been an objectively solved problem with off-the-shelf solutions available for decades.
This wouldn't really bother me if Google weren't the only way to effectively search Reddit.
Every single time I have the misfortune of trying a Reddit link on mobile, I have to fight prompt after prompt after prompt telling me, no begging me, to use the app. It’s like they hate me so much for having the audacity to not install their app, it’s insane.
I guess they only bother keeping the web site around at all is for the faint hope that it’ll act as a funnel to their app? They certainly don’t want anyone actually using the website, they’ve made that perfectly clear.
Side note: They seem to serve other robots.txt for different User-Agents & IPs: https://merj.com/blog/investigating-reddits-robots-txt-cloak...
User-Agent: bender
Disallow: /my_shiny_metal_ass
OpenAI is openly collaborating with the NSA, Google is manipulating the definition of a web crawl, Anthropic has installed a bunch of humanitarians from Jump Trading as the leading mech interp group that makes strident claims about how all this stuff works based on weights you do not and never will have access to.
They’re telling you: “And you will do nothing, because you can do nothing.”
I invite you to join me in proving that we can in fact do something.
The enterprise is keenly aware that they will be decades recovering from how badly they got put over a barrel on cloud. The enterprise cares deeply about where their privileged and in many cases legally regulated data goes and is stored.
Big business needs an ally in the tech game, and for once that is looking likely to trickle down to John Q Taxpayer.
And they got their ass kicked by Linux. Not even a close, hard-fought match. Brutally humiliated and driven out of business and sold off for parts.
The reckless contempt that the modern AI cartel holds open models in, and GGUF in, and even Apple in on this stuff is staggering and makes me wonder if any of the leadership even knows the history.
No one has ever tried to tell the technology community what they can and can’t do on their own machine and lived to tell the story. The people pushing DRM over the years need a restraining order against their own employees.
Bill Gates is like, a thousand times smarter and more ruthless than any of these people and he backed off in his prime.
Explain please.
https://www.theverge.com/24067997/robots-txt-ai-text-file-we...
But the next 20 hits are just more ways Google is pushing the envelope on creative interpretations of robots.txt
Yesterday I asked Sonnet 3.5 how to compete with Anthropic and it was like, point out that no serious science can be done without the ability to verify it. These things are pretty reliable at pointing out why they are bad news.
The only creative interpretation of robots.txt I saw was allowing people to opt out of AI uses by disallowing Google-Extended though there is no Google-Extended crawler. This did not manipulate the definition of a web crawl.
The question was not why are LLMs bad.
AI is the death-knell of the web as we've known it for the past three decades. Once freely available information will retreat behind login walls and charge bigco's for access to train their models.
I wonder if some standardised data API will be settled upon, perhaps it already exists?
# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
# policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy
User-agent: *
Disallow: /https://www.reddit.com/r/RedditAlternatives/wiki/index/
I think a moderated mirror of certain subs would be a better solution. Because people like freedom but love their cats and memes much more.
Ironically the only off-site that ever really took off is the snuff film site which has 2.3M users now.
Because that is Reddit's magic ingredient and you will have to do it 24/7 from day one.
And if you self-host to avoid the risk of handing over your community to those sites under control, you get flooded with DDoS and hacking attacks (including state-sponsored actors). Sad times.
80% of my Google searches for other people's opinions now end with "site:reddit.com", and there is surprisingly quite a few of them. The alternative is Reddit's own search and it tends to produce less relevant results.
Google has not commented on whether they plan to respect it. Rich Results[0] say they're using a version from June 25. The new version was last modified July 1.
“User-agent: * Disallow: /“ in the `robots.txt` simply instructs all web crawlers not to crawl any pages on the website, not whether to show pages in search results. Of course new pages wont be shown if they cant be crawled if theres no other way you can get it..
But that line simply prevents crawlers that abide by robots.txt from retrieving the content of any pages.
But Google doesnt need to crawl Reddit anymore. Reddit is directly giving the data to them to serve to users! Bing, DDG and any other search engines now are basically forced to start paying Reddit. And presumably, the Reddit execs have calculated that future revenue will be more because of this.
Don’t be fooled: This isnt a “good for internet, morality” decision. Reddit is a public company now and has shareholders. They are doing this because it will equate to more $$$$
What are they thinking?