How F5Bot Slurps All of Reddit

Maybe one is just an alias though? I wonder if you can make a r/u_$unused_username and then later register $unused_username

edit: nope, you can't make a sub that starts with "u_"

https://www.reddit.com/r/help/comments/6nxqjm/maximum_of_100...

I pulled the dumps from Bigquery and am going to load them into PG at some point here, so I can do arbitrary queries without hitting BigQuery every time. Haven't looked at how to do that in realtime though, retrospective queries are mostly what I'm interested in.

If you don't want to do that, there's also https://redditsearch.io/ - if you want to search back farther, be sure to set the toggle to "all" instead of "day"!

Thankfully, services like pushshift[1] exist, which has a sane API and the option to use plain elasticsearch.

[1] https://github.com/pushshift/api

It's not really related to search, most of the cause is a pretty bizarre optimization method that reddit decided to implement fairly early on. The database structure is unusual and not very conducive to indexing (it's similar to an EAV model), so at some point they decided to basically write their own "secondary indexing"-like system that stores the "listing indexes" in memcached/Cassandra (with the data itself in PostgreSQL).

Whenever something happens that affects any listings (new post created, voting, etc), the site figures out all the listings it needs to update, and where in each listing the affected post now belongs and updates them all. So for example, if you make a new submission to /r/pics, it will go through and add the post's ID in the right spot to the "new posts in /r/pics" listing, the "hot posts in /r/pics" listing, the "new posts by yourusername" listing, and so on. As it's going through and updating all these lists, it also trims each one down to 1000 items.

It's conceptually pretty similar to a normal database indexing system, but basically maintains all the indexes "manually" and restricts them all to the top 1000 items.

If you're curious enough to dig around in the code, this is probably the main relevant file: https://github.com/reddit-archive/reddit/blob/master/r2/r2/l...

It's still quite slow compared to C/C++/Rust/Go, more than 10x slower:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Example query which searches for 'f5bot' in the past day and correctly finds the corresponding posts on Reddit:

   #standardSQL
   SELECT title, subreddit, permalink
   FROM `pushshift.rt_reddit.submissions`
   WHERE created_utc > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
   AND REGEXP_CONTAINS(LOWER(title), r'f5bot')

I mean, I enjoy the idea of human readability as much as Jon Postel, but at certain scales you have to wonder about the hidden cost of petabytes of human-readable data flying over the wire, never to be seen by anything but computers.

If I'm right, that would probably speed things up significantly.

Sigh, just use Perl. Writing code with the general regex engine took me only one minute of effort, but it runs already nearly 500× faster than codeplea's optimised special purpose code.

Why 100000 loops and not 10 like in the original code? Otherwise Benchmark.pm will show "(warning: too few iterations for a reliable count)".

----

benchmark.php 100000 loops:

    Loaded 3000 keywords to search on a text of 19377 characters.

    Searching with aho corasick...
    time: 329.3522541523

benchmark.pl 100000 loops:

    Benchmark: timing 100000 iterations of regex...
    regex: 0.691561 wallclock secs ( 0.69 usr +  0.00 sys =  0.69 CPU) @ 144927.54/s (n=100000)

benchmark.pl (fill in the abbreviated ... parts from benchmark_setup.php):

    #!/usr/bin/env perl
    use Benchmark qw(timethese :hireswallclock);
    require Time::HiRes;
    my @needles = qw(
    abandonment abashed abashments abduction ...
    );
    my $haystack = 'unscathed grampus ...
    heroically';
    my $n = join '|', @needles;
    timethese 100000, {
        regex => sub {
            my @found;
            while ($haystack =~ /($n)/cg) {
                push @found, [$1, pos $haystack];
            }
            return @found;
        },
        index => sub {
            my @found;
            for (@needles) {
                my $pos = index $haystack, $_;
                push @found, [$_, $pos] if -1 < $pos;
            }
            return @found;
        }
    };

Their default listing mode works very poorly. It would certainly be more requests to use a hybrid system like you're talking about.

It's pretty easy to get a firehose.

She/He's not. One request can supply 100 IDs.

> So here are 2,000 posts, spread out over 20 batches of 100 that we download simultaneously. It assumes you’ve already got the last post ID loaded into $id base-36.

> Here’s how we do it. We find the starting post ID, and then we get posts individually from https://api.reddit.com/api/info.json?id=. We’ll need to add a big list of post IDs to the end of that API URL.

The votes aren't real and they don't matter. This is true for FB likes or whatever else you can imagine. Reddit goes to a LOT of trouble to counter bad actors, but if you're in some subreddit that you think is full of abuse, then it's like any other part of the internet. Go somewhere else. This has little to do with Reddit as a whole, so the proclamation seems unnecessarily volatile.