You guys have built quite a nice and useful product. I was thinking of a similar project which required crawling all of reddit. But presently there are some hiccups due to the large amount of data involved.

I wanted to ask how you guys manage the reddit data? Do you use praw/pushshift to get all the comments/post titles or is it limited to selected subreddits? Including all comments and all subreddits is quite a lot of data I guess. Any suggestions to collect that data and then work with it effectively? Thanks.