Caching at Reddit
redditblog.com
redditblog.com
I remember looking into this a while ago and was bewildered to find that when I upvoted or downvoted, there was no XHR call to the backend! There was no hidden iframe/image, no silent form post. Absolutely no network activity. Yet when I refreshed, my vote was shown correctly. I thought I was going crazy.
This was long ago so I'm a bit fuzzy on the details but after a bit of digging, I found the most elegant data collection technique I've ever seen. Instead of sending network data when I voted, a local cookie was set with the link id and vote value. Then when I went to another page, my browser naturally sent the cookie to the server, where I believe it was processed, and then a fresh cookie was sent back to my browser. I could vote on 10 links, the local cookie would get large and then on the next page refresh, the backend would receive my batch of votes, process them, and send me a fresh cookie again.
I don't think they do that now and I've never seen anyone do something like this. Even HN just makes an XHR call on voting. After twenty years on the web, it's not often that I am surprised so this was quite a thrill.
This would mean none of my votes would ever have made it to the server!
There are ways around it if it were a problem though:
* Have a timeout on updates. If a vote has been sat in the local cookie send for some time send an XHR request to push the current queue to the server
* Use onunload event handlers to push any remaining queued votes when a tab closes. You could even try maintain a count of how many tabs are currently open (in the same cookie) and only send that request on the last one closing.
* If the amount of votes hits a certain length, also send a XHR request to register them.
Probably not perfect (I have in the past found onunload events to not be terribly reliable) but it would capture most of the otherwise lost votes you describe. It might be too much work to write/debug/maintain though, if the number of lost votes would be small anyway.
You could add an onunload handler that fires off a request with the cookie to mitigate this.
It took 8.4 seconds to load the Reddit front page on my phone. Hacker News took 1.1 seconds. This feels like advice from the overweight gym teacher on how to do pushups.
The desktop Reddit site took 2.2 seconds over the same connection, by the way. It seems like it would be much more valuable to optimize whatever is taking up >75% of page time on mobile.
T 0: white page
T 5: to see the reddit header with logo, the rest of the page is still white
T 16: to see the bar stating that I should use the app, click on the tiny mobile site link
T 25: page visible behind the "app" bar stating I should use the app
T 32: the bar is gone, and the site is "usable"
The interface is great. You can swipe for (almost) everything, which makes it easy to use one-handed.
They just released for beta a new mod-mail that is unusable, so I'm guessing this new trendy but shitty design will eventually replace the whole site. I don't know what will happen after that, but I won't stick around to see it.
The subreddits vary from highly toxic to well moderated (spacex for example), but most are so lightly moderated that they have some good links but little interesting comment content. The contrast with HN is interesting - I think the moderation here is much better and the culture is more welcoming as a result, but the home page here now just goes past too fast for it to function as a firehose on specific subjects.
It's perfectly valid to criticize something for sucking even if there isn't a better alternative.
Reddit is a cesspool, and there is no proof that's the only thing that can happen with large online communities. There is plenty of room for innovation there. It's not like online communities have been around for thousands of years and this is clearly the best solution as proved by extensive research.
Disclaimer: I'm a Madrid fan, not Barca.
Isn't that just Sturgeon's Law?
I have no idea how many subreddits there are, but I'd wager there are probably dozens, if not hundreds, of Hacker News equivalents - albeit tailored to different interests.
I tend agree with you that the default homepage is a flaming trashcan, but I honestly haven't seen it in years - I just see my curated subreddits.
Reddit really is a hit and miss barrel of communities. We (r/SpaceX mods) tend to find that a lot of our users don't even bother commenting in other subreddits. I certainly don't. Some of communities and the stunts they pull are downright embarrassing, and being hosted on the same platform sometimes feels a little ugly; so having our own little island that's peaceful and courteous is a big breath of fresh air.
Back in the day the SomethingAwful forums were hugely popular largely because they had subforums covering a big variety of topcs. They're still somewhat popular but Reddit has taken a huge amount of that traffic.
One big problem with Reddit though is ongoing discussion. Anything older than a few hours and you may as well not comment. Whereas traditional forums excel in that area.
I feel like there'd be a use case for a more traditional forum, but where users can create and moderate their own subforums. In fact I wonder why no-one's done that already.
It's basically a third party equivalent.
Contrast with Facebook, Twitter, or other massively-accepted social networking sites. They start you out with ... a blank page! This not only allows but requires the user to craft an experience that will make them feel comfortable, and there's no need to confront them with things that may not be palatable.
I think you can only get away with a real "front page" if you're targeting a specific niche. Otherwise, you have to go really generic. Twitter usually recommends that new accounts follow very generic things, like late night talk show hosts, sports stars, and popular singers.
A lot of reddit's growth problem is that if a Republican or a religious person or someone over the age of 40 hears about it and decides to check out reddit.com, they're likely to leave angered, offended, shocked, or all of the above, and that's before they try to participate/comment, which is a whole 'nother can of worms.
Combine with occasional news stories about reddit's less-savory underbelly and the prevalence of pornography and profanity, and it really is no wonder that reddit struggles to find mainstream acceptance. The issue that reddit must now deal with is to become mainstream-acceptable without destroying its existing userbase in the process.
and you've got basically all of human civilization there.
I think it's more like slashdot. I didn't leave slashdot for reddit or HN. I just left. It got bad. I'm thinking the same thing is about to happen for me on reddit as well.
At some point, I might be following a whole different set of subreddits than I am today, as old ones die/jump the shark and I discover new ones. But I don't see what would specifically push me to leave the entire site, as long as there are individual active communities there I'm interested in—any more than I would choose to categorically stop visiting blogs hosted on Wordpress.com or categorically choose to never read a post on a phpBB forum again.
Of course, I might leave the whole site anyway, but not because of a push; rather because activity on the whole might drop off in favour of something else, and all those communities I like might migrate away (like many of the chatrooms I visit have migrated from IRC to Slack), leaving a hollow shell of a site that's still, in theory, a host for many communities I enjoy—but not their canonical home, just a secondary one. At that point, there'd be no reason to visit.
It went from me reading a dozen or so subreddits actively to that place I find free amateur porn.
Maybe I just grew up, but I also think a lot of subreddits declined, particularly larger ones.
HN, hubski, voat, etc. are picking up the slack. I'm also trying to spend less time socializing and more working on stuff.
All it's missing is search and it'll be perfect.
But regardless, what does the second page load take? What does the 3rd take?
For many people, they go on reddit and stay on there for a significant amount of time. So that first page load is much less impactful, especially if it can enable them to have faster "pageloads" while on the site. If the site is optimized for them, then having the image/text post load in a fraction of a second, and the comments in under a second (which it does for me) seems like a good bet.
That being said, they really could spend some time reducing that impact.
And just to throw my experience in with the others, I'm getting much faster load times. Timed using the chrome dev tools connected to my phone, it takes 2.76 seconds to the "load" event and about 3 seconds until it's fully usable. (cache disabled, 4g tmobile network, high-end android phone, obviously these will be much worse on slower phones, slower networks, etc...)
Most menus don't open for me on the first try (especially in landscape) or open in the wrong place with touch targets that are "off" so I have to rotate the phone a few times.
The old iOS 6 style site didn't age well visually, but it sure was fast. I miss the days when mobile sites were the fast and light version meant to work over 2G.
To me it's insane a non-SPA desktop site would become a SPA mobile site
Not to me. Yes, you might pay additional time on the first load, but you save massive amounts of data (and thus, due to TCP, highly latency bound packets) when you just transfer small bytes of JSON. Oh, and that saves battery too - mobile data transfer is the hugest drain on any phone's battery.
All 3 of which have to get transferred with a SPA or normal site.
And mobile radios don't immediately go to their lowest power state after transferring the data so I'd be shocked to see a SPA dynamically loading smaller pieces as you move about the comment section save battery over a static one that loads large amounts in one shot, sporadic small buts of data is much less efficient than large chunks on mobile in general.
Not to mention the CPU overhead of the SPA's Js which is non-zero even if radio battery usage is a larger issue.
On a 2G or nasty 3G link all that latency adds up. Oh and please don't mention "but clients should cache stuff"... caching stuff on clients is broken beyond repair, buggy as f..k (hello iPhone, hello Firefox, hello Chrome - honestly, I haven't been able to get ALL major clients on ALL OSes to properly use caching!) and if you expect half of your userbase likely has a totally-broken-caching client (i.e. old Android phones) you better build your infrastructure to require as few requests as possible => SPA route.
> And mobile radios don't immediately go to their lowest power state after transferring the data
Now imagine me reading a typical reddit thread on the front page. Takes me about 30 seconds before I am disgusted by some troll comment, and I go back to /r/cats. In the non-SPA case I'd have 2x approx. 10s of the radio blasting at full power, in the SPA case I'd have 1x 10s for the initial load of the app and maybe 2s for the second load of the 30 posts of /r/cats. Oh, and not to mention I, like many other people, like to reddit while on the train - which means in any case that 20 kbytes/data for a simple JSON is better than 2 MByte for each simple request. That small JSON can be transferred via EDGE if the signal is as bad as it usually is but good luck for your nerves if you have to wait ages for every website.
> Not to mention the CPU overhead of the SPA's Js which is non-zero even if radio battery usage is a larger issue.
Yeah but unlike broken browser caching this one is something YOU can influence, not having to hope your users some day have enough cash to buy a phone with proper up-to-date software.
Edit: missed yet another thing. An SPA can request thumbnails matching for the device display resolution dynamically whereas a non-SPA solution (usually!) has only one thumbnail size.
Taking that example the radio would be at it's full power state for 20 seconds in the second case and 15 seconds in the first, not 12 as you'd expect. And as soon as the connection isn't so horrible that it takes 10 seconds just to get back everything is cached the 5 seconds of high power state starts to look less and less likely, and in most cases you'll be at parity if not using more power.
The number I've optimized/seen optimized around in the past is assuming it takes about 5 seconds of inactivity for the radio to go back to low power state (It was taken from the 3G days but is still what Google's Android documentation bases it's recommendations for efficent downloads on: http://www.research.att.com/articles/featured_stories/2011_0...), so each request gives at least 5 seconds of high power usage.
Of course this just goes back to what I meant about other content dominating load times, if your connection is so bad that getting back responses for cached items takes 10 seconds, I'm sure the download time for those images and videos will be through the roof.
Reddit solved the problem of reddit on a bad connection like that long ago with .compact. Before the "mobile" reddit links were .compact links (which is why there's now an annoying banner pushing you towards the SPA). .compact links being the non-SPA version for mobile/low-bandwidth.
You can see it by appending .compact to a link: https://www.reddit.com/.compact
It had hardly any assets to load and did tiny thumbnails on the subreddit's page, and comment page, instead of trying to load a giant image of the content.
Comapre the SPA on simulated GPRS (so pretty the definition of a bad mobile connection): https://m.reddit.com/r/gifs/comments/5otign/cool_hedgehogs_d...
To the old .compact support: https://www.reddit.com/r/gifs/comments/5otign/cool_hedgehogs...
The moment you click on the SPA link it takes a whopping 30 seconds to load a preview image of the GIF. It also takes an entire minute just to load the code and assets for the SPA itself if it hasn't been cached. Compare that to 2 seconds for the .compact link's document + styling. If you're visiting that SPA on a bad connection you'd better hope those browser caches do their job, and that ProductionClient.js hasn't changed recently...
.compact worked well on the kind of connection where bandwidth was so low you'd want to evaluate content before dedicating to a 30 second wait because you clicked on "Show Comments", which used to be any mobile connection in general.
>Yeah but unlike broken browser caching this one is something YOU can influence, not having to hope your users some day have enough cash to buy a phone with proper up-to-date software.
That'd be fine if it hadn't been chugging on an iPhone 6 plus while scrolling since it's beta til now. An iPhone 6 might not be the newest hardware, but at launch (and even now), it's what most mobile developers would treat as "modern hardware" and "modern up-to-date software". Code size doesn't need to equate to how CPU intensive a script is, but it's not exactly a mystery that the SPA chugs while .compact is fast when the SPA has three times the Js and twice the CSS just for it's "core". .compact looks dated because of the few assets it uses (and trying to look like an iPhone app). Nothing would stop Reddit from making .compact links look modern, except they're clearly trying to drive .compact users to the SPA (they were much more aggressive about it back when it was in "beta")
Also, do you really have a problem where an asset falls out of cache from one page view to the next? If so, the server wouldn't be returning a 304 in that case.
General guidelines: Always doubt advice from startup folks. Usually the more unicorn and hype they are, the less qualified they actually are.
Basically, they grow by recruiting a lot of people (most inexperienced in average) and grow in all directions without a plan. To caricature, it's like a big playground.
which they insist (for me anyway) on every single page I load.
They just spent the past year relaunching "reddit - the startup" so their team is now plenty big, they just don't have much to show for it.
That's just not true. Reddit has a ton of extra functionality, including creation of subs, subscription to subs, controls of voting rules, user bots that run on reddit's servers (example: RemindMe), theming, tons of settings even for the regular users, moderators/admins interfaces, apps (example: ChangeTip), reply notifications, self-serving ad management system, image/video embeds.
I'm sure there's a lot more, these are just the obvious features that HN doesn't have.
RemindMeBot runs on its creator's own server and communicates through reddit's API as if it were a regular user. AutoModerator is the only bot I know of that runs on reddit's servers.
If that were true, this would be a much better site.
https://github.com/reddit/reddit-mobile/compare/e3f1119e3358...
I
How could you compare the 2? The former has images and rich media. The latter is all text, like a unix terminal. LOL
The new mobile version is a complete pile of junk!
Use the old one by adding '.compact' to a normal reddit link, it's better in almost every aspect.
You can easily open up the front page of HN as a logged in user and see that it contains information about which storeies you've voted for and which you've flagged. On top of that, you can click "hide" to hide stories. These will only be hidden for your specific account, not for every person who loads the front page.
What's more, the go-to call-to-action by HN admins during really popular stories is to have people log out, because logged in users don't get cached results. Have a look at the comment by HN admin dang on the Trump winning story. [1] The key part being "please log out to read HN today, unless you want to comment. Then we can serve you from cache".
They're two completely different operations, with completely different complexities behind them.
The "hide" feature is not analogous to what reddit is doing.
When I go to reddit.com, I see a list of stories from subreddits I've personally selected.
To be totally honest I'd say the comment is verging on toxicity. They're basically telling these backend engineers to piss off and save the blog post until the mobile site loads faster. It's quite likely they have little-to-no control over the mobile site and caching improvements are one of the few indirect ways they can improve it. This exactly the opposite of what we should see in response to content like this.
That's not really what I'm going for. The way they framed the blogpost just lacks perspective. Perhaps that's due to organizational structure, but maybe there's a lesson in that too.
I am a little disappointed I ended up as the top comment. The technical comments are more interesting. The upvotes on my comment probably just represent frustration.
This was over a different connection, so I tried HN again. It took 1.0 seconds.
I'm probably overreacting but reading this really got to me. My apologies if I came off too aggressive or negative.
By the way, and I'm going out on a limb with my shameful plug, I built a Varnish-as-a-Service kind of infrastructure called Cachoid ( https://www.cachoid.com ). But to my own defense, I'm putting my energy, time, and money where my mouth is.
Logged out users see a "snapshot" of the page updated every so often.
And I really don't think that caching pages per session would really help with their load all that much. Why not just use HTTP cache headers at that point?
Plus while you don't really need to show votes ASAP, logged in users will want up to date comments.
> For example, when new comments are added or votes are changed, we don’t simply invalidate the cache and move on—this happens too frequently and would make the caching near useless. Instead, we update the backend store (in Cassandra) as well as the cache. Fallback can always happen to the backend store if need be, but in practice this rarely happens. In fact, permacache is one of our best hit rates—over 99%.
They basically have their application state duplicated in both places. Interesting architectural choice.
Reddit, on the other hand, is the worst performing out of all the top sites that I can think of.
Most of my current job involves solving problems that are caused by cloud limitations.
The on-prem alternative is to run something like OpenStack, which is a major PITA.
54 x R3.2xlarge EC2 instances
On demand = $314,571.6/year
w/ 1 year term = $166,860/year
w/ 3 year term = $110,340/year
w/ convertible 3 year term = $150,174/year
Is that a lot? Seems like a lot
Going from a 50th site in the world in a niche topic (stack overflow) to a 27th worldwide that target all audience. It's entirely possible that it grows the traffic by 100x.
Anyway, who cares about money? 100k a year is peanuts. At this scale, if you have a solution that works well, you're good. Not even worth thinking about it.
They're better now then they were ages ago, but it still happens. Combine that with slow load times, etcetc.
Now, don't get me wrong they're likely doing better than i could - it's a very difficult problem to solve. But, something must be fundamentally wrong if a site who's growth has been steady and not "overnight", it just seems like it's always been very very slow. By overnight i just mean, it's growth has appeared predictable; Unlike the poor sites that get HN/Reddit hug of death.
edit: Not sure why i was downvoted, but i'll elaborate:
From a user perspective, Google never goes down. It's a big event when that happens. I've never seen Amazon down (again, personally!). I've never seen most of the big players down. Yet i've seen Reddit down more times than i can count. It's a meme on the site.
Regardless of how difficult the problem domain is, the end user doesn't care. All i'm saying is that the end user experience is not good enough, in my opinion. I don't think that is an unfair statement.
So if this caching is what is powering the current state of things, what can be done better? From a technical perspective, is this the best that can be achieved? I surely hope not. Would you argue it's the best we can hope for?
Anecdotally, I also get way more errors/dropped connections/random slowness on Reddit than on any of the other massive sites. Plus the mobile site is beyond terrible, unusable on my phone.
I'm sure it's not an easy problem to solve, but they should be able to afford some hosting with all the reddit gold that seems to get thrown around.
Dunno about reddit. Do they even have a revenue model except some ads that 60% of their user block?
Also there's reddit gold but who knows how much they get from that.
1. You use less bandwidth in/out of the box. This matters in AWS because your bandwidth is (relatively) limited.
2. Your CPU usage to (de)compress is distributed over a larger fleet. The clients spend the CPU time, not the cache servers.
Maybe only small because most californians were still sleeping? I was only up due to a sick kid. :D
But... their use case is IMHO one that you can accelerate tremendously by using Redis. I ran the most popular Italian Reddit-alike site for years, and I wrote a simple Reddit clone that uses Redis so I had the opportunity of exploring the problem a bit.
That said - we use Redis in at least 2 (soon to be 3) capacities across the site for different services (outside of the monolith) and it works really well.
(This is a joke please. Understand it as such. I know reddit has the problems it has because it is severely understaffed)