Lessons from Building and Scaling Reddit's Ad Serving Platform with Go
about.sourcegraph.com
about.sourcegraph.com
It'd be like writing Apache or Nginx from scratch. It seems simple on the surface and then you wake up to all kinds of inconvenient and unexpected technical challenges.
I'm not sure Reddit's engineering resources are best allocated to building ad serving infrastructure and software. It was going to be an ongoing headache from the start, and worse, it sounds like it was build with a language and stack that was fairly new to the team.
My prediction is that it will become a high cost and difficult to maintain project that will also prove difficult to hire for when this engineering team gets tired of working on the (currently new and fun, soon to be old and scary) adserver and moves on to other jobs.
My point being that some challenges are best left to third parties who make solving some problem its full time job. I truly do not say that out of bias, just experience building and managing such a complex system.
They're one of the largest websites in the world now, so what holds for 'ordinary' websites and their best practices may simply no longer apply to Reddit.
The whole Reddit saga for the last few years is quite the story of turnaround management done well.
But then I was thinking that if it's going to be so important, why a 3 person team with no experience running Go in production?
Internally there were some napkin math and found out while the 3rd parties would save development time the returns of implementing features their customers were asking for now instead of waiting for 3rd party company paid for them to hire 3x the engineering staff the 3rd party company even had.
Size matters, Reddit is at the scale of tiny percentage gains, if they control everything and can drop a standard 90ms down to 10ms and that converts to a couple of percentage points on their revenue it is worth it.
Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days. Reporting and ad setup from the user is much more challenging.
10ms, after request processing, ad selection logic, and faster-than-light packet travel? I think that's far too optimistic of an improvement. As the post said, 30ms was the goal. My adserver delivers in 20-40ms. But since ads are routinely loaded asynchronously, is speed concern truly valid? The adserver can usually make a selection and deliver before the content is fully rendered in the browser.
> Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days.
If you developed a system that delivers 1BB impressions per day (most large scale ad networks deliver only a modest multiple of that number) and didn't find it challenging, then it's either a very simple system or you're Elon Musk ;) That's 12k requests per second, generally uncached, hitting all sorts of internal services most likely.
I'm just saying that I think Reddit's best bet is to offload the task to a company that works on these problems full time. Or acquire one like AOL and Facebook. Unless the use cases are so unique and key to the company's future plans, avoid building in-house. And if you have to, perhaps it would be a larger effort than a 3 person team developing the thing in a language that the company never used in production before.
I know it isn't the actual ad delivery itself but don't have public docs for that :(. Here is the ETL layer to back up claim: https://docs.google.com/presentation/d/13x_wdXA_C9ZymA2gcjsb...
But yeah, it should be loaded async. Been a while since I worked with ad stuff, but I thought 10ms-20ms sounded more likely.
More/faster hardware can obviously help, but since everything's async from the browser's perspective you tend to run into higher costs and diminishing returns by pushing for the lowest latency possible.
At the scale of reddit, and because this is how they make their money, this is clearly something they want in house and something it would be a competitive advantage to control, if nothing else it gives them a much clearer conduit to their advertisers, and much greater control over things like ad selection.
More-pertinently, I wonder how many of the developers of reddit's new ad system themselves have ad blockers installed. Do any of them actually see the results of their work?
But even then, I whitelisted reddit.
So my guess is yes, they see the output of their work.
They had a misbehaving ad (downloaded a ton of "stuff" over and over and over). I reported it on one of the their support subreddits and was told "don't recognize the add urls. scan your system for a virus". It was a pretty good error report too (screen shots, console log, etc).
Installed ublock origin that day and haven't looked back.
It may appear "emotionally driven" because my experience with reddit ads has been terrible. I struggled through it, i followed the example ads, i created very topical ads and when i finally got all approved and done, they just didn't perform well either. I talked with peers in the industry and they all avoid reddit ads for the most part because of similar experiences. I haven't spent any money with ya'll since that bad taste experience.
for me, it would just be easier to site target reddit through google and do adwords for desktop/web traffic.
The editorial review literally feels like some redditor judging your ad on personal opinion vs an editorial review for passing some kind of targetable creative process. That's a terrible way to treat paying customers and i don't that experience on any other market i advertise on.
On top of that, the traffic we received from reddit was very bad for our e-sports/gaming-related website.
p.s. If any engineers or PMs are interested in helping us build out a world class, high-scale ad platform please get in touch (email in profile). We are hiring for senior and above positions in NY and SF.
(Although even without adblock, are there actually all that many overt ads on Reddit, other than the sidebar ad which nearly always is an ad for a different subreddit?)
I however, will not relent.
I still use old.reddit.com, even on mobile, because the new React version is slow and constantly tries to get me to download the app so that they can show me ads.
Curious, which one is it? I suspect the former.
Most will be lucking to reach 10 comments.
Done reading there.
Not sure if bad written or just not knowledgeable, please do your research.
EDIT: duh, search for "rpc" on wikipedia[1]. The first came a few years before SunRPC out of Xerox PARC.
[0] https://en.wikipedia.org/wiki/Open_Network_Computing_Remote_...
IMHO, it only brings problems.
It will not save you time typing: each route can have a transport request decoder and response encoder (encoder's are often shared), a request and response type, the actual business logic, a function that takes the request struct, applies it to the business logic function and returns a response object, a function that takes the previous function and connects it with the transport machinery (building the actual endpoint), and possibly middleware implementations.
However, by teasing all of these things apart from the very beginning it's very easy to come back later and add a circuit-breaker, or (as the article says) switch your transport from HTTP to grpc (or support both).
How is that great?
PS: Not talking about native ads.
If so, what stats have seen (impressions, clicks, conversions, etc.) for your ads?
We stopped advertising on reddit after the rise of the_donald and other white supremacy subs led to our ads being placed next to a lot of bigotry and white supremacist content, even in subreddits you'd generally consider innocuous. The hate groups have spilled over into most subs now.
Where do you get info about user and whats in it?
Because all the hard work to serve ads more efficiently goes to waste when the site is a mess and unusable.
Obviously Facebook/Instagram/Twitter/Snapchat kind of worry them, but then again, I feel like people are "done" with "social media" so I wouldn't be _really_ worried if I were Reddit. They have the advantage that they can give the user very interest-focused information; when you follow a friend on Facebook, you get their programming advice, baby pictures, and political rants. With Reddit, you get to skip the baby pictures and political rants.
Also, unlike Digg, Reddit hasn't been bleeding members to another site for some time now, so there's nowhere for Reddit users to go if they become dissatisfied. Does anyone have any ideas about why that is? Reddit is not, fundamentally, a hard product to emulate, and yet I know of no site even remotely as popular nowadays (HN aside, of course). Slashdot, Fark, and sites like those all seem to have mostly been consumed by Reddit. Is it the strong network effects that Reddit has, or does Reddit's moderation system provide a unique way of hosting a community that no one has reproduced?
I think Discord has been siphoning off some of the users on the gaming related subreddits, but it fundamentally serves a different purpose so its not a wholesale replacement.
However, the reddit mobile experience leaves much to be desired, and I will never install their app.
Maybe they are planning for the audience they want rather than the one they have? Or maybe they just feel safe since there aren't any alternatives?
That's definitely a reasonable possibility since it appears to me to be formatted for the Instagram crowd.
"Or maybe they just feel safe since there aren't any alternatives?"
Possible as well but ironic since its creating a space for an alternative.
I like this sites design. Clean, simple, and nothing superfluous. Seems to understand people are here for the user shared content and not the site itself so it stays out of the users way.
I submitted some complaint in their redesign subreddit but immediately random people replied to say that this was as designed and I should use keyboard to scroll, and not the scrollbar. But then the issue was that when the lightbox was shown it wasn't focused so I first had to use mouse to click in the lightbox to focus it and then use keyboard arrows to scroll.
Such a joke. Amateurs.
To put a cherry on top of it, the new layout now intersperses ads into the post lists and styles them to look like posts.
Additionally, ever since reddit decided to host it's own media, it no longer possible to directly link to a video or image. I don't want to send a link to my friend of comments about a gif. I want to just send him the damn gif. For this reason alone I'll continue to use imgur.
I'll be really bummed out the day I can't click "switch to the old layout", but I'm sure the time when they're removing that is coming.
It looks like we're finally seeing a series of decisions being made that caters stockholders instead of users of the site. They managed to hold out for this long, but I guess the day has come where it has finally happened. The big difference between reddit and all the other sites that have done this, is reddit used to be the hip place people went to get away from these kinds of user hostile moves.
The only question left now: is reddit so entrenched in it's position that it won't lose ground to a newcomer from the fallout of these decisions? Probably, but I know I'll definitely be keeping my eye out for competition that is gaining steam.
Also: all the hard work to serve ads more efficiently goes to waste when it get's blocked by a simple ad-blocker.
There was some wonkiness where my profile preference to have thumbnails not shown kept getting overridden for some reason, but that seems to have stopped.
Second, it forces learning a new layout and system. I don't understand instead of gradual changes in UI to improve experience they went with whole new thing.
Third, it's slow, bloated, and really hard to navigate through. Old version can display more pertinent information per screen resolution.
In a nutshell, hinders user experience without any noticeable improvement.
Finish product is bad design because initial idea was bad design to begin with (i.e. the idea to fix nothing and break everything).
* No way to easily access personal multi-reddits
* Saved/Hidden/Blocked have disappeared
* A lot of information in the subs sidebar doesn't show up anymore (things like related subs, wiki and so on). Now it's limited to rules and welcome message.
The multi-reddits thing is especially problematic to me. If you've subscribed to many subs, it becomes unmanageable without them.
Still, the interface is much clearer on old Reddit, new Reddit really shows the optimization of watching pictures and videos instead of articles and comments.
That they make their mobile site intentionally unfriendly and frustrating is infuriating.
The old reddit is buttery smooth in comparison. I really don't get it
There are a few other dark patterns hidden in there too, all coincidentally pushing in a direction that's good for reddit.
I suspect that's the real purpose of the redesign and that all complaints will fall on deaf ears until someone actually manages to disrupt reddit.
I mean at this point I don't think there's any doubt whatsoever. Is there even a plausible other explanation?
I only lurk on reddit but it's rather comical how bad it's become. And not just the interface, the content too, /r/all is basically Facebook with a slightly younger audience and you have to get deeper and deeper into niche subreddits to find worthwhile discussions.
Which raises the question wouldn't that be better served with a forum...
If Reddit isn't charging and managing those using their platform for that purpose -- not an insignificant number, surely -- they are fools.
poor performance, instead use fasthttp
Furthermore, the HTTP stack doesn't show up on most web servers' profiles. net/http is better understood by the community, interoperates with libraries better (e.g. golang.org/x/oauth2), and is better supported.
Use net/http.
Fasthttp is great if you can limit your interactions to a minor testable subset of calls but otherwise its not suitable.
However, last time I looked fasthttp was great if you don't care about http2 support or streaming bodies. Unfortunately I do.