Hacker News Reader

They were already using a third party service. It was slow and opaque, so they ditched it.

At the scale of reddit, and because this is how they make their money, this is clearly something they want in house and something it would be a competitive advantage to control, if nothing else it gives them a much clearer conduit to their advertisers, and much greater control over things like ad selection.

The problem with Reddit ads isn't their tech, it's their people. The ad approval process is a farce - and human-managed. Oh, and those humans are assholes. You have to go through many denials - for no reason - submit a ticket to get another approver and approved - for no reason stated. It was the most upsetting process I've ever dealt with.

This is the QA process that yields us precocious-looking stock photo 'MIT grads' trying to sell me a wine subscription on every other pageload?

reddit has ads? I hadn't noticed …

More-pertinently, I wonder how many of the developers of reddit's new ad system themselves have ad blockers installed. Do any of them actually see the results of their work?

FWIW, when I worked at reddit, I never ran an ad blocker. In fact, I didn't run one for about five years after, until my CPU started spiking on just about every web page I visited.

But even then, I whitelisted reddit.

So my guess is yes, they see the output of their work.

Reddit ads were what finally pushed me into install ublock origin about a year ago (redditor since 2007).

They had a misbehaving ad (downloaded a ton of "stuff" over and over and over). I reported it on one of the their support subreddits and was told "don't recognize the add urls. scan your system for a virus". It was a pretty good error report too (screen shots, console log, etc).

Installed ublock origin that day and haven't looked back.

What were you trying to advertise out of curiosity?

It seems odd that there's a stringent ad approval process when the site proudly lets Redditors post any garbage they want. I'd have expected anything SFW and not obviously fraudulent to be allowed.

It's obviously not very stringent. A lot of the ads they run lately are outright scams, including even advertising Scientology: https://i.redd.it/6cd1ricqc8j11.jpg

It sounds bureaucratic, not stringent. Lots of forms to fill out, and a lot of order, and probably not even many requirements to meet (but a lot of different ways saying you did meet them) but lost its purpose a long ways back.

This is a pretty emotionally driven comment and would do well to provide some sort of concrete story behind why the ad approval process is a farce and why this particular group of individuals happens to be full of assholes?

I advertised for a travel site in the travel reddits. I never got a single ad approved "out the door" - only after I opened a ticket and asked why they were not approved did someone actually respond and approve it. The denial was within a few minutes of submission and the approval only after I opened a ticket in which case the response was "We're sorry, we've approved the ads" - but no one ever answered the question of editorial review.

It may appear "emotionally driven" because my experience with reddit ads has been terrible. I struggled through it, i followed the example ads, i created very topical ads and when i finally got all approved and done, they just didn't perform well either. I talked with peers in the industry and they all avoid reddit ads for the most part because of similar experiences. I haven't spent any money with ya'll since that bad taste experience.

for me, it would just be easier to site target reddit through google and do adwords for desktop/web traffic.

The editorial review literally feels like some redditor judging your ad on personal opinion vs an editorial review for passing some kind of targetable creative process. That's a terrible way to treat paying customers and i don't that experience on any other market i advertise on.

I agree that it's important to keep the money-making component of a business free from external dependency to a large degree, but I think there's some liability that's being ignored here. A 3 person team built such an important piece of Reddit in a language they never used in production? I hope the team was at least experienced in ad-tech.

Maybe they have other future projects with Go ?

It seems biased. I had a similar situation with a client of mine. They went out looked at 3rd party companies, and everything looked decent on paper.

Internally there were some napkin math and found out while the 3rd parties would save development time the returns of implementing features their customers were asking for now instead of waiting for 3rd party company paid for them to hire 3x the engineering staff the 3rd party company even had.

Size matters, Reddit is at the scale of tiny percentage gains, if they control everything and can drop a standard 90ms down to 10ms and that converts to a couple of percentage points on their revenue it is worth it.

Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days. Reporting and ad setup from the user is much more challenging.

Out of interest, how did those option compare with the napkin maths for paying a 3rd party company to include the needed features in their product?

> if they control everything and can drop a standard 90ms down to 10ms

10ms, after request processing, ad selection logic, and faster-than-light packet travel? I think that's far too optimistic of an improvement. As the post said, 30ms was the goal. My adserver delivers in 20-40ms. But since ads are routinely loaded asynchronously, is speed concern truly valid? The adserver can usually make a selection and deliver before the content is fully rendered in the browser.

> Talking from the experience of building out a system that handled +1B impressions a day, the actual ad serving isn't challenging at all these days.

If you developed a system that delivers 1BB impressions per day (most large scale ad networks deliver only a modest multiple of that number) and didn't find it challenging, then it's either a very simple system or you're Elon Musk ;) That's 12k requests per second, generally uncached, hitting all sorts of internal services most likely.

I'm just saying that I think Reddit's best bet is to offload the task to a company that works on these problems full time. Or acquire one like AOL and Facebook. Unless the use cases are so unique and key to the company's future plans, avoid building in-house. And if you have to, perhaps it would be a larger effort than a 3 person team developing the thing in a language that the company never used in production before.

Welp, guess I'm "Elon Musk" ;).

I know it isn't the actual ad delivery itself but don't have public docs for that :(. Here is the ETL layer to back up claim: https://docs.google.com/presentation/d/13x_wdXA_C9ZymA2gcjsb...

Total cakewalk. Impressive!

30ms really sounded a bit much for me, if you take network-locality as a precondition (i.e. adserver on same physical sub-1ms-network than webserver).

But yeah, it should be loaded async. Been a while since I worked with ad stuff, but I thought 10ms-20ms sounded more likely.

If it was straight ad delivery I'd agree that being closer to 10ms is achievable. But in the process of pulling in ad data, user profile data, running it through a selection algorithm that can be pretty complex for even a simple adserver, calculating and plugging in macros, gzipping, and doing that thousands of times per second per server, the latency tends to creep upward.

More/faster hardware can obviously help, but since everything's async from the browser's perspective you tend to run into higher costs and diminishing returns by pushing for the lowest latency possible.

They could have gone with Iponweb[1] or something like that but at Reddit's scale it might actually (for a change) make sense to roll their own.

They're one of the largest websites in the world now, so what holds for 'ordinary' websites and their best practices may simply no longer apply to Reddit.

The whole Reddit saga for the last few years is quite the story of turnaround management done well.

[1] https://www.iponweb.com/

Yeah I was thinking that if what they need or are planning is truly unique or just needs to be homegrown (a la FB's ad system tightly integrated with first party data), maybe it really does make sense. Or maybe any alternative third party company they engaged tried to pitch them a huge/prohibitive contract.

But then I was thinking that if it's going to be so important, why a 3 person team with no experience running Go in production?

Honestly I think you make a good point and implicitly they could have used those resources instead to improve the user experience (for example, improve the sites efficiency and load times)