Incident affecting Google Ad Manager | Hacker News Reader

Incident affecting Google Ad Manager

ads.google.com

179 pointscelestialcheese143 comments

Open articleView on HN

I took one for the team and disabled my adblocker so I could get a screenshot of the page.

https://i.imgur.com/9n7HuLu.png

I can view that page without turning off ad blocking on Brave.

Btw, I like your username.

Thanks <3

"Ad serving has now been restored for the affected users."

https://i.imgur.com/rnRiF6Z.png

How do I get to pull the actual image from the poisonous pile of garbage called imgur? It runs JS from 16 domains.

Use a PC. On mobile it's hopeless - on top of garbage scripts that sometimes lock up even a recent flagship phone, the service does aggressive lossy compression that's hard to work around. If someone uploads an image (whether PNG or JPG, doesn't matter) that has small details like text, forget about discerning them from the blurry mess imgur sends your way.

For single images, imgur will redirect from /<id>.png to /<id> which requires the ton of javascript (you see a message about javascript required to run). Just re-add the extension to the end of the url and you'll access the image directly.

This trick doesn't work for albums.

You can use rimgo which is a frontend for imgur. https://codeberg.org/cobra/rimgo.

Try this https://rimgo.pussthecat.org/rnRiF6Z.png

If you libredirect extension, imgur links will be automatically redirected to rimgo.

Seeing these big sites with no ads is strange. CNN, NYPost, etc. This is going to be wildly expensive for Google.

Anyone internal know what's going on? We started seeing GAM not returning responses about an hour ago, yet their status page and accounts were silent till 10 minutes ago.

Does this mean every page is loading faster? Or do they just hang halfway trying to load ads?

Should be faster I think. Pihole just refuses to resolve ads servers and generally improves performance, right?

If the ad server is instead taking a long time to respond, it'll probably make pages load slower.

Only very very poorly built websites. So all the news sites.

There are definitely a thousand people here more qualified on this stuff (I write Fortran lol) but I thought JavaScript was all asynchronous now? Shouldn’t the actually desired content load independently of the ads?

Fortran is cool, what do you work on?

Parallel linear system solvers.

Grad student, so mostly I use Fortran so it’ll plug into my professor’s code more easily, haha.

No complaints, though. It is a fine language.

Requests for remote content don’t delay rendering of the page. Everything will be significantly faster.

Anecdotally it feels like all of these sites are super fast right now. I even reinstalled the heinous Weather Channel app and it’s working wonderfully. With ads that app is essentially unusable on an iPhone 14.

CNN is zippy without the chumboxes. Time to party like it's 1999.

Genuinely curious, do you not use an adblocker?

It always feels bizarre when someone on HN complains about lots of ads on web pages.

I used to browse without one when I worked on an ad-supported site and had to implement and make sure they were working correctly. Gotta assume it's a similar situation here?

I worked at one of the top ad networks for a decade and blocked them the entire time. Would usually just open a separately configured browser window for testing.

I have a "clean" (no addons at all) Chrome profile for web development purposes.

On personal machines, yes. Work involves ads so I keep it off the work machine so I understand what's going on.

Nobody on here is going to be authorized to talk about about it while it's going on.

Eh - worth a shot. HN surprises me sometimes with anons jumping in and giving a behind-the-curtain look.

> Seeing these big sites with no ads is strange.

I've been using ad blockers for many years, so for me it's the opposite: it would be extremely strange and unsettling seeing any ads on my own devices at all.

Same. When I help relatives with their tech woes, loading a website on a vanilla browser after all these years without ads is shocking.

All these flashing and moving gizmos and pop up videos. It's so jarring, like having a flashlight flickered in your face constantly.

I have honestly no idea how people manage to work the web without adblock nowadays.

yep... I've started loading pages I want to send to my mom with adblockers turned off just to check how bad they'll be because somehow despite turning them on every time I visit by the time I get back they have been turned back off...

just teach them to download adblock. It might be an annoying 20 minutes of your life but it'll do wonders for their mental health in the long run

I'm going to guess it's one or more of:

- Lack of experience or awareness that running an ad blocker isn't something weird or niche, but is in fact basic hygiene when on-line.

- Sites that ask, demand, beg or cajole visitors into turning your ad blocker off, many of which provide detailed instructions aimed at non-technical people; your mom may have needed something from one of such sites, and didn't bother or remember to turn the adblocker back on. If I were to bet, I'd pick a link to an interesting article on a news site that employs this technique.

- Perhaps your mom uses one of those cashback extensions to get some money back when shopping on-line? They're somewhat popular and regularly advertised by vloggers and influencers. Disabling ad blocker is AFAIK necessary - for technical reasons - for the service to award you cashback.

My wife uses one of those cashback services, but we did some research on their inner workings and business model first, to be sure she's taking advantage of it, instead of being taken advantage of herself. And she follows a strict routine: she disables uBlock before visiting a participating e-commerce store, does the purchase, and re-enables it immediately afterwards.

— There are so many ads on this page, it must be intolerable. Do you want me to install an ad blocker?

— Nah I'm fine.

haha, the same thing happened with my 10 year old nephew

I was watching live TV, and he just randomly started screaming at me

I was like "Why the hell are you screaming, kid?" and it turned out it was because the TV was so loud and obnoxious

Turned out he managed to go 10 years without watching ad-infested live TV!

Same experience. People universally hate ads, yet they're not too keen on blocking them. I'd be curious to learn why.

I think some people are so annoyed or afraid of tech that once they’ve got something that at least somewhat works they avoid making any changes in fear that it’ll break their workflow.

Ads lower the barrier of entry to the web by paying for content. By blocking them and encouraging others to as well that on-ramp becomes weaker, motivating producers to pack more ads for what audience still sees them.

Funny thing though, I worked in ad-tech for ten years at a small player (outbrain). They're present on most major news sites around the world, which means I got to dig into the source of many of these sires. Easily 90% of their source was dedicated to serving ads or tracking behavior for campaigns. And we're talking a ton of code. It's like they're in an arms race with themselves where their site is an expensive nightmare to maintain so they add ads which makes their site an expensive nightmare so they add more ads.

Like having unprotected sex.

With random strangers you bumped into, that is.

Damn I tried turning off my adblock just to see but it's probably been years since I've seen those sites _with_ ads so I have no way to relate to this experience

It's always weird when something like part of Google goes down. I just don't expect it because of how massive Google is and how much money they put into their site running reliably. I guess it's interesting to be reminded that Google can go down just like any other website, it's just a really big one.

A few weeks ago, no one would have predicted a major Google service going down before Twitter.

Ouch. You got me. I’d have bet hard against this.

I, for one, are caught red-handed on this. It is so weird that Google's money-maker has been down, especially that I never have remembered when was the last time it was down (maybe practically never?)

Google laid off the one guy that was keeping this application stack running

It's a nice reminder that even the big players can make mistakes.

Would love to know why the title of this story was re-written to majorly soften it (it was the number 1 story when published as Google Ads are down worldwide)

"Ad Manager is not delivering ads for affected users" is definitely a reasonable title, I am also curious on why it was changed into a vaguer title (which is very likely a mod edit).

It's a reasonable title, but it downplays the significance of the issue that the original title conveyed.

It was a mod edit.

Google is soft-balling the impact of this, and I think the watered down title doesn't do it justice. Google's ad server is behind nearly 90% of all ads served outside of the walled gardens, and the ad server has been down now for 3 hours.

But because the ad server is a part of GAM, and their UI and reporting are still up, they can call it a partial outage.

“Ads down worldwide” was and is not accurate. The currently disclosed scope of this issue is Ad manager clients, which tend to be large publishers with significant direct ad sales.

Google search is still showing me ads.

I'm not sure where to divide the line. All Google properties still have ads on them, but despite using a vanilla Chrome I cannot get ads* outside of Google, so it is indeed larger than just large sites.

* Okay, I can get Taboola and Outbrain ads, is this counted for this discussion?

Edit: "Ad serving has now been restored for the affected users." (https://news.ycombinator.com/item?id=33917305)

Taboola/Outbrain are the only major players that don’t run through Ad Manager (DFP) because they build their own stack from scratch outside of the standardized IAB formats. All others must kiss the ring and pay the toll to Google. Facebook bought and killed Atlas, it’s only credible competition.

For a full decade Google has own the display ad server wholly, Ad selling and buying tools for automated and manual transactions, the lions share of web analytics traffic, most of the display, video, mobile, and search inventory on the entire web, and first and last bid option on all auctions against their competitors. Don’t forget the myriad of ways they collect exclusive purchase, behavior, demographic, and location data with meaningless consent. All of that data is used to multiply the value of ad sales, and is otherwise unobtainable.

If you look at their business models abstractly, you’ll see what it looks like to have Horizontal and Vertical integration, regulatory capture, price control, and centralized censorship powers. The amount of influence and operational responsibility that Google has for various aspects of US government+military would shock most cynics.

If there is justice in this world, most of what they have been doing will soon be illegal and should have been the whole time. It’s a dangerous duopoly with all non FAANG publishers and tech fighting over the remaining 10% and still getting fabulously wealthy from it.

All of froth from the quarter trillion it makes each year has flowed into AI for at least a decade. It has all of the ingredients to have an exclusively owned AGI, and may have had one for several years already. I’m now well into the realm of speculation, but I submit that the AGI has or will have access to the worlds largest corpus of human knowledge along with intimate accounts of individual human behaviors. All to sell fucking ads.

Disclaimer: I was in Google Ads, but a long time ago, so this may be outdated. In particular, "Ad Manager" is not a term that means anything to me.

Nonetheless, there was:

AdWords (ads in Google Search, the big money maker, which stayed up, and I tried it),

Display Ads (which I guess was down), and

AFS (ads that appear on third party search pages, like Target or Walmart, or Google properties other than Search).

Display Ads (or DFP) are the blinky things everyone's complaining about: ads on third party pages other than search pages. I never really worked in that area so I don't know much about it.

If Search Ads stay up, the Google Money Machine is still running, just not on all cylinders.

And the internet kept on working. In fact, better, faster than before.

You are ignoring the concept of debt. Everything doesn't immediately implode. Especially for a temporary outage.

Don't take for granted how many ad supported sites and services you use :)

I have a website with several writers that serves almost half a million monthly pageviews and we are entirely ad supported. The only reason my writers have jobs and we are able to provide the recipes and information that we provide is because of ads.

So many viewers, why not start asking for voluntary donations? You could dial back ads, and thereby tracking, to the same degree you receive donations. Over time see how far it goes.

This is naive. People simply don't donate, it doesn't work. It just doesn't

This is naive. If you don't try, you never know for sure : )

But that easy retort aside, there are many creators, who live off of donations. And as a matter of fact, it also cannot be correct, because I count myself as "people" too and donated to several projects/creators.

We aren't a "project" and we aren't individual creators. It works okay for certain types of audiences, but does not work for my type of audience. I know my business model well and there is a reason none of my competitors ask for donations either - it doesn't work.

Interested to hear how many "hands on deck" this gets vs a Google Cloud or Google Workspace outage. Don't fuck with the money!

Does GCP not make Google money?

You wouldn’t think so based on the quality of their support team

No. GCP consistently loses money.

One of many sources: https://www.theregister.com/2022/07/27/alphabet_q2_2022/

> Google Cloud brought in $6.3 billion of revenue, a 35 percent year on year increase compared to Q2 2021’s $4.6 billion revenue. But the $858 million loss was 45 percent higher than the $591 million deficit recorded last year.

You might ask: "how is that possible, with those same margin-intensive prices that AWS and Azure use?" I have no clue. It makes no sense, given the prices they charge.

What is or isn't cloud is different at every company on top line but especially bottom line. For example how far down the stack are engineers still considered cloud when they are supporting both cloud and internal products? Investors are fine losing money on growing cloud and also love to see ad margins up? Then everyone is cloud.

I don't think salaries are the driving factor behind GCP's balance sheet, but that's just my opinion. I obviously can't wave a magic wand and find out for sure either, but my own back of the napkin math can't see that reasonably working out. They brought in billions of additional revenue, but their losses also increased.

Similar reasons to a company with 30 tps 8am - 6pm has a 3 h/a Kafka running on a k8s cluster, multi master MySQL cluster, all the trimmings

Built to scale that isn’t here today ? Resume Padding ? Data centers are expensive and slow unto build

GCP revenue losses from being down are approximately lower bounded by (downtime in seconds * revenue per second). Their costs don't decrease from being down.

Labor costs to keep all of these features/servers/infra running is expensive. Google is probably charging less due to competition from AWS/Azure/Oracle.

Similar to how Amazon.com is subsidized by AWS. GCP is subsidized by the ad biz

P0, page everybody. But their incident response planning and training is on point so there’s one incident commander and everyone doing individual tasks reports to them. I suspect not more than a dozen cooks are in the kitchen as it were. Any more and people would be getting in each other’s way.

I seem to recall that ads runs on one of the largest distributed MySQL instances on the planet. That’s probably the most complicated component at play here. Then again everything has to have active, secondary and tertiary so maybe it’s something upstream like dns dos protection or cdn.

Not a Google employee, just guessing.

I like to think that at a certain point, more “hands” doesn’t help solve the issue faster (maybe even counter productive), so that in both circumstances the outages get the same amount of hands — the maximum.

This is one of those good outages

Good, keep it that way forever

would you pay subscriptions instead?..

Interesting question. I probably would not, which might mean I would spend less time on the internet reading useless stuff and more time outside on my mountain bike.

There can be subscriptions and mountain bike riding at the same time you know.

I do, and in some cases, I still get ads.

No, and I also use an adblocker

Would you offer the option to?

services likely do studies to decide do they want to be ads based, or visitors are willing to pay for subscription.

And then, historically, show ads to their subscribers.

Cable, streaming video, etc.

Because advertisers pay more to reach people they know are willing to pay for stuff. It's whatever the opposite of a virtuous cycle is... a race to the bottom?

The conventional wisdom is that offering paid subscribers ad free experiences poisons the well for advertisers. After all, it eliminates the people that have enough disposable income to pay a nominal fee!

Not sure how this translates to targeted ads. Would offering such a thing violate Google’s TOS?

There's a New Zealand Journalist, David Farrier (https://en.wikipedia.org/wiki/David_Farrier) and I actually subscribe and pay a monthly fee to be able to view what he writes. I think he's really awesome (some will disagree) but I just like him. He also has a TV series on Netflix called Dark Tourist if you're able to see that.

He has a new movie out at the moment that I really want to see, but his last movie Tickled was _crazy_ like... Tiger King type craziness.

So yeah - if I know someone who actually puts in the work to give me quality output I'd pay for it. Just like I do now.

Yes, can I subscribe to the internet?

They have the internet on computers now

Ads are an enabler of bad products. If things required a subscription, half the shit on the Internet would be gone and for good reason. Also, often you get ads anyway regardless of the subscription. This is a false dichotomy.

would you not serve me ads if I did?

All newspapers/magazines, VHS/DVDs, cable TVs etc. were are 'subscription' in the sense you had to pay for them. Ads were still there. If paying a subscription for websites was normalized then soon enough they would as well be filled with ads.

Yes. But I wouldn't subscribe to "content" that is used as a minimum skeleton to hand ads on, and I include the google search engine in that definition. I'm happy to pay for something I value. Most web content I value at $0 and may browse opportunistically but I would never pay for it, including by looking at an ad or providing my email

I've spent hundreds of hours of my life contributing to Wikipedia pages and may spend hundreds more on open source projects. Capitalism has made us label any not immediately self-serving as illogical. We are not homo economicus

There is this concept of supporting people you like without them having to charge people. The more you can read upfront, the more you can make an impartial judgement if you actually like them. A lot of people subconsciously convince themselves that they like something just because they spent money on it. If I like a project or open source software that I use then I go support the developers. A lot of other types of arrangements are basically signing a contract waiving your rights before you've even had a chance to form an opinion.

What's the difference? You already pay the ads on top of every product you buy making everything you buy more expensive, it's already a tax.

The only difference I see is all the money on salaries and expenses related to ads is 100% a pure economic loss in the ad model.

(Of course I'm only talking about the economics here, there's a lot of other bad side-effects of ads not covered here)

It always amazes me being reminded that the majority (vast majority? _overwhelming_ majority?) of Internet users do not have ad-blocking. Bless their hearts for subsidizing so many sites for those of us who prefer not to live in a real-life manifestation of Idiocracy.

Crazy to see websites load this fast without ad-block. This event must be eye-opening for non tech-people...

Google is A/B testing impact of switching to their new business model which will be paid search+AI ;)

I wish

I'd like to see some normalized numbers to see how the absence of ads affected sales, neg, pos or neutral.

Finally iOS safari adblocker!

Might I recommend Firefox Focus for Safari ad blocking?

You’re joking but adblocking has been available on Safari iOS for several years

I didn't know! Mainly because I just got in a habit of avoiding the crappy ad-ridden sites when on mobile.

If you'd like to keep enjoying the ad-free experience:

1Blocker: https://apps.apple.com/us/app/1blocker-ad-blocker/id13655310...

AdGuard Pro: https://apps.apple.com/us/app/adguard-pro-adblock-privacy/id...

Or, for something different, Orion Browser by Kagi runs Chrome or Firefox extensions:

https://browser.kagi.com/

Is the link to a description of the problem, or it is just an example of a dead page? (Not sure if I’m getting a dead page because of their issue or because my ad blocker blocked it, haha).

Wonder what the $ loss per second is…

From their recent 10Q for investors, they list a revenue of $7.8 billion for "Google Network" in revenue for the preceding 3 months before September 30, 2022 [1] There are 90 days in 3 months, so their revenue for a day is approximately

$7.8 billion / 90 days = ~$87 million/day

$87 million / 24 hours = ~$3.64 million/hour

$3.64 million / 60 minutes = ~$60,740 thousand/minute

$60,740 thousand / 60 seconds = ~$1,012/second

And, depending on their agreements with their advertising partners, they might be liable for some of the profits lost by their partners.

[1] https://abc.xyz/investor/static/pdf/20221025_alphabet_10Q.pd...

edit: It was initially using the wrong row, it should be correct now.

I don't think that's the right number to look at, given search ads aren't down. You probably want just the "Google Network" line item, which is about 1/6th of that number.

You are correct, I just misread it.

edit: It should be fixed now.

Exactly. Although this is likely one of the biggest days of the year for display and video advertising, so it'll be likely more than 1/6 of $25m/hr.

It doesn't look like it's affecting their search ads (or if it is, it's not fully down for them), so not the full advertising revenue of GOOG. But it's still likely tens of millions per hour.

Haha go search and view a recipe. There’s no ads! It’s glorious

Doesn't immediately solve the problem of all the SEO spam that exists out there. But in the long term it could. I guess that's what it'd take to make google search good again. For google to die...

If you want this to persist, you can always use ChatGPT.

...until someone figures out how to embed ads into LLM responses, then that'll be the Google-killer.

Unfortunately, this has already been figured out. ChatGPT is no worse at this than average YouTuber. See https://imgur.com/a/8EIEGL7, via https://old.reddit.com/r/ChatGPT/comments/zdjowg/i_had_the_w....

MindGeek ads are still working. Shows why it's important to diversify your ad delivery

They had outages far more than Google ads outages.

They would love to use Google if it wasn't against the TOS and nobody wanting to deal with them because ethical and morals clauses.

Source: Worked there in the past

I still see ads when using google search. What’s the exact scope of this issue?

...and it was at that moment that riotous celebration among all mankind swept the globe in unanimous rejoice.

In unrelated news, productivity around the globe went up 5%.

... A record amount of text content in the form of articles were read in their entirety without mind-flicking distractions.

Relevant Onion article: https://www.theonion.com/the-economist-to-halt-production-fo...

Attention spans got a boost.

From their page it's not even a "Service outage" but instead a "Service disruption." Wonder what an outage would look like.

This page is blocked by archive.org, so we cannot record the url there. I think this would be an interesting page to allow archives of.

Google does not want anyone remembering their service can go down.

Traditional “tech news” sites won’t cover this either, as it’ll bring attention to the ads that power their properties and may inadvertently encourage people to get ad-block, if they call out the presence of ads.

Either way, Google’s PR team will be cranking out some distractions shortly.

I wonder how much CO2 emissions have been saved from 24 hours of the internet without ads

It would be neat to see how much data usage dropped globally because of this.

Back up as of 10:29PM ET.

I can't load this link thanks to pihole :smug:

Burn it before it lays more eggs.

Beautiful

I trust that payola pre-written content is still going up as usual, though.

What’s payola?

back when radio stations played records, record companies would pay DJ's to play specific songs or albums [1]. I was a scandal at the time. It's almost quaint now.

1 https://en.wikipedia.org/wiki/Payola

Technically still illegal (IANAL), since advertising is supposed to be labeled as such. But widespread.

Is this a party?

Didn't even notice. Ad blockers work?

an outage? or perhaps a dream…

World becoming a better place momentarily

and nothing of value was lost

Good

thank god

Reply on news.ycombinator.com