Google used past bid data to boost win rate in advertising auctions | Hacker News Reader

Google used past bid data to boost win rate in advertising auctions

wsj.com

228 pointsAaronFriel69 comments

This view from a publisher convinced me in 2017 that Google should not own both the dominant business in DoubleClick's market and in AdExchange's market:

https://talkingpointsmemo.com/edblog/a-serf-on-googles-farm

> But here’s where the rubber really meets the road. The publishers use DoubleClick. The big advertisers use DoubleClick. The big global advertising holding companies use Doubleclick. Everybody at every point in the industry is wired into DoubleClick. Here’s how they all play together. The adserving (Doubleclick) is like the road. (Adexchange) is the biggest car on the road. But only AdExchange gets full visibility into what’s available. (There’s lot of details here and argument about just what Google does and doesn’t know. But trust me on this. They keep the key information to themselves. This isn’t a suspicion. It’s the model.) So Google owns the road and gets first look at what’s on the road. Not only does Google own the road and makes the rules for the road, it has special privileges on the road. One of the ways it has special privileges is that it has all the data it gets from search, Google Analytics and Gmail. It also gets to make the first bid on every bit of inventory. Of course that’s critical. First dibs with more information than anyone else has access to. (Some exceptions to this. But that’s the big picture.) It’s good to be the king. It’s good to be a Google.

> There’s more I’ll get to in a moment but the interplay between DoubleClick and Adexchange is so vastly important to the entirety of the web, digital publishing and the entire ad industry that it is almost impossible to overstate. Again. They own the road. They make the rules for the road. They get special privileges on the road with every new iteration of rules.

There are a lot of people bothered by this, so this data discovery could prove very expensive for Google.

After years at Google on DFP/Ad Manager these quotes read kinda sad to me because this is a lot conspiracy sounding stuff and in reality it’s just not that kind of operation.

Sell-side ad tech (ie. DRX) doesn’t make much money for Google, relatively. If Google likes AdX why isn’t YT inventory on it? Because they actually like the buy-side tools. Sell-side gets basically no love, no real fresh ideas. I say this having the utmost, deep respect for Ad Manager/AdX’s current and former eng & product leaders.

What happens here isn’t about market dominance, no one is thinking that. They already killed the display publishing golden goose, by Google standards there’s nothing to dominate in pub ads the business died. It’s a small market, now, putting display ads on other people’s websites. Big business is O&O which exploded around it, this is obvious.

Projects like this are about two forces colliding: smart engineers without a deep understanding or connection to the publishing industry and an internal org losing relevance/importance as its size relative to its peers decreases.

IMO, (this is definitely just my opinion) the real real problem is none of the ads you can by through AdX actually work. Using OA a lot leads to users tuning out your ad units because they are often really weird or bad-quality & register as noise.

Sure DFP isn't where the money is, but on the buy side Google is abusing their monopoly and data in every way they can. The simplest example is that Google decided on privacy grounds to restrict access to the raw logs from all other ad-tech (typically running on AWS), while at the same time launching a new product called Ads Data Hub in Google Cloud where you can actually access all that data. If it was really about privacy they wouldn't use the data at all. The whole privacy thing is bullshit, they've just found yet another way to give themselves an advantage on the ad market.

Google has lost their "don't be evil" thing many years ago. Now they're just a global privacy disaster that's abusing their monopoly in ways that even Microsoft in the old days wouldn't dare.

I don’t think this is a convincing argument. I don’t believe it’s good to allow third party tracking pixels on youtube.com (for instance) and the alternative way for buyside to get that data is ADH. It sucks compared to prior state for people who need to use it, but it’s necessary to provide user protection.

What are O&O and OA?

OA is just open auction which is what most people think of when they think of AdX, ie. more or less anyone can bids on your inventory.

O&O is owned and operated, where you actually own the audience. In the simplest case this is like YT or Insta for Google and FB but it can be kinda nuanced with more traffic falling under the umbrella than you’d think. More and more this is where energy and attention is being focused, especially as cross-site data becomes scarcer.

Could you explain O&O more? Surely all sites where ads appear are owned and operated by someone, that sounds kind of tautological. Or do you mean, the ad infrastructure itself is owned and operated by the site owner - in which case, what is there for Google to do?

It refers to when the same company both owns the site/content and operates the advertising for it.

Google has search, maps, gmail, youtube. Facebook has its social media sites, Amazon has amazon.com, Yahoo has its own sites. Bigger publishers with enough scale also run their own ad infrastructure, for example Washington Post has some of the most advanced in-house adtech.

You want to flip this. Google/Facebook operate the ad tech and own the site (or, more accurately the audience) so they don’t need to deal with third-party publishers on a per-request basis at all.

The point is that selling ads on third party sites is not where the money is at these days. It’s in selling ads on your own properties. For Google, that’s YouTube, search and Maps. For FB, that’s FB and Instagram.

I was going to ask as well. There’s an incredible amount of jargon around internet ads and marketing in general. Impenetrable to non practitioners imo.

First is “owned and operated” as in YouTube is a property that is “owned and operated” by google.

And guess what, thinks will get way worse with FLoC as it will add an technical unavoidable dependency on Google to all advertisers, even those which are currently independent of Google...

While is at the same time questionable that it actually improves privacy at all and likely will not be ad-block able.

> likely will not be ad-block able

Surely if it's local you can delete it or substitute for dummy data.

Sure if you custom compile your chromium or similar...

Or decide not to use chromium, it's better for the web if we bring more balance in browser usage.

It's not about what I do.

it's about what the majority of technical not very versatile users will do.

Convincing them to install a ad-blocker is one thing, but convincing them to use a completely different browser is way harder.

Furthermore if google pushes it through it's quite likely that it's just a matter of time until many sites will refuse to work if you don't have it enabled (similar as some sites don't work if you have an ad-blocker enabled).

So then it would be just a matter of time until many alternative browsers (e.g. firefox) are forced to have it enabled by default.

A big scary extension that can read all your data should be able to do that.

Gmail data stopped being used for advertising in 2017.

https://www.nytimes.com/2017/06/23/technology/gmail-ads.html

I don’t believe it. I get target ads in gmail based on search or browsing queries.

You see ads, but your Gmail data isn’t used (ie. scanning emails and adding insight to your advertising profile).

These two comments don't contradict each other.

> There are a lot of people bothered by this, so this data discovery could prove very expensive for Google.

Apparently none of the right people are sufficiently bothered, because it’s been this way for a decade and we’ve seen no real resolution to date.

I don't know if we'll see a "real resolution" or not, but Google is currently unpopular with the GOP and this story might have the power to sour many Dems on Google. A bipartisan push for antitrust measures would be very dangerous for Google.

Even if we don't see an effective clipping of Google's wings in Congress, the court case was already likely to be expensive just on the basis of the cartel with Facebook the Texas lawsuit has already showed. Recall that the EU fined Google €1.49 billion for restrictive contracts with AdSense. - https://edition.cnn.com/2019/03/20/tech/google-eu-antitrust/...

Can someone help unpack this?

So from first read it sounds like google is using bid stream data from their ad server, to inform suggested bids that would clear/win for customers buying this inventory through dv360/admob/adwords/whatever they brand buy side as now? Which sounds like a win/win for google, higher pub revenue, higher margin through buyers.

But other DSPs also have a large amount of bid data, even without clearing price on a loss notification (do any exchanges disclose?) they have so much scale that they can accurately price? If you're consistently losing auctions you know the clearing price is higher. But then I guess another problem is:

What about with first-price auctions?

This could theoretically help with bid shading if google does in fact leverage 'private' clearing data on what the true value of the impression is?

I guess that goes to the kind of unclear ethics of owning all sides of the transaction. Help your ad buyer win at a lower price by not bidding as high, but publisher loses out on revenue. Or the above up the price on all sides. I guess for Google it's whatever has the highest margin %

Google adx actually discloses clearing prices for auctions where you participate in, so this is even stranger complaint.

https://developers.google.com/authorized-buyers/rtb/request-...

That's what I thought but didn't know for sure. I know that clearing price is in the official openRTB spec on loss notification.

I think Google also optimizes for lack of a better term, ‘engagement’.

So, even if an ad has a higher margin, it may be better to ‘slot in’ a lower margin ad that would get more aggregate clicks, so the net profit is higher.

Poor example:

Ad is $10 per click but is only clicked once per month versus an ad that is $1 and clicked 20 times per month.

Probably better for publishers and consumers too. That's part of FBs algo too, though the CPC bidding is outdated in FB I think it leads to a bunch of crap you're just optimizing for bounces & fat thumbs.

Google operates one of the largest exchanges for advertising - ad sellers place their inventory up for auction using Google tools and each integrated ad network can place a single blind bid to try to buy the inventory.

The catch is that Google operates their own ad network and can leverage their visibility into the exchange to be only bidder who is not blind.

For any given inventory, Google can wait for every ad network to submit their bid and then make a decision as to whether or not they should place the highest bid to win the auction.

The information asymmetry allows Google to bid more efficiently - if they're prepared to bid $5 but they know everyone else is bidding $1, they'll lower their bid to $1.01.

Their cheating is exactly as you think it is, but the last example is not how this market works. It's a second price auction, so after 10 other bidders the highest bid is $1 it wouldn't matter whether you bid $1.50 or $10, you would always pay $1.01

But the rest of it is true, Google has access to the data on all sides and is abusing it. The acquisition of DoubleClick should have never been approved and the earlier we break them up the better.

Facebook uses a second price auction. Google does not.

https://www.adexchanger.com/online-advertising/google-switch...

> Google isn’t abandoning second-price auctions for products where it controls the end-to-end buying experience. YouTube, Google Search, AdSense for Search and other Google properties will continue to use second-price auctions.

Google owned properties are not being discussed here; they aren't exposed externally and therefore have no external data to collect.

Thank you. That is a much clearer explanation than in the article itself.

Aren't ad auctions second-price though? I thought the magic formula for ppc ads was that the winner pays whatever the second highest bid was.

That used to be the case, and so for space efficiency the long tail of 3rd and lower bids from every company was truncated before even going into the auction process.

That is about the only hint you need to know to understand what project Bernanke was (apart from the willingness to assume that it was fair, and the past bid data was used only to compute initial experiment constants in the adjusted bidding formulas).

Much of the industry has returned to first-price auctions (for many reasons like the rise of header bidding)

I suppose it's nice to have confirmation. I was working in the keyword bid ad space around the time Google bought Doubleclick, and they essentially ate the margin that every company in that space was working in. It was apparent that they were front-running the auction process. Agencies that survived primarily became white-labeled adwords support teams -- my group just shuttered and we all moved on.

Yeah, I think everyone in the industry knew that this was likely to be shady as all f*ck. But the fact that a bunch of businesses closed shows that this caused real harm, and indeed this action by Google has definitely cost their still existing competitors ridiculous amounts of money.

> The program, known as “Project Bernanke,”

Oof. You'd think people would learn by now that funny codenames for questionable projects never are a good idea...

Can someone explain for my sake what bernanke means in this context? He is a person but I'm not familiar with it's relation to this.

Ben Bernanke ran the Federal Reserve for a couple terms. So he controlled the US money system. When things get hard for the economy, like they did when he worked there around 2000, a solution is to "print money", which is basically the government deflating currency by creating more USD. Much like they're doing now with stimulus checks and "printing" trillions of dollars that didn't exist. It can keep the economy moving, but someone will eventually be footing that bill. So this project, called Bernanke, is kind of a joke name about how it's printing money at someone else's expense.

I mean if I were going to have a project that would allow me to print money by giving me an anti-competitive advantage I would make sure my project name steered clear of references to money printing or anti-competitive behavior.

I think I would have called it project AD-Efficient

And then if anyone ever complained about our project I would laugh a bit and say I never actually expected it to be so beneficial, that's why I made the project name sort of a joke, it ad efficient, but it's also a deficient - heh heh. I'd wear a cardigan and some granny glasses on the stand as I explained my little joke project. Anti-competitive me? Nobody who makes such a lame pun could ever be an evil genius!

Helicopter Ben is famous for printing money, this is Project Money Printer.

Or the project is so profitable it is like money is falling from from the sky.

Seriously, they might as well have called it "Project Helicopter Money."

What's newsworthy here?

I would be surprised if Google don't use their past bid data.

Im trying to unpack it to - that's my take as well.

I think? the controversy/what TX is arguing is that it's not fair because google owns both sides. That using publisher bid data - that if they were an independent exchange they wouldn't have access to - to inform bids for buyers is anti-competitive.

I personally don't buy it, but maybe it's so complicated a jury might.

It is 100% not fair. If we think that ad auctions are like security markets, then this would clearly be securities fraud.

Google owning both sides is fine if the information is firewalled, and they compete on the same terms as the other exchanges. If this is not true, then from an EU perspective, this is anti-competitive. Dunno about the US, but the fact that Texas is bringing a suit suggests that they think they have a case.

The issue here is that securities markets are (sometimes) meant to be fair for complex reasons that only really apply to securities markets. Google doesn't owe their customers a fair market. No one does. Nothing legally requires a market provider to provide one.

> Google doesn't owe their customers a fair market. No one does.

do you mean this as an exclusively textual interpretation of existing laws?

or something else entirely?

because I'd assert that any and every agent has the responsibility to act in a fair and balanced way. Both ethically and legally. Alphabet may have found a loop hole in existing laws, but isnt' that the whole idea behind antitrust. You're free to eliminate all your competition by being better than them but not by restricting their ability to compete.

There are 2 separate issues here: the moral and the legal.

The moral I leave to you. Morality and fairness are always matters of personal opinion. I don't think fiddling markets you control is "fair" for what it's worth...

The legal is pretty cut and dry. Especially for b2b like this. Maybe we should change the law? But then you're going to have to change it everywhere. Check out the shit supermarkets pull, they're a good example of a market provider that plays both sides. Google might have been unfair, but they're not in the top 10,000 for unfairness in this regard...

Sure, that's why I said it was ethical more than legal. That being said, a lot of the tech company marketing materials talks about how fair the ad auction process is. Stories like this make a mockery of such arguments.

I really want this case to work, as I think the dominance of FB, Google and soon Amazon in this space is really bad.

Sorry, I misread you comment!

It seems like if this case succeeds then it would have big effects in other industries. Supermarkets are famous for legally "abusing" their market positions and data...

Google is the only major advertising company which requires you to use their own ad server (DFP, now known as GAM) in order to compete in real-time with its own inventory. Since they are the largest advertiser, everyone is basically forced to use Google's ad server to manage all/most of their inventory, including inventory from other advertisers.

My understanding from the article is that they were using the data they collected about publishers' other advertisers within the ad serving system to give themselves an upper hand. This is not surprising when you learn all of the other more obvious ways that Google gives themselves an upper hand in GAM.

I got the impression that it's similar to Amazon, where officially no 3rd-party-seller sales information (from the Amazon Marketplace) is used to decide what products Amazon would sell itself, but it later came out that they simply lied about it and did use the data to get an advantage.

Archive link: https://archive.is/HBPOb

It generated hundreds of millions of dollars in revenue for the company annually, the documents show.

This isn't suggesting that Google's success is fundamentally due to these practices - unless ofc there are a large number of projects like this.

Alphabet's 2020 revenue was $180B.

According to this[0] ad revenue accounts for 82% of Alphabet's first quarter revenue in 2020 and I imagine a similar portion of previous quarters. You may have accidentally illuminated why this is probably a bigger scandal than it appears at first.

[0]https://venturebeat.com/2020/04/28/alphabet-earnings-q1-2020

You may have accidentally illuminated why this is probably a bigger scandal than it appears at first.

I don't understand what you mean.

This is the bread and butter of google. The meat and potatoe if you will.

It kills me that they quoted Hal Varian in this piece. Hal is an Micro-Econ legend (full disclosure: I interviewed him whilst he was still at Berkeley and we exchanged 1-2 emails over the years since the move to Google). I highly doubt—and more than anything, hope—that he didn’t have anything to do with this. It’s important to see good people succeed, it gives hope to the rest of the universe that success isn’t merely winning a race to an unethical finish line!

Not a fan of Google in any ways, but isn't it their data? They can do whatever they want (unless they expose users' privacy etc) with it, and they are using it for competitive advantage, which is the most natural thing to do.

If you act on behalf of others, you are not entitled to make sure it benefits you above your customers. Maybe this doesn't exist in law, but it sure as hell exists in ethics.

And the data belongs to the person who generated it (so the people and sites involved), not Google (again ethically, potentially not legally).

You’d be surprised. This isn’t even a hard and fast requirement in fields like life insurance and retirement planning. In fact, the Obama administration tried to put strict rules in place, then the Trump administration halted them, then watered them down substantially, and the Biden administration only recently allowed the watered-down rules to go into effect.

Basically, unless someone specifically tells you they are acting as a “fiduciary”, or in very narrow circumstances like an attorney or Realtor (but not an investment advisor), they have no obligation to put your interests above their own.

The most common case, still legal, is advising clients to put money into mutual funds that carry a sales commission that is not disclosed to the buyer, even though those funds are rarely the best choice.

https://www.shrm.org/ResourcesAndTools/hr-topics/benefits/Pa...

Again, as I keep having to point out in this thread, I'm talking about ethics, not law.

The fact that this would be illegal under EU law makes no difference to the ethical point I'm making here.

The ground rules that we’ve decided on (at least here in America) is that agents should be presumed to be acting in their own best interest, unless you have specific assurance that they aren’t. You might decide it’s unethical, but if it’s literally standard operating procedure for industries like retirement planning and insurance sales, then either your definition of ethics is too narrow, or everyone is operating unethically.

If for only that they are using past data, nothing to worry about, with their power they could simply earn more from just investing.

How many other projects has Google worked on that have hurt clients/consumers?

Reply on news.ycombinator.com