What if performance advertising is just an analytics scam?

> Technically, when someone does a Google search for “Williams Sonoma Cast Iron Skillet,” they probably would have clicked on one of the first 10 organic results, EVERY ONE OF WHICH leads to their website. But, y’know what ol’ Billy Ma’s performance marketers couldn’t then do: prove their value to their bosses.

> [picture of that search term and williams sonoma ads with shopping links]

The main problem here is that if Williams Sonoma was not advertising on that search term, Lodge and Food52 etc etc would, and then those companies would be above the Williams Sonoma organic placements.

The spend is necessary in a defensive way because Google creates a bidding war even for the hyper relevant.

edit: I just checked and if you search "williams sonoma skillet", if WS was not paying for [green] then the very first "result" (ad) would be Food52 [red] https://imgur.com/a/9Nnxs6h

I just tried "airbnb paris" and the first result is, somewhat predictably, an ad that is not airbnb. But the second one is also an ad, this time from airbnb. So they clearly didn't keep their spend dialed down to zero, and are aware of the need to advertise on their own keyword.

There are so many holes in these arguments that you could drive a truck through it. Which is so infuriating, because a big part of my professional career consisted of watching all of Rand's SEO videos and really appreciating them. I really thought he was a genius. But then over the past few years, he started sharing more general views on entrepreneurship, and those takeaways just didn't really make much sense. Basically, his own VC-funded company turned into a shit show and suddenly he started advocating against VC in general (as opposed to taking an honest look at the mistakes that his company made). So in the past few years, I tried to reconcile in my head: how can a genius make such imperfect conclusions? My initial takeaway was that he's blinded by his own mistakes and shifting the blame, which seems perfectly reasonable and understandable. Frankly, I would probably feel and act the same way.

But after reading this article, it finally dawned on me. He makes imperfect conclusions in everything he touches, it's just that in some fields those conclusions can be more easily proved to be wrong than in others. SEO is the perfect field where a polished presenter can get away with imperfect conclusions for years - trust me, I know, I made a living for years in this field, and I am very familiar with the nature of this work. Most of the time, you have no idea what the black box really does, and instead you're just trying to guess what might have happened. Most importantly, there are many ways to skin a cat in SEO, and just because your approach is net positive doesn't mean that you truly are delivering the global maximum (or that the net positive gain was ROI positive). In short, it's impossible to know who's right and who's wrong, and Rand's videos convinced me that he's right, but I am no longer sure. I just rewatched one of them, and can easily see how his conclusions are just... opinions.

While we may or may never find out if his SEO opinions were the global maximum, we can quantifiably demonstrate that his opinions on content marketing are not solid. This whole essay he wrote can be replaced with "hey performance marketers, don't trust the platform numbers and instead do your incrementality studies." Platforms like Facebook will give you those for free if you reach a certain spend level, and you can also get them from 3rd party providers like measured.com. In other words, if you're a performance marketer and you're not conducting incrementality studies, then you're very early in your career and are not following the best practices. Simple as that - no need to extrapolate from there and reach all sorts of additional conclusions (which is obviously a pattern in Rand's behavior) - calling into question a perfectly investable marketing channel, conflating the needs of a public company with everyone else's needs, using words like scam, etc.

I am really disappointed to have to write this, but you would have been better off not reading this article. If Rand is really advocating that the majority of entrepreneurs should follow his advice and focus on PR instead of performance marketing, then perhaps an honest thing to ask would be - how is that working out for his own company? AFAIK, SparkToro is nowhere close to replicating the growth of his previous company, which is honestly disappointing for someone with such a huge reach and name recognition.

I was at one company several years ago and made two big changes to digital advertising:

#1 - Eliminated all paid search other than some limited branded search terms and shifted all the money to affiliates who were way better at making profits on the keywords we were competing on

#2 - Eliminated all display advertising after running numerous experiments showing it provided almost not incremental conversions, even though the platforms happily took credit for them.

Those two things drove our blended CAC down substantially and by building better affiliate relationships, sales actually increased.

The lesson here is that you need to try a lot of things out and you should be continuously questioning what you're doing and running specific experiments to gut check effectiveness of any ad platform that is slapping cookies on wide groups of users and claiming conversions.

My suspicion is that this is near impossible at any large organization, even one as new as Airbnb. I can just imagine someone walking into a team of 20+ performance marketers and suggesting they need to experiment to determine if any of it is remotely effective. COVID forced them into this but it's something that they should have already been doing.

I also suspect that the top line focus/obsession of most VC-backed companies make this type of exercise seem almost counterintuitive. Don't mess with or question the momentum.

So much of what we rely on today for Internet applications and infrastructure has at its core a fallacious advertising revenue model. As companies who spend money on advertising realize the low returns they are getting, they will turn off that ad spend, which will turn those platforms into increasingly needy and demanding in your face advertisers, being more intrusive with your data. Since the average "user" is the product and not the customer, users will continue to see fewer and fewer features that don't help with advertising revenue and the quality of the overall service to the user (not the customer, the advertiser) will decline.

We're already seeing that. The quality of Google products continue to decay. Facebook and LinkedIn are increasingly both becoming shallower advertising hustles (LinkedIn just this week turned off post notification for events to force people to buy LinkedIn ads). As other apps and websites get snapped up by these FAANGs, we'll start to pine for the Internet that was 2008. The decay is already well under way.

What is truly sad is that some of the smartest computing minds of our day are spending their efforts at these FAANGs not advancing society but rather helping keep people more addicted to social networks, optimize for clicks on video and web streams, pushing products in all your channels, and optimizing for the wrong things. How have we gone so astray?

People have a tendency to view advertising as ineffective on them. "I never click ads." Mostly this is true. Most ads people see don't make them do anything. People see a lot of ads though.

Meanwhile, it's a marketing cliche that "half the budget is wasted, but we don't know which." It's also true that google or FB provided analytics, using default settings often grossly overestimate ad effectiveness. All true. A journalist somewhere is writing a version of this article at any given time.

But... From the merchant's perspective, the existence-proofs for advertising's effectiveness are undeniable. Launch a site. No visitors. Advertise. Now there are visitors. People subscribed to something or bought something. The ROI may or may not work, but the principle isn't in question.

For a blank slate, newly launched business performance marketing is easy to measure precisely and you can have a reliable ROI. For BMW, GoPro or geico insurance... the world is more complex, ROIs are more theoretical and "half the budget is wasted, give or take 50%" applies.

The same was true for TV. A mattress store run ads with a crazy guy screaming "Sale!" and the next day a lot of mattresses get sold. The fact that ads made people come buy mattresses is trivially true, from the merchant's POV.

Related: I happened to be acquainted with a founder of one of the major coupon sharing sites, after they’d sold it for a rumored 8-9-figure sum.

It didn’t make sense to me that it could be so attractive to brands to promote coupons there. People have already decided they want to buy the product; why would you want them to go hunting for a big discount after they’re already sold?

He told me that the people who like it are the advertising staff and consultants, so they can generate evidence for their bosses/clients that their campaigns are working.

I get that it can be partially effective. Sometimes I’ll think “I’ll buy this product if I can find a coupon that gets the price under X”.

It’s just a funny old world when an advertising professional is motivated to spend money with a third party to give their customer a big discount on a product they’ve already decided they want, in order for the advertising professional to justify their existence.

The example given in the article seems very off-base: the author is vaguely planning to purchase some type of planner, and he gets an ad for a specific planner from a specific retailer. There has to be a pretty tiny chance that he would have ended up at that exact site without the ad, so if the ad does lead to a conversion, it's incremental revenue.

(So I do think that performance advertising is partly a racket, but it can work really well if you know what you're doing).

One additional thing people don't call out is that a lot of the budgets spent on these platforms are "learning" budgets. Agencies play this card really well. They'll tell you, "oh, you need to increase your budget, and test all these different combinations of ads/targeting/landing pages/etc so that you can learn what works (or the AI behind the platform can learn what works)". And obviously, in "learning" mode, you're ignoring the ROI.

I've seen people spend substantial amounts of money in "learning mode", and the platforms are kind of incentivized to make the learning less efficient so it takes longer and more spend for you to get to ROI positive (or to learn that you will never get there).

During pandemics, I got really suspicious about adwords.

My ads were working reasonabilly well considering the low investment I was making, with a fair amount of prospects filling my contact form on a low but steady rate. I was satisfied with the return I was getting.

However, the pandemic caused a significant drop on my product's demand. I thought I was going to get little to no contacts from the moment the lockdown was announced on, but:

1) I kept getting clicks at basically the same rate -- therefore my budget kept being depleted as it used to be;

2) Bounce rates increased A LOT;

3) The few actual people who got in touch were not actually looking for the product I announced, but similar ones (which I didn't announce nor sell);

So, according to my experience, I can't say adwords totally doesn't work.. but I'd say their algorithms are optimized to spend your money regardless of the results you're going to obtain.

I've been beating this drum for a decade.

I once even had a marketing department shut down with analytical proof. I got tired of the marketing department with probably 50x the IT department budget constantly jumping down our throats about how "IF ONLY IT HAD DELIVERED X BY X we could have had 100,000,000,000x /s sales this month"

I made a dynamic report dashboard in my first react project to analyze market spending and prove that even if you wanted to move around metrics to be comically generous the marketing was doing basically nothing to drive sales. MGMT got rid of them and literally nothing changed except everyone had better budgets.

I'm going to repost my original comment (thread here: https://news.ycombinator.com/item?id=21468505) about this which is still applicable:

> Reliably someone comes along every few months to question [performance marketing]. I always come back to analyses of incrementality as the real proof.

> Take an audience of X people. Divide them in two. Show ads to your test group, don't show to control. Watch your business grow and gauge the lift between the two audiences. The companies that know how to advertise at scale do this constantly and can gauge the real effect of their ad dollars. Facebook, Google and others make these tests possible in their platforms, while other software suites such as Impact Altitude and VisualIQ allow you to do this kind of analysis and testing as well.

> In the end, most of it proves out to be incremental. There are notable exceptions of course, but when are there not?

Didn't drill into the details here, but I think there's one area where digital advertising is indisputably useful: the cold start. So many start ups have benefitted hugely from buying Facebook/Twitter/Google ads when they hit that point where all their friends, investors and employees are already using the app, and they've maybe to got a handful of organic users, but how do they get from a few dozen to a few thousand users?

This is really the sweet spot for digital advertising. The cost is low, both because you don't need that many new users to make a huge difference, and because the volume is so low that the targeting works really, really well, and conversion rates are really high. For a more established business, sure, you're getting people that would have bought anyway. But in this case, you're getting people that would have bought if they knew you existed, which they don't.

I think performance advertising is a great idea if you don't rank very high in search results for your category.

"Rental stays in XYZ city" will bring up an airbnb result all day, probably in the top five results. Therefore, paying $3 a click to be placed above your own search result is probably silly.

I happen to have a small side project and advertise it with a very low budget on facebook, google, and bing. It works. I don't rank very high because my SEO skills are poor, but google ads absolutely drive real and interested people to my site.

The example given in the article of the planner ads isn't a good one. The author is trying to make the argument that performance marketing mostly unfairly takes credit for sales that would have happened away, but this particular sale, even if it had happened, would have been unlikely to have gone to the specific brand that was advertising. If they hadn't placed that ad, they wouldn't have made that revenue. Some other planner company would have.

(If you have a near-monopoly on planners, of course, such an ad would indeed have been a waste. The author would have come to you by default once he decided to get a planner. )

It's weird that Rand Fishkin has been in the marketing space for so long, but somehow hasn't encountered a single effective performance marketing team that measures their spend well and complements the efforts of SEO and social media marketers (like Rand).

Hard not to see this as a clumsy sales pitch for his company, especially when it starts with such a disingenuous example/quote (of course AirBnb didn't have to spend on performance marketing during a global pandemic where everybody was suddenly looking for a getaway...)

What if this article is missing the forest for the trees a bit? In my experience performance advertising is almost always paired with awareness advertising. The latter makes you aware of the brand/product/whatever then the former nudges you to act/buy/whatever.

So if you’re buying or even just evaluating performance ads without considering the bigger picture you might come to erroneous conclusions.

Take the Lego Movie example from the article. The $65 million movie is no doubt an awareness play. Could you make the case that you should also increase your performance budget to help capture more of the demand you just generated with the movie? Or should you just hope that people go from the movie theater to buy Lego unprompted? Is it worth it for Lego to advertise to people who walk out of the theater and search for “Lego Batman set” or whatever? I think so, even though evaluating such branded search campaigns individually might make them seem inefficient.

It seems very easy to dismiss the performance advertising as a scam when you evaluate it in a vacuum. As noted in the article it’s important (and very difficult) to understand the incremental outcome of any channel or campaign. That incrementality includes awareness campaigns.

After more than a decade in advertising and marketing I am now more than ever unwilling to accept simple or definitive answers to highly complicated questions. At best I hope that we can unwind some of the overall complexity so we can have a chance to trust some of those definitive answers.

I forgot to do something and basically ended my Google Advertising spend. I haven't missed it in the least. I don't know if I would have done better without it, but an increasing number of people still find my website and phone number even though they are no longer clicking on the ad that Google provided when they searched my exact company name, costing me the majority of my ad spend.

AirBnB has such good brand awareness that it’s not surprising they don’t need to advertise very much anymore”. On the other hand, advertising might be a pretty good investment for their competitors, like VRBO.

Hi, I started a company, Promoted.ai (YC W21) that solves this. My background is from Facebook ads eng and Pinterest ads eng. Large advertisers are already aware of this, and so is Facebook and Google. At Facebook, I led a research team to help big accounts to drive more incremental conversions. At Facebook at least, there is a lot of energy to try and generate the most true performance lift and measure it as correctly as possible.

However, every advertising platform will take your money if you give it to them, so buyer beware.

True incrementally is a challenging measurement issue and therefore even more challenging to predict for delivery via ML. It's real, though, just hard.

Buyer beware. When buying ads, think the stock market. If you hear stories about 20x ROI from random people, do what you when you hear 20x ROI on stock picks: nothing.

In performance marketing(=KPIS are a)sales volume and b) Cost per action - CPA) it's very simple - you can't scam how much you've spent on ads. Also you can't scam how much you charged the traffic you've bought. Both figures are reported by your finance department with pretty much 100% precision.

Yes, you do have a challanging problem of attribution. But the spend and revenue figures are what matters at the end of the day. And neither of them has any area for scamming (let's ignore edge cases).

Disclosure: only skimmed through the article and my arguments above are just directed towards the headline. However credible and opinion leader the author - Rand Fishkin - is, the article itself at the first glance did not inspired me personally as a worthy my attentive reading time.

When advertising is a monopoly or quasi monopoly, it transforms basically into a tax on all transactions.

I think this is overall a waste of resources and I'd like to see a more virtuous system, but I fail to imagine one.

It’s all well and good to claim that PPC offers little to no incremental benefit to brands like Airbnb and Williams Sonoma. But the gaping hole in this argument has to do with small brands. If you wanted to found a new cast iron skillet company you’d be damned pleased to be able to promote your wares directly to high intent purchasers who — so far — are only aware of the Williams Sonoma option. I’m quite certain that incremental benefits of PPC decrease as you scale and become a household name brand, but MOST brands aren’t.

I’ve been in growth for years and while I dislike PPC and find it boring and obfuscatory (FB and Google will both happily claim credit for the same purchase), there’s no denying that in many cases it works. I’ve been at early stage ecomm co’s where we’d pause the spend to see what happened and — what do ya know? — sales would plummet. Brands relying on it for discovery often benefit substantially.

I’d argue that most of the benefits of PPC accrue to smaller businesses (and the platforms, obv). The error in our collective framing is to hate on advertising because we think of it as coming from big brands. But these days it enables vastly more entrepreneurship and competition than I think many on here realize.

And to everyone cheering on Apple’s recent changes that make FB ad tracking impossible — who do you think suffers as a result? Big brands like LEGO can just redirect funds to a Hollywood film or some other high-visibility activation, because they already have their customers’ attention. It’s the small brands paying the price.

I take issue with the conflation of Performance Advertising with Awareness Advertising.

The whole point of _performance_ advertising is that it's effect is _measurable_. If AirBnB spent $500m+ on performance advertising, they should be able to trace that back to an exact amount of revenue. If you are a brand in this scenario, you can conduct split tests by sampling the conversion rate of users from advertising vs non-advertising. Again, it should be simple to see if the conversion rate for users targeted via advertising has increased or is unchanged.

In the branded search examples, again, as an advertiser, you can see what searches are associated with your leads. While you do have to compete for attention on your own branded search (to compete against competitors taking the slot), you should also be able to recognize unbranded terms which drive conversions for you. Again, assuming this is actually _performance_ marketing, you would be able to look at the cost of these placements and the ROI, and the impact would be measurable.

The rest of the article is largely composed of straw-man arguments that imply the results are not measurable, when in fact they are (if done right).

disclaimer: I'm the CTO of a performance marketing platform. The vast majority of conversions on our platform happen same-session. There's a very easy way to measure this effect -> pause your campaigns and immediately see conversions fall.

The article does have a point about attributed sales vs incremental sales - as a CFO (whether I qualify as the "hard-nosed" type in the article, I don't know), I'm bugged every time a marketing guy starts talking about how this campaign has been/will be "ROI-positive", and have had a few heated discussions on why this is mostly not demonstrable until you're willing to pull the plug (which you can do with more or less intelligence in order to minimize your risks).

On the other hand, while I do indeed believe that the "ROI" from Performance Advertising is something between just false and deliberately misleading, the bigger picture that I'm interested in is marketshare. Because when looking at market share, it's not a question of incrementality anymore, but whether you're growing slower/faster than your competitors, and your cost of doing that, and at what point you're OK to 'buy' marketshare, in the sense of losing money in the pursuit of growth, and how much. And then, OK, let's talk about ROI on that basis - most of the time, achieving this will indeed require tools from the Performance Adversiting toolbox, which allow you to conveniently track the amount of marketshare (i.e. sales) you bought.

This article should be taken with a huge grain of salt, as it's coming from a company that sells a service that is essentially is a competitor to the performance media channels it lambasts.

Any article that says "Don't buy X buy Y" loses a lot of credibility when it's written buy a guy who sells Y.

Anecdotally, I've seen in the past with both Google and Apple Search Ads that a large portion of the spend is paying for taps that you would have gotten organically anyway.

When you start using these tools, they almost seem to replace what would have been an organic acquisition with a paid acquisition.

As the article notes, why isn't this obvious for most large advertisers on these platforms?

Likely due to:

- Their spend is so large that it hides the fact that they would have gotten a large % of it for free anyway

- Large advertisers rarely stop all advertising for a period of time to notice the % of truely organic traffic (that isn't just opportunistically attributed to the platforms)

- They have been doing it for so long that big spends on these platforms are considered mandatory and not even questioned

I've run ads for well over a decade: above the line and digital. All media works, to some extent and the main issue is the way marketing teams on the advertiser measure them. I've also run campaigns you'll all have seen or heard of.

So the idea it is a scam is stupid.

Ha. It's much worse than this.

Some years ago, I treated myself to some nice Sophia Webster shoes.

I'm a tall man, my feet don't fit in any of her shoes, but a colleague had pointed out that er, they're for looking at, and if they're on my feet it's actually harder for me to look at them, so, why not buy them and put them on a display shelf like I would a nice sculpture. That made good sense. I bought an arbitrary size that was available - it's way easier to get the shoes you want if they don't need to fit.

Now of course as a tall man, even though it's not unknown for me to buy heels (in a size I can actually wear) it's rare, and I don't buy skirts, have no use for a bra and so on. But still, advertising isn't very smart, so it's not a huge surprise that after buying those shoes I got considerably more adverts for stereotypically female clothing and accessories even on platforms that explicitly know my gender and previous purchasing habits.

What did surprise me is how many adverts I got for the exact pair of shoes I had just bought. Not similar shoes from other designers. Not other shoes Sophia Webster has designed - the exact identical pair of shoes, often from the exact vendor who just sold them to me.

And my theory as to why is pretty simple. Advertising doesn't understand time's arrow. If you only ask whether these adverts are associated with my buying the shoes that's true. They showed adverts to me, I bought the shoes. The only problem is that causality is reversed, I bought the shoes and then they showed me the adverts.

This was confirmed to me by adverts the grocery delivery company I sometimes use ran at that time. The adverts specified the exact goods I buy from them, perishable goods which I had already booked a delivery of, saying hey, why not buy these from us. Well, I already did, duh? But so long as nobody remembers to analyse whether my actual purchase decision (not the delivery) was before or after the advert these adverts probably look like they have amazingly high conversion rates and you couldn't have targeted a traditional advert so narrowly, it must be worth it...

One of the strengths that Amazon has in this space, is that they are both advertiser and seller, so they can A/B test their ads in a way Google cannot, because they have perfect transaction data tracking.

If you enjoyed this article you may enjoy the series of articles I wrote on AdTech in 2019:

https://lockwood.dev/advertising/2019/06/07/adtech-sucks.htm...

https://lockwood.dev/advertising/2019/06/14/adtech-sucks-thi...

For those unable to view the article because the site is "analysing" / fingerprinting your browser to determine if you are a human, and not a bot - https://web.archive.org/web/20211014062026/https://sparktoro...

My take on this is that its sort of like the prisoner's dilemma in that if everybody said "We'll not spend anything on ads[1], let the organic results speak for themselves" then you might get one result, but anyone who has been de-indexed from Google knows that having no results on the front page means a huge drop in traffic for you. And if your business depends on web conversions well that is a pretty measurable loss of revenue.

[1] or black hat SEO for that matter.

The answer is less attribution measurement, more incrementality measurement. Incrementality solves the famous "handing out coupons outside the pizzeria door" problem: https://www.adweek.com/programmatic/lower-ad-fraud-will-be-a...

All I can say is "No shit, Sherlock".

I'm constantly bombarded by ads for stuff I've already decided to buy (or more usually that I already bought last week). This stuff has zero value. The point of advertising has always (until now) been about taking your product to people who didn't know it was there. Targeted advertising would be a funny joke if so many people didn't take it seriously.

see also "Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment" (Tom Blake, Chris Nosko & Steven Tadelis) in which EBay did the same experiment (they stopped advertising), and also "The Unfavorable Economics of Measuring the Returns on Advertising (Lewis and Rao) which talks about how hard it is to measure impact.

SEO has ruined search. It's so hard to find a website that isn't trying to sell you something. I tried searching for an explanation of an aspect of astigmatism correction and I couldn't get anything but thinly veiled ads for optometrists and glasses companies, and no in depth explanation of the detail I wanted explained.

It's well known that the self reported performance numbers from Google Ads and Facebook Ads are inflated, however those ad channels still do drive real value. I run an ecommerce marketing attribution company (ThoughtMetric) so I have first hand knowledge and data about this subject.

The most common source of inflation in Google/FB self reported performance numbers is multiple ad channels taking credit for the same order. If a customer clicks a Google ad then clicks a Facebook Ad then makes a purchase, each ad channel will claim credit for that purchase. In reality each ad platform only has claim to ~50% of the purchase (depending on what attribution algorithm you want to use).

In terms of knowing that real value is produced from these ad channels, I see it every day in many of our customers data. Clients will increase or decrease ad spend and there will be a correlated increase/decrease in sales.

tldr; Google/Facebook ads over report their numbers but do ultimately drive sales according to the data I have first hand access to.

I learned of a bro through a targeted advertisement and I am looking at it for my next rental. I don’t believe AirBnB’s conclusion can be interpreted as “advertisement is irrelevant.”

Perhaps for these ad campaigns the product is the measurement, not the increased conversion rate. The sellers have to adjust price accordingly, by a few orders of magnitude.

What if adtech is not really about selling ads? What if search business is not really about providing search? Perhaps the data collected from these ads/trackers everywhere is the raw material of more valuable predicition and manipulation products sold and classified on the income statement as advertising. The content producers that carry these so called ads are also now servant to the revenue provider, the so called adtech company, and not their readers.

How is it a scam? They sell dashboards that s the business.

That's why you need to use uplift modelling.

That's why we need uplift modelling.

This is pure and utter bullshit.

Performance marketing = Distribution. Without performance marketing your competitors take the sale instead.

So they compromise everyone's privacy for nothing?

Advertising as we know it emerged as a melding of government propaganda research and behavioral economics during world wars, so of course it’s often scammy.

It sounds “deep state” but it’s actually plainly documented in government files and written about by reliable sources.

Remember we’re still emerging from an era of whispering the same old story of morality and obligation to each other.

I am not at all interested in helping someone build a fertilizer empire or pillow brand. Politically my hands are tied to doing so if I want a life.

Poor people effectively live a life of quota and state sanctioned limits on their access to material support by cutting social programs with public support.

Advertising America as anything but a sanctimonious police state is a scam.

No, advertising is not a scam. It's only a scam if you pay for the visitors that would have come anyway (buying your own keywords, and also retargeting/remarketing in most situation).