Facebook Overestimated Key Video Metric for Two Years
wsj.com
wsj.com
They didn't "overestimate". They plainly gave false numbers to drive ad sales.
Coldtea's law: Never attribute to incompetence what can be explained by profit.
Things that appear to be generating a ton of money don't get looked at that closely, because everything's going great. Things creating losses get looked at much faster.
So it's not pure deceptiveness. Probably nobody made the call "let's defraud everyone". However, a bug that undercounted the viewing time would have been uncovered much sooner.
-Upton Sinclair
Of course, the government has proven themselves time and time again unwilling to punish management so management might as well figure out how the illicit gains are being made so they can replicate such processes when the big producing trader or salesperson is invariably poached by a competing shop.
Has something to do with Gillette.
Except if you think that incompetent Facebook-level engineers are more possible than greedy Facebook-level execs.
I think it's very clear that "stupid Facebook engineers couldn't manage to see such a basic metric in two years" fails the Occam razor test much more than "Facebook as a business likes ad money".
This is neither a bug nor a "mistake". It's straight up fraud. Fraud because Facebook uses these false #s to prove to their advertizing clients that they are getting a HUGE ROI, when they really aren't. Also the investors and analysts bump up their ratings and the stock goes higher and higher on false ad view #s.
By counting a view as a legitmate view after the video plays only for 3 seconds, their "algorithm" counted billions of views even when the user has not even seen it, because Auto-play is enabled by default and you have to opt-out / disable it and no-one does it. It takes a person roughly 4 seconds to scroll off their feed as they quickly "scan" their friends posts.
http://digiday.com/platforms/advertisingweek2015-facebook-fi...
Facebook spend an awful lot of resources tracking all sorts of user behaviour inside it's "walled garden".
I got really sick of it after a while and decided to stop working for them (even as a contractor)
If you pay solely for video views on Facebook or Youtube without any tracking, you're not being a smart advertiser and it doesn't really matter if view counts are more or less correct - you're blindly throwing your money away.
If you do however track your ROI correctly it doesn't really matter how many true view counts you get.
This is where as an advertiser I would be very interested in the distribution of view lengths, to better understand the user's engagement and likelihood that the video made any impact on their behaviours at all.
Although being totally honest, you can't rule out the impact of even a passive 3 second peripheral glimpse of brand video content. Most traditional advertising - TV and outdoor, for example - is not actively viewed, but rather passively 'seen'.
Ultimately it comes back to being pragmatic in the way you assess advertising effectiveness - the promise of digital is perfect ROI analysis and measurement, but the reality is far trickier.
The article here is about how the average view length is only calculated from exactly those impressions which lasted more than 3 seconds, where the outrage is that it should have been computed for all impressions. In other words, that the threshold was too high.
On my shareholder hand, this seems slightly like fraud.
On my advertiser foot, this seems slightly like fraud.
They should have a histogram of durations.
It sounds like this was exactly what was happening.
From the article: "Video views of under three seconds were not factored in, thereby inflating the average. Video views of under three seconds were not factored in, thereby inflating the average. Facebook’s new metric, “Average Watch Time”, will reflect video views of any duration."
What really should be done is getting rid of auto playing videos at the HTML spec level. While there are legitimate uses for autoplaying videos (YouTube for example), it has too much potential for abuse by ad networks.
If your ad is > 50k then you're paying a fee per display, even if you're "pay per click"
The alternative is going back to large, low quality and bandwidth-heavy gifs, which isn't the better option in my opinion.
Most of Facebook's video traffic is in their mobile app, getting rid of it in the HTML spec level wouldn't make a difference for FB.
When intent is captured (e.g. specific "autoplay" flag) my browser client can interpret the flags according to my choosing. Capturing intent > not capturing intent.
<video autoplay>...</video>Or, let autoplay be a an explicit flag in the spec that is advisory as to publisher's intent, and establish a norm of user agents providing at least "big switch" configurability (Autoplay video on/off) and, ideally, optional per-site override of default on/off configuration for autoplay.
If the metrics and the view counts match, I don't see a problem.
If they calculate the average for > 3 seconds, but count EVERY view, I agree--fraudish.
Making decisions based on "average" view time is still silly.
If 0.1% watch all 30 seconds, 0.9% watch to 28 seconds and quit at the credits, and the rest all watch 0.2 seconds as they scroll through, that tells you something that "Average watch time = 0.5 seconds" doesn't tell you. Especially if last week, no one watched the whole thing, but a bunch of people watched for 2 seconds and then quit, and your average was 0.9 seconds.
It's not uncommon to require at least 50% of pixels in view for 1 second before you have an impression for static images [0, 1]. AOL defines an impression requiring 2 seconds for images [1]. Facebook likely did some analyses to find that 3 seconds was a good cutoff for their site.
There are more sophisticated ways to estimate dwell time but they seem uncommon in practice; perhaps due to their difficulty communicating to advertisers what the impression metric actually means.
For sites with many bots or inbound marketing you often find users bounce quickly which drives some of this timing. I'm a bit surprised it needs to be that high for Facebook without many bots or users bouncing quickly. Perhaps this is for mobile users scrolling quickly.
[0] http://advertising.aol.com/specs/terms/aol-viewability-terms [1] http://mediaratingcouncil.org/063014%20Viewable%20Ad%20Impre...
Therefore both the true and reported values of X are completely irrelevant because you are incapable of using them.
Please explain how the above reasoning is incorrect.
That's when you rely on metrics like bounce/view period to decide if your advertising worked.
As far as I know Facebook does the same thing. I haven't read the Facebook advertising documentation in a long time but know that they outline many of their metrics in detail. People haven't been getting overcharged and it hasn't effected billing. From the article: "... it did not impact billing, and we have notified our partners both through our product dashboards and via sales and publisher outreach. We also renamed the metric to make it clearer what we measure. This metric is one of many our partners use to assess their video campaigns."
What are the implications of this? "Due to the miscalculated data, marketers may have misjudged the performance of video advertising they have purchased from Facebook over the past two years."
The other part about making it difficult to compare against competitors is true regardless of having the same metric due to site and user differences. Maybe facebook users spend more on average than twitter users after clicking on your ad for an ecommerce site? A/B tests towards the goal you're interested in is much more fruitful from a business perspective then say comparing click thru rates.
I've spent 4 days investigating this, so it's worthwhile to share the data.
For every 100 clicks I purchase, at a strong CPC ($0.10 - $2.00 CPC), I have found: - GA tells me i got between 40-60 sessions - piwik tells me ~70 - server logs show that page was requested ~400x (not indexed, a FB-only lander). I have IP and UA recorded, and both are rotated. My guess is that FB preloads the HTML LP when the impressions shows from mobile, so the browser is primed to quickly load assets (w/o sig resources to d/l images/etc -- just request the page that initially loads all that)
My point is attribution for conversions and link clicks is hard.
Any marketer worth their salt avoids the thought of "I paid for 10 clicks but GA only shows 6".
A better marketer thinks, "I paid $660 to acquire 2 customers. Is that acceptable?"
Source: someone who has had to deal with clients who insist that all our various js-based tracking methods should 100% match backend data and each other.
You're right, but that doesn't have a bearing on whether the advertising you bought was fraudulently presented though. Yes you should be wise, but we should still imprison people for fraudulent acquisition of other people's money, even if you still thought you ultimately got a good deal.
Not only that but Facebook actively brags about video impressions in their conference calls. This is fraud. I think this is something that should be investigated by the SEC to be perfectly honest. Doesn't mean they should be charged, just investigated; because if they are misreporting this, what other metrics are they misreporting to shareholders?
You also can't really directly compare these simple metrics across sites as they don't get at customer lifetime value. With these simple metrics you run into problems where the average video view may be higher on site A, but the average spend for users is higher on site B, and simply following the metric will lead you to an ineffective campaign. Whose responsibility is it to effectively manage an advertising campaign?
While I agree that more oversight and transparency is welcome here, I think jumping out and calling it fraud is exactly the kind of reaction that makes me feel this is overblown.
The dimensions to break out reporting columns for various view duration percentages has always existed. Any advertiser worth their salt is looking at the breakdown for stats on meaningful views because out outside of cases where your message is delivered in a couple seconds you typically only care about longer views and would just look at those numbers.
So the question is how deep this polluted other metrics.
The lie is about the effectiveness of video on Facebook.
I directed a fair amount of additional ad spend to Facebook precisely because Facebook users on average seemed to be giving a lot of attention to videos on the platform, even if they were promoted. It actually seemed like there was finally something that could work to replace the kind of lift we would see from TV advertising 5-10 years ago.
This calls that into question (which may be putting it mildly). It also calls into question all of the other metrics Facebook provides advertisers.
It is not about the specific number of seconds. It is about the implied engagement and focus of the audience. It is about a reported difference in the behavior of Facebook users vs other online properties.
That's kind of why Facebook/Google was ever supposed to be valuable: they empower the analysis. How do you do good analysis with bad inputs?
In the 1300s, feedback to the advertiser took years.
In the 1900s, months.
In the 1960s, you could now measure the effect of blanketing a MMR with ads, and see how sales there compared to nationally.
This century, you have still-imperfect, yet much better than before, attribution.
Good analysis in the 1930s was different than now. Good marketing was the same, but with cruder tools.
Having wiggle room on numbers isn't a bad thing. Lying about the amount of wiggle room on said numbers is.
But assuming the FB/Google reported numbers are accurate, would you pay for them and just convince yourself they are actually increasing your sales?
You should be able to at least plot sales numbers against ad spending per week over a year.
So, we make decisions based on imperfect information. When it was TV or Radio ads, there were 3rd parties that measured the market and reach of the station. With Facebook, Youtube, etc we have to take their word for it.
Also, our marketing doesn't exist in a vacuum. Weather, competitors, a traffic jam down the street, variations in employee performance, etc all impact the performance as well.
It's not like we can just A/B test Facebook video ads vs Pandora Ads and pick the winner.
So, we shift our budget based on the available information.
And the average video view length is part of the information that we used to make decisions.
I know multi-touch is hard, but doesn't that mean you should be less confident in your statements because the data is fuzzy?
Most small advertisers wouldn't be able to do this, and attribution is hairy as hell, but the options for measuring this are there.
I don't disagree, but FWIW the ads still had better ROI than Facebook's competitors, so the oversight (while embarrassing) may not have actually changed the willingness of ad buyers to pay for ads.
I'm not saying they are doing that, I am saying be especially careful with anyone in advertising who is misleading and who doesn't try to protect their reputation by being exact in what they say.
If they buy something, who cares?
My understanding is that they only counted it as an impression if it was a view for at least 3 seconds. If anything, including non-impression views is lying about the quality of impressions their ads received, because they're not impressions.
I don't know if that's how video ad impression data is normally reported, but it sure sounds more intuitive to me.
Or indeed if all advertisers are using the same misleading metric.
Soooooooo... fraud? They say it didn't impact billings/revenue, but I have to imagine they will now be in a position to give discounts, if not refunds.
I expect to see a class action lawsuit because of this. If you are a big advertiser and you can somehow show damages because of this error there may be a case.
There are some adtech companies recently who focus purely on reporting the metrics of your ad purchases. Kind of like acting as a third party. I forgot exactly what they are called.
I can see exactly how things like this could happen in large corps. You'd be surprised.
When it comes to Facebook I guess there are slightly different rules:
A good rule of thumb for Facebook might be: expect malice.
We have seen it with the way the made everything public.
We have seen how they decided to sell out its own user a year after proclaiming their love and how this wouldn't happen.
Facebook wants to have the cake (the hype surrounding its success to be a self-perpetuating thing) and simultaneously eat it too (always expecting the benefit of doubt around its integrity around its self-reported success metrics).
While I welcome people with means doing philanthropy, I can't help but wonder if the recent announcement of philanthropy from Mark Z is an attempt to generate good press. If Mark Z is so keen on doing charity, perhaps he should divert all that money to the top 10 privacy watchdogs in the world. In today's world, that is as much a just cause as any other, particularly because no one really wants to talk about it.
Note: I am not saying the other tech giants don't indulge in these practices. Two wrongs can't make a right.
I'd imagine that view time distributions are way more important to new media companies that rely heavily on video media like Buzzfeed or anyone putting out content on Youtube.
Most video ads I see are more informational than a call to action. Who really wants to click on an ad while in facebook on a phone?
> "Who really wants to click on an ad while in facebook on a phone?"
Advertising for apps (read: games) is a massive, massive market.
99% of the time I click on an ad, it's on accident.
For sure Facebook should attempt to name their metrics as descriptively as possible, but also advertisers should make sure they understand how different conversions are measured, what's included and what's not. Another example would be "Clicks" metric, which included all engagement (i.e. likes, shares etc), instead of just "Link clicks".
> what happens is that views under 3 seconds aren't counted towards the average. So if you have 999 999 views of <1 second each and 1 view of 3 seconds, the reported average is 3 seconds / 1 view = average view time of 3 seconds, while the true average is something like 500 000 seconds / 1 000 000 views.
This is massive fraud, especially if you're shouting from the rooftops "Look at how much more attention people are paying to your videos in our platform!".
Aufgezeichnet wurde zwar die Gesamtsehdauer aller User, geteilt wurde dieser Wert allerdings nur durch diejenigen User, die das Video länger als drei Sekunden ansahen
So the average was not calculated correctly: They accumulated the duration of all video views (including of those shorter than 3 seconds), but divided it by the number of 'legitmate' views - i.e. only those longer than 3 seconds. So you get a pretty big offset if you have many views under 3 seconds (which they probably do, thanks to autoplay).
[1] https://www.heise.de/newsticker/meldung/Facebook-Unklare-Vid...
I hope/assume this is part of a suite of metrics that they provide to advertisers, so you can understand it in the context of other things like video completion rate or counts of views that reach X% through the video.
Less than 3 seconds is not a view any more than seeing the picture of the video is a view. It doesn't count I'd say?
Or do advertisers have to still pay for a 2 second view?
Likewise, smart advertisers look at lift in conversion metrics when possible, in which case this stat is irrelevant.
That said, FB has not exactly helped things by making metric definitions a little obfuscated in general.
Personally, I see a bigger concern is them giving 100% view through conversion credit with a 1 day window by default as part of any website conversion action tracking. There are very few cases (like some retargeting situations) where you'd ever want to give full weighting for a VT, and while there is likely value in VT's, I probably wouldn't give 100% credit to them by default given the rate at which people scroll on mobile. But advertisers like to see big numbers, agencies like to show big numbers, and so you have platforms like FB aggressively try to push metrics like this and some of their rather loose definitions of "engagement" without great explanation of the nuances or pros and cons. These are largely left up to the advertiser to determine since, to be fair, they are very subjective.
Savvy buyers know this and configure their reporting and tracking settings accordingly because FB and PMDs give you those options. They are sometimes just buried.
Ultimately, IMHO FB and Google's greatest defense towards any of these sorts of claims is better and (more importantly) transparent attribution data and tools. If they can prove their value on the bottom line, other things often don't matter to many advertisers. Attribution is a tough nut to crack, but for advertisers spending large sums, it is critical to be successful in these channels.
"For purposes of Video inventory, an impression is considered “In-View” to a user when at least fifty percent (50%) of the pixels in the ad are in the viewable browser window for a minimum of two (2) consecutive seconds."
Google here https://support.google.com/dfp_premium/answer/4574077?hl=en says "For in-stream video ads, 50% of the ad’s pixels must be visible in the browser window for a continuous 2 seconds."
If they are only counting a view as over 3 seconds. And then only averaging the those views what is the problem? They tell me 1000 people viewed it (over 3 seconds), and then say the average view time was 10 seconds for those 1000 users. How is this an issue? Why do I care that 9,000 people scrolled past it and weren't counted in the viewer numbers? Now if they were reporting this as 10,000 viewers and saying 10 second average, I see the problem. Is that what was happening? Or what am I missing?
I.e. suppose your ad was displayed 5 times for 0.1 seconds and 5 times for 5 seconds. They would report this as "10 views, with average viewing time of 5 seconds".
http://www.adweek.com/news/technology/facebook-gave-ad-agenc...
> UPDATE: Facebook authored a blog post today to futher explain the metric-based miscue. David Fischer, vp of advertising and global operations, wrote: "About a month ago, we found an error in the way we calculate one of the video metrics on our dashboard – average duration of video viewed. The metric should have reflected the total time spent watching a video divided by the total number of people who played the video. But it didn't – it reflected the total time spent watching a video divided by only the number of 'views' of a video (that is, when the video was watched for three or more seconds). And so the miscalculation overstated this metric. While this is only one of the many metrics marketers look at, we take any mistake seriously."
[1]: https://techcrunch.com/2012/07/30/startup-claims-80-of-its-f...
They didnt include views of less than 3 seconds in their "average view time" metric, which is very misleading. Their metric was "average view time of video's in our view count".
Which is not the same thing and the latter vastly inflates the effectiveness of facebook video