Google could have killed Facebook with the flick of a switch | Hacker News Reader

Google could have killed Facebook with the flick of a switch

shaneosullivan.wordpress.com

193 pointsmsh139 comments

Today it's called Business Manager and it remains a user hostile pile of JavaScript with dubious design and riddled with bugs.

The only reason anybody puts up with it is because there's no other way to access powerful targeting tools, and we are forced to depend on Facebook.

More to the point of the original story, no, Google couldn't have killed Facebook by flipping that switch. The Power Editor users would have done whatever it takes to access the interface, installed an insecure older browser, ran commands in the console, build a relay computer in their back yard, and so on. They will do whatever it takes because they are not choosing that platform for its features, they are professionals doing a job, and that job is reaching the user base of Facebook with their ads. If they can't do that they will lose their jobs and clients.

I find business manager has a more sensible design than AdWords, having used both a lot. TikTok is funny because they basically copied the FB ads dashboard pixel by pixel.

I agree but only the actual /adsmanager/manage section and the subsequent tools (events/audiences). Basically power editor 2.0.

But it seems less powerful than the old one for some reason might just be in my head though. It has a pretty low ad limit per ad group I think 100. Though some of the 'combo' ads where you can put 5 headlines, 5 copy etc help up that number.

BUT I do not like the business manager which houses all the pages, users, audience assets. I find the 'line of business' really confusing and hard to see how they want it organized. Like what's the model supposed to be? Agency line of business -> client subs. But then you need a specific line of business for certain offline attribution etc so having it agency parent doesn't make sense to me?!

I think it feels less powerful because it is a buggy mess and takes forever to use. I loath setting up Facebook ads now because I know it is going to take me forever to do it properly.

I pine for the days of PowerEditor. It had a confusing interface, but like Google Ads, once you had learned it you could get what you wanted to get done quickly.

I don’t mind the interface of Business Manager or the Ads Manager platform at all, but I find the ads platform to be riddled with weird bugs. It has perhaps improved slightly in the past year or so, but only slightly. They also constantly change how it works, so you can’t just copy the campaign or ad set of an ad and build out new ads from a year ago, it will almost always have to be built out from scratch because something changed, but they don’t tell you that before you copy the campaign. Why even let me copy it if it isn’t going to work?

Oh totally agree with this

This story is a fun read, but the "Chrome could ruin us at any moment" angle is more story telling hyperbole than reality.

At that time on Chrome we were just figuring out a process to remove features and were pretty conservative. We debated a lot about the right way to remove things without causing damage to the web ecosystem. In an early attempt we had failed trying to remove a legacy feature and needed to revert it. It had broken a big company who had then contacted Google directly about the business impact and it taught us a lesson about being careful.

IIRC WebSQL had very high usage and we didn't seriously considering removing it. It's also pretty self contained so the cost on the engine isn't that high.

Chrome still supports WebSQL today because the usage remains quite high at 1%: https://chromestatus.com/metrics/feature/popularity#V8SQLTra...

.. and even if Chrome had removed WebSQL back then, couldn't that have just kept using whatever version of Chrom e they were already using?

My understanding is that clients would have to use an outdated version of Chrome, which is probably an untenable ask.

I suspect a lot of FB customers are using a browser in a business setting where IT manages the Chrome updates. FB could have used a polyfill though, or used Safari, or shipped a desktop wapper around an older Chrome temporarily. Electron was brand new at the time so it probably wasn't a great option, but FB certainly had a lot of emergency options.

[author] There were some contingency plans, but we had no time to get them ready in advance. If Google had actually pulled the plug, we wouldn't been down for everyone for many weeks, and given that the vast majority of our large customers are non-tech-savvy (low paid employees at ad agencies), a large percentage would have remained with a broken experience forever. They'd have just given up and moved their money to Google.

Sounds like the South African government approach, except they released their own browser with the last Flash version: https://www.zdnet.com/article/south-african-government-relea...

I don‘t know if this is better or worse than the whole South Korea ActiveX thing, which became so ridiculously drawn out it was an election promise in 2017 and finally killed in 2020. https://www.theregister.com/AMP/2020/12/10/south_korea_activ...

However, things have still been shitty even since ActiveX was duped, if you're doing any kind of banking or printing out documents from the government now we're forced to run some dubious exe install files, which will spawn startup processes always running in the background taking up CPU and memory (regardless you're using the banking service or not!).

At least now online payment is much more convenient now (for most sites it even works on Linux, given that you're using a mobile app for authentication...)

You have to have mysterious software running to use a bank website? That sounds terrible, I would just go in person.

I hope AMP is also killed off like ActiveX, Silverlight and Flash.

[author here] Unfortunately no - we had many thousands of customers, most of whom had no control over their machines (low level employees at ad agencies). As a company, Facebook has no control over what browsers their customers use. The product would simply have broken overnight

> It had broken a big company who had then contacted Google directly about the business impact and it taught us a lesson about being careful.

How can normal users accomplish this same task?

No.

Isn't that a slightly different argument?

Title: "Google could have killed Facebook with the flick of a switch"

Reality: that seems to have been true.

Your take: yes, but we wouldn't have done so because we're not in the business of breaking backwards compatibility on a whim.

Although it's good to hear reconfirmation of that fact, it's not technically a rebuttal nor showing that the story is hyperbole.

TLDR: Facebook built the app responsible for their entire ad platform on a chrome-only tech (WebSQL), and when it was deprecated, instead of dropping in a widely-used polyfill, they decided to do a major 3yr rewrite risking 25% of their revenue in the process.

I guess this is the stupid shit that being "too big to fail" permits.

I didn’t know that such a polyfill existed but even so, that was pretty much my reaction. Facebook would be quite capable of developing one, and potentially a route there to something more service-based once the panic is dealt with.

Another option would have been to use a chromium build with WebSQL still in place.

They would have to get all their advertiser customers to use that custom chromium build, which would be possible but would not be good for their ad sales.

Given how long some of us had to hang on to Internet Explorer 6, I'm pretty sure the advertisers would have required their IT staff's to comply and install the "Facebook Browser".

Wouldn't have necessarily had to be nearly as bad as IE6 for security, either. Just lock the special browser to facebook.com only, or even just the ads API, and there's not a lot of room for exploit.

There are a lot of big, messy, "terrible" applications with dumb dependencies that seem to live far longer than they "should". It's mostly in large companies but not really related to these companies. These same applications usually have unique functionalities and user who have become expert in the use of apps, including knowing the "stupid parts".

Writing these apps is hard and it's a completely rational decision to jury-rig deprecated libraries to make such an app work however irrational it might look from the outside.

Oh, I'm totally aware of technical debt, etc. I have worked on both maintaining and rewriting a number of such enterprise apps myself. It is just in this case, if the revenue WAS ever at risk (don't forget we are talking billions here), then you would need to be a bungling idiot to not fix the issue at hand before embarking on a rewrite.

Personally, I think the article is a dramatisation - I would certainly be too embarrassed to write such a thing if it were true.

> I guess this is the stupid shit that being "too big to fail" permits.

Or just how technology actually mostly works out in the wider world.

Very interesting story!

The title (Google being able to kill facebook) seems a bit exagerated though. If this app was so important (25 to 50% of all facebook revenues), and so important to customers (most powerful editor to manage ads), I'm sure if Google had disabled websql, they would have forked some older chrome / electron with websql support and bundle it with the app.

that would have made builds massive, unless you're referring to just the desktop experience...

I'm saying they would have find a way to deliver the app to customers, whether in desktop or web form. And customers would have downloaded the app if needed, as it seems to be the only powerful way to manage ads.

Indeed, people will always find a way, especially if money is involved. And running outdated browsers for legacy support, while being a huge security risk, is not uncommon for a lot of use cases, especially back then.

A hit of 25-50% of revenue would certainly have hurt, but probably not killed Facebook.

That people will find a workaround is absolutely true. I once worked on a system to order mobile phone subscription from a website that only worked on IE6. So we had a website that accepted orders from customers, created jobs that would be sent to one of 10 VMs that ran IE6, where a watir script would steer IE6 to enter the order into the old website.

It's the most horrible hack I ever worked on, but it worked.

Absolute cringe, thanks you for this.

When BBSes started telling me to call back when I could connect faster than 300 baud, I discovered they would let me in at 110 baud.

I used to have to manage mailing list subscriptions based on data in a SQL database. The mailing list software only had a web UI. So I worked out what GET and POST commands and data I needed, and scripted it in bash using curl commands to interact with the list software. Ugly, fragile, but it worked.

If the product was responsible for 25% of the company revenue, why was only half of one engineer assigned to it? And after the vulnerability was discovered, they still only had 5 people assigned to it while it increased in importance to 50% of revenue?

I've seen this sort of thing happen many times. What usually happens is the original team that built it leave the company or move on to other teams, and the one engineer left actually has a different job as well now but is stuck supporting it part time as a favour. It's not that you couldn't assign other engineers to help support it, but they wouldn't have any idea how to do so because there's probably no documentation and only that half an engineer around to show them how it works.

It was because Facebook practiced open allocation for engineers at the time.

Essentially, you had a choice of teams and the manager would need to convince the engineer to work on the project.

At the time, ads was not a high status part of FB engineering, and as such, it was very difficult to get people to work on ads products.

It's funny that people want to work at Facebook or Google but don't want to work on ads.

In other companies, everyone wants to work near the profit center and avoid cost centers.

Because those projects are hard.

It is much easier to make a greenfield project that will get 70% done and cancelled before anyone even asks to fix bugs.

Can't have been that hard if half an engineer was enough to do the job.

The half an engineer was just the last team member around, only doing essential maintenance to keep it working.

So why allocate more engineers to the project when it was doing fine with half a head?

The article states that they needed to rebuild most of it because the entire thing depended on a deprecated Chrome feature that could be removed at any time.

Because the collections agent for that technical debt hadn't called _yet_

To borrow a phrase from the investment industry, the fact that a ticking time bomb has never previously gone off is not always a reliable indication of future performance.

Hard to imagine that anything responsible for 25% of revenue (billions of dollars, at FB scale) would not be seen as "high status" if not by the engineers, then by management.

[author] It wasn't. The plan at the time was to deprecate Power Editor and build a new version of Ads Manager. We undertook both projects simultaneously, with the plan being to keep Power Editor going until AM v2 was ready. Unfortunately that took too long, and since Power Editor was the only product that could ship all the new ads features, by the time it came that Ads Manager V2 was ready, Power Editor had 50% of revenue share and Ads Manager about 20%. So, in effect we rebadged PE as Ads Manager, pulled over some of the new features from AM V2 and called it done.

[author] This is very accurate. As the manager of the team I had to spend a huge amount of time trying to hire internally for it. No one came to FB to work on ads, they wanted to work on stuff that their friends used. I had to develop a smooth song-and-dance to get the right type of people interested. In the end, this served me very well in my career, being well practiced at getting engineers excited to join your team is a valuable skill.

I hope those 13 engineers got paid millions. So much weight and responsibility on their shoulders! They were responsible for nearly $2 billion in 2013 (25% of $7.87bn), and nearly $14bn in 2016 when the work was finished (50% of $27.63bn)

I see this sort of comment occasionally. If those engineers hadn't been assigned to work on the project, somebody else would have. It doesn't look like they did that fantastic a job of it either. Also this was a front-end to managing the ad system, not the ad system itself, or the infrastructure it ran on, or the features of facebook that drew in the audience in the first place, etc, etc.

Yes some engineers genuinely do make contributions that have unique value, but frankly most of us are guns for hire and if we're lucky enough to get on a fun, interesting and valuable project that's great, but frankly one out of three is the best we usually get.

Smacks of maybe survivorship bias. If OP hadn't noticed this chrome dependency, how are we to speculate on the likelihood of someone else noticing it? They even said nobody else wanted to work on it.

Well OP is tooting their own horn

[author] Many did and it was well earned. The team was awesome and had great camaraderie, but the work was horrible. Everything that broke led to a SEV for our team, even though 80% of the time it was not our code (somewhere deep in the API that had insufficient logging and observability). All our customers hated the product since it was so buggy, and our internal customers (sales ppl at FB) hated it for the same reason. They didn't see the 16 hour days, nights and weekend that we were all putting in to fight fires, do a fundamental rewrite from two JS frameworks to an untested new one (React), and add hundreds of new product features to keep up with FB's exploding Ads org. It was three years of hell to be frank, but very rewarding if you're into that kind of thing. It took a special type of person to join that team, knowing how long it would be before any payoff was really visible in the product, all while being vilified inside and outside the company even though you didn't build the thing, you just agreed to jump in and help fix it.

You see this all the time. As companies get larger, they become more and more difficult to understand. This is even more so the case with software companies since they are essentially complexity factories.

> If the product was responsible for 25%

Being a (however necessary) part of the toolchain is importantly different than being responsible for the revenue. Such tools often fall under "it ain't broke, don't mess with it". Until they are broke, of course.

[author] There were a number of critical parts of the chain responsible for FB Ads revenue. It all started at the UI or API (which were 75% and ~25% of FB revenue when I left in 2017). Then you had the DB layer, managed by another team, the AI/Targeting layer and one or two others.

If any had failed fundamentally, FB was in big trouble. However, other than Power Editor and Chrome, the rest all ran in our data centers and were under our control.

[author here] Getting 5 engineers was actually a huge ask. Just a year earlier the entire Ads Interfaces org had 5 engineers in it (I was #6). That was 5 engineers for all API and front end code for a $4B/year revenue company. Facebook has a history of using tiny teams to lift impossible mountains - e.g. FB Photos was originally written by two engineers and overtook Flickr, the current leader, with just that investment.

Facebook's real moat is it's massive user base not it's ad revenue, even losing 25% of it's revenue would not have effected or killed Facebook, Only building better viral social network could have killed it. Ad revenue is just a side effect of this massive moat. Just like Google can survive a temporary year or 2 of 25% drop in revenue if their search dominance continues or increases but not opposite.

That so many smart and talented people choose to work on AdTech or systems that support AdTech is such a shame. That being said, I love these little anecdotes, they are super fun to read.

There are so many aspects of adtech that are ethically neutral. Just as advertising itself - there are even ads created by great creators that are works of modern art. It's only when it's abused adtech becomes evil.

Wow, I bet I'll miss out on so much important culture by not getting visual pollution all over the place unless its regulated out of existence.

No thanks, I'll continue to block it everywhere that I can, and even going further and inspecting elements to get rid of obnoxious paywalls as well! Wow, am I a hypocrite? Maybe develop real, sustainable revenue models that aren't dependent on clickbait and being a lobbyist for corporations in the form of selling eyeballs to the highest bidder. Just yuck. I'll be inclined to buy into $WHATEVER if you just let me use it for a bit, if I find it of my own means. I promise you I'm not going to buy a Swatch because of your fucking banner ad. Seriously, I promise. If I just bought a book I promise you I'm not going to buy it again if I see it appear in my side banner. 100%.

Or maybe I'll just grow less dependent on external services every time I have to consider another subscription... wow! We'll see!

Have you checked the art vs bullshit ratio?

That probably costs extra

It may be "many" but it does not mean that the defining nature of advertising is ethically neutral. I think I just quote Banksy/Tejaratchi:

"People are taking the piss out of you everyday. They butt into your life, take a cheap shot at you and then disappear. They leer at you from tall buildings and make you feel small. They make flippant comments from buses that imply you're not sexy enough and that all the fun is happening somewhere else. They are on TV making your girlfriend feel inadequate. They have access to the most sophisticated technology the world has ever seen and they bully you with it. They are The Advertisers and they are laughing at you.

You, however, are forbidden to touch them. Trademarks, intellectual property rights and copyright law mean advertisers can say what they like wherever they like with total impunity.

Fuck that. Any advert in a public space that gives you no choice whether you see it or not is yours. It's yours to take, re-arrange and re-use. You can do whatever you like with it. Asking for permission is like asking to keep a rock someone just threw at your head.

You owe the companies nothing. Less than nothing, you especially don't owe them any courtesy. They owe you. They have re-arranged the world to put themselves in front of you. They never asked for your permission, don't even start asking for theirs."

Edit: Credits also to Tejaratchi

This is such a great quote. I've checked out a couple books on Banksy but haven't ever run into it before. Thanks so much for sharing!

One time I saw a fascinating article about how scientists were working to grow skin complete with natural looking hair from stem cells for burn victims and such. I thought the article was really neat and I was in a hurry so I emailed a the link to myself.

The next time I logged into my gmail to read it I happened to be at a machine without adblock and I saw that google was serving me ads for garbage treatments for male pattern baldness.

Fuck you Google and your shitty advertising. There's nothing wrong with going bald and I don't need you to try and make me feel insecure about my hairline when I'm trying to read about scientists helping burn victims.

The ad probably promoted a product that could regrow hair for bald people. You just took that B̶a̶n̶k̶s̶y̶ Sean Tejaratchi quote and applied it to them. You chose to be offended by it. Sure baldness isn't wrong. But a lot of people like hair better and therefore a market exists for this type of product. Nothing wrong about that.

Repeated viewing of these kinds of ads is precisely what introduces the insecurity in people that makes them seek out the kinds of products being advertised. It's the same thing with skinny models and anorexia in girls.

Advertising is the cause of the 'problem' that they can conveniently solve. Besides some of these 'cures' for baldness have alarming side effects and there should be regulations around advertisement for these products. https://www.reuters.com/article/us-merck-propecia-suicide-ex...

Why are burn survivors unacceptably ugly to you, but baldness survivors are acceptable looking?

I used to deliver food to a burn unit in a hospital. I'm not going to deign this question with a response.

"Baldness surviviors"

That's a new one for me on the victimhood trivialization scale.

> Fuck you Google and your shitty advertising.

I really don't get this response.

Since very soon after Gmail launched, Google has been very open and public about mining user's emails to serve ads to them. This isn't some kind of dark pattern. There was a lot of fuss, to the point of politicians threatening to hold hearings about it.

And then people sign up for Gmail and complain about the ads.

I can understand complaining about cookies tracking you all over the Internet, and how hard it can be for the average Joe to avoid them. But Gmail and their ilk? It's trivial to use something else.

> I think I just quote Banksy

Banksy is not the original author, Sean Tejaratchi is: https://en.wikipedia.org/wiki/Sean_Tejaratchi#Crap_Hound https://i.imgur.com/D0pTs.jpg

Thanks, I edited my post to include credits to Tejaratchi. Not sure how this should be done properly, as that specific quote is from Banksy, so I just left credits for both.

Please note that all these examples from the quote refer to a certain genre of ads targeted at the general public. Many teach ads I see are completely different. I often click on ads for microcontrollers for example, and I have't yet seen one I would find offensive in any way. They don't make me feel small and unworthy, on the opposite: the make me feel empwered. So it's really about the content of the ad, not advertising itself.

Advertising is crucial for everyone. When you have a product or service, you need to advertise it. Even if you choose not to buy ads, you need to find another way, organic or not, to make your project known. If you don't have a products/service yourself, you work for a company who has, and there are very few lucky enough not to need any advertising (because e.g. they secured long-term goverment contracts already etc.). We should focus on how to make advertising human again rather than destroying it completely.

Note that the quote originates from the 90's, way before ad tech as it is currently known. Advertising is a bit like finance. There exists an important, tiny core that is surrounded by a massive amount of activity whose societal benefit is doubtful at best.

That's not just ads or finance, that's all of capitalism. (If I'm good at programming I have to convince a farmer its in his interest to give me food, even if he doesn't need my software.) And communism has its own problems.

> so many

Please name 5

OK. Let's separate the ad industry from ad technology for now. So as for the latter, have a look at the Revive ad server [0]. It allows you to deliver ads with intricate rules as to when they should be served, on which websites and so on. It allows you to be less dependent on Google/FB duopoly. You can host the ads either on your own websites or contact the owners of other popular websites and arrange an ad deal that doesn't involve user tracking. Yes, this is the opposite of what modern ad networks are doing.

In any case, advertising without user tracking is, in my opinion, one of the most ethically neutral forms of making your product/service known, in contrast to morally dubious methods such as advertising with user tracking, product placement, advertorials, advertising disguised as articles/videos etc.

[0] https://www.revive-adserver.com/features/

There is a difference between the medium (ads) and the targeting and distribution technology (adtech).

A great ad evokes some kind of feeling, can live with you for a long time, and even bring back memories that you cherish. Yes, I agree with you, it can be a form of art in that regard.

Adtech, however, is an intentionally invasive technology that exploits any means necessary and technologically feasible to more narrowly target potential consumers.

You can have ads without adtech. But you can't have (commercially successful) adtech without invasion of privacy.

> But you can't have (commercially successful) adtech without invasion of privacy.

I think we (as a society, not just tech people) specifically need to take care of this point. Removing tracking will make it less effective, so what. We've had advertising without tracking since 19th century.

There are more positives in Adtech than negatives.

Adtech allows people and businesses to find products and services they want and need. Gives consumers choice in discovery and businesses reach and targeting customers in need.

At fundamental level, Adtech is there to help people. That and the many business opportunities are why so many talented people work in Adtech.

Keep telling yourself that if it helps. ;) While at some level ads probably do help consumers (indirectly), my impression so far is that at a fundamental level they are about companies competing for money, in a way which wastes everybody's time. Since your point of view is so vastly different, I'm curious in which ways you feel like they give consumers choice?

"At fundamental level, Adtech is there to help people."

Haha. Adtech is not there to "help people". Back in the day, prior to television and radio, ads were there to provide information about a product, but those days are long since past. Ads now-a-days appeal to emotional and irrational wants/desires, this is common knowledge. Ads attempt to generate needless sales by marketing to irrational and emotional desires.

Adtech on the other hand, is there to drive irrational and unneeded consumerism and consumption just as well, it just happens to have some additional evil sprinkled on top. For example, adtech violates human privacy, adtech marginalizes the open web, and turns it into a creepy surveillance nightmare.

To say that "At fundamental level, Adtech is there to help people." is delusional and unintentionally hilarious. I am going to swing in the dark and say the GP must be an employee of facebook

If only Google had a search engine to help people and businesses find products and services they want and need.

I share many concerns about tracking people, but let's acknowledge that adtech democratizes access to services that most people (worldwide) would not be able to afford. "Free" is the only price that works for the 36% of the world's population that is in poverty.

That is not how Chrome removes features. It's based on usage. WebSQL is still supported today, in fact: https://caniuse.com/sql-storage

Ehehe nice story.

I remember when we were to be shut down by AWS because the bill was 2 months due and the administrative contact was a inbox nobody looked at. We discovered it by chance, doing a nightly maintenance: services started to disappear one after another (including the Redis that powered the login, we weren't able to login into the product for half an hour!). We immediately phoned AWS support and asked, no, begged them to turn them up again. They did and we paid everything the morning after.

Hadn't we been in front of the PCs doing the maintenance and witnessed as it was happening, we might have woken up the day after without an online bank (yes) because the production environment would have ceased to exist...

It raises interesting questions of whether Google (advertising company) should be the guardian of the web's primary browser; strategically it could deprecate features direct competitors depend on (let's assume FB is now also a ad company). The fact that web's primary ad platform also owns the web's primary browser sure is "interesting". The inverse of that is that it might not have incentive to deprecate features - which Google the company / partners depend on or might not be able to deprecate due to legal consequences of doing so (shutting down 25% of FB's revenue might raise eyebrows in the gov).

iirc it was Mozilla who put the nail in the coffin of WebSQL and if you read this post, it actually mentions that Chrome was the last browser to even still support the feature after it had been abandoned by Firefox and Opera already. What does this have to do with Google owning Chrome?

Of course, the question itself is valid but I don't see how it relates to the topic.

> This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google.

Considering the tracking Google does with Chrome, I suspect they recognized the traffic still occurring over WebSQL and left it active for exactly this reason. It's highly unlikely that nobody at Google was able to recognize this. Alternatively, maybe Google didn't know about Facebook's reliance on it, but was aware of other services/corporations that still relied on it for some time, and waited until they had an alternative.

At the time this story takes place Chrome didn't have a way to collect web feature usage metrics broken down by site. It was aggregated across all sites to protect user privacy. That came years later with Rappor: https://www.chromium.org/developers/design-documents/rappor

WebSQL usage was too high at the time to remove though, so while the story is fun to read Chrome never would have actually turned it off.

[author] Hard to say. If Google leadership had heard of this vulnerability, right as they were launching Google+, there would at least have been a serious conversation about it. They spent $Bs to build Google+, changed their whole company structure to support it, and were losing hundreds of people to FB every month. I doubt bending one little rule about removing a deprecated technology would have bothered them too much.

I've heard, and keep hearing, so many stories about frustrations with Facebook's tools for advertisers that the situation described doesn't surprise me at all. Somehow this massive organization with >50 000 employees and billions of users can't seem to produce quality software that allows people to pay them for access to those users.

Small tasks turn into multi-day endeavors just because the tools keep crashing over and over again, losing all state. Managed advertising accounts just vanish from interfaces. Random buttons turn into "try again" roulette.

It's a good thing their ads are (apparently) effective.

Quick note: it's an unfortunate fact of life that some engineers make a career of exaggerating things.

This is not that, though.

I heard this story from people involved at the time, and it's not exaggerated in any way, shape or form.

But isn't it? I can believe people inside Facebook really thought it was a critical problem, but I don't see how it could have become any more than an inconvenience.

- There's a websql polyfill. I don't have experience with it, but it looks like a drop in replacement. This is such a strong counterpoint, the rest almost doesn't matter.

- Features tend to be disabled in Chrome with some advanced warning. Not overnight.

- websql is still enabled in chrome today

- if the api were disabled, ad customers could be directed to flip the feature switch in Chrome. If money is on the line, seems like an easy enough task.

- Safari supported websql until safari 13 (september 2019), long after this issue stopped being relevant. How many people in an organization need to use PE? Probably only a few in marketing. Surely these few people could have just bought a macbook. A few macbooks must be a very small expense compared to facebook advertising.

- Or IT could have arranged a VM with an old version of Chrome.

So this was an application used by extremely non-technical customers (i.e. marketers) spending money on behalf of many, many brands.

Relying on their IT to do this just wouldn't have worked and was definitely not scalable.

For context, if Power Editor disappeared along with its revenue, Facebook would have needed to report this to the market.

The idea that businesses would forego Facebook because some non-techies find the most powerful ad manager to be difficult to run completely mistakes where the focal point is. Brands need FB more than the other way around.

If Chrome were to remove WebSQL, bosses would have told people to 'run this or else', and you'd have found that every process was scalable with enough human labour.

There were multiple workarounds present. Probably they'd have bought a mac, at worst they'd just have used a 3rd party app until FB fixed this, which wouldn't have taken long with FB's resources. Hardly fatal for anyone.

Interestingly, Google later wrote their own relational db in javascript called lovefield, which was motivated primarily by the deprecation of WebSQL: https://github.com/google/lovefield/blob/master/docs/dd/00_i...

This is a good story, but Google can still literally kill any business that relies on the Play Store or Chrome Store with a flick of a switch today (like mine). No human even has to get involved. At any moment, some code running on a Google server can decide to flip a bit and disable access to an app or extension in their stores.

That's not entirely true. They can kill any small and medium business easily. Taking out larger competitors in such a way will likely invoke lots of unwanted antitrust attention. Over the entire world. But they can easily kill any small competitor.

Couldn't they have compiled sqlite to javascript using emscripten as a websql replacement?

sql.js seems to have been started in 2012, and even if it might not have been production ready at the time, I'd expect facebook to be able to pull off such a project by themselves.

extreme urgency.. facebook would shut down.. new ads features blocked.. urgent creation of a team to replace product.. 3 years later it was ready (!)

Best example of leading by FUD if I ever heard one.

I'd call it "managing upwards". :-P

To kill FB I don't understand why Google doesn't have a full time team of people working to extend/fork Signal to add a doomscrolling "blog posts" feature with required group/circles/whatever-bullshit.

This isn't about Google capturing the FB space. Simply denying it to FB.

Honestly:

- no ads. (Seriously Google, no ads. But you know they can't which is why this is a pipe dream....)

- could sort your own timeline by timestamp, number of comments, whatever

- can't be shutdown due to content.

Bonus points for: - Events and adding Events into your local phone calendar.(Please don't give me another calendar to cross reference.)

Google twist the knife effort: - Free public facing web hosting for those who actually want their "Wall" public. IE companies, businesses, "personalities" etc.

They did this for Android to compete with Apple. They can do it with Signal to compete with FB.

If nothing else, could they "buy" the features from the Signal team in a similar fashion to Blender? This'd be my preference so that Signal maintains independance at least somewhat.(Money always talks I know.)

In the early days of FB, they relied on Google Docs and Gmail.

More importantly, they used pre-built social networks by letting you import your GMail contacts.

Anybody have insight into why WebSQL was chosen in the first place? I don't think I've even heard of it prior to this article.

[author] Power Editor was originally written by a brilliant engineer named Vladimir Kolesnikov (yes, all by one engineer), and he was solving a very specific problem of letting customers scale their ad spend. So he chose a technology that let them download all of their ad account locally to disk, do all their work, then upload it again.

WebSQL was flexible, fast and easy to work with. It's a real shame it was deprecated to be honest

Could have killed you with the flick of a switch - but you took three years and a team of 5 people to perform the rewrite?

[author] Unfortunately yes. It was not the only thing happening at the time. FB had just gone IPO and this put a huge amount of pressure to grow revenue. When I joined the Ads org in 2012, it was about 100 people I think, with just a few products. When I left in 2017 it was well over 1000, and my job had changed to managing relationships with 55 other teams, all of whom depended on my teams product to ship their individual features (think Video ads, Carousel ads, hundreds of others).

So, while doing the rewrite, we had to transition from being a product org, where a few engineers built all features for a bunch of back end teams, to an infra+product team, where we built all sorts of support for all the new teams that seemed to spring up weekly.

If we had done only the technical rewrite, and not enabled all these other teams to ship their products on the UI surface that our customers actually used, FB's revenue would not have gone from $4B when I started to around $30B when I left.

These things are never that simple. I wish....

Today's episode in "Read Your Deprecation Warnings"

Google actually had a successful social network project, Orkut, they just continuously marginalised it for it to be too NIH for US headquarters.

As far as I can tell, Orkut was the original incarnation of a modern social network, but they failed to capitalize on it.

Orkut was a clone of The social network created by the namesake's previous company.

It was launched about the same time as Facebook and after Friendster.

Wasn’t it very popular in Brazil for some time?

Yes it was. Source - old Orkut user but not from Brazil.

I'm guessing India - it's where I grew up and it was the network of choice for teens until we moved to Facebook

Also tried to band everyone (MySpace among them) together with OpenSocial, but too late.

I don't know how they are doing it but Google manage to fail at anything social. By that I mean communication platforms, social networks and instant messengers. They have promising starts but they fuck it up later. Their only successes I'd say are GMail and YouTube.

I'd say for GMail, it is just an email client they started out excellent, thanks in part to its huge storage space and good search features. And they didn't break it enough to make it unusable.

As for YouTube, it is great for hosting video, but its social features are terrible. There are communities tied to popular channels, but it stays in a producer to consumer model, with no interaction between community members. The comment section is made in such a way that dialogue is almost impossible.

Maybe they should hire a few people who understand human beings, because right now, it looks like the company is run by robots.

disclosure: I work for google

"People who understand human beings" and the have the ability to synthesize those insights into compelling technology products are exceedingly in demand in 2021. Rest assured, we hire as many as we possibly can and offer them a lot of money/benefits. However, they can pretty much get jobs anywhere and many choose to start their own venture.

You seem to think it's quite easy though. Do you think you can do a better job at Gmail than the Gmail team currently? Or a better job with Youtube than the Youtube team?

Maybe it's not as simple as 'understanding human beings'

Can you make full text search work in Gmail?

Can you make youtube to not to recompress video?

Can you make Youtube search filtering by time work for at least few months? Having sorting by time fixed for at least few hundred search results will be good too.

Respectfully, the thing is, back when G+ came out, myself and many others pointed-out how Google's real name policy combined with Google merging all the existing Google accounts into G+ sank the effort - 'cause lots of people didn't want all their youtube or other data available to their friends and lots of others didn't want strong real name policy enforced - and Facebook had a real name policy that was and is only mildly enforced.

I don't think I'm a genius here since many others said similar things. Rather, the G+ action felt like corporate policy stepping on sensible thinking. Yeah, youtube, gmail and etc are hard to run, hard to alter and so they weren't going to change. But someone also felt they need to leverage all those desperate things - that just feels like what middle management does, not simply what people with not that much understanding do.

These are all valid feelings / hypotheses about what happened, but even with the benefit of hindsight we can’t confirm them.

For example, you think the real name policy tanked the effort. But did it really? Twitter verified status is considered to be coveted, people are lining up to get their real named acknowledged. Is that not enough to introduce plausible doubt?

Similarly, we always complain about “network effects” of social media these days - that’s why Facebook can maintain a “monopoly”. Well when G+ was rolling out, it was by limited invitation only; clearly in hindsight an extremely rookie mistake for a social network. Perhaps nothing else matters - perhaps not allowing ppl into your social network dwarfed all other causes.

We may never know as we can’t A/B test history.

It’s really easy to look at failed features in hindsight and say “ah I knew all along”. Statistically, most things fail. And it appeals to our innate desire to seek confirmation.

But I invite you to seek alternative explanations than “corporate messed it up”. And hey, if you really do know how to make successful social products, join ycomb, raise money and go be a billionaire

Twitter has a different culture. It's mostly public, whereas YouTube is mostly anonymous.

It _does_ sound like a "corporate messed it up", because if they had understood the culture surrounding their products they would have probably done things differently.

Here's a blatantly obvious example of that: https://en.wikipedia.org/wiki/Google%2B#Commenting_on_YouTub...

What about Facebooks 2006 redesign (although to consider this case study, we have to put aside the HN cultural disdain of FB and look at the situation clinically)

https://www.weforum.org/agenda/2016/09/facebooks-news-feed-i...

Lots of people hated it can claimed in the moment that corporate ruined everything. Then it went on to become one of the most successful products of all time.

Even if you hate FB today, you have to recognize that FB was beloved between 2007 ~ 2011 by pretty much everyone.

How was anyone to know, in that moment, whether the G+ name thing would be a major flop or a world changing product?

In my opinion, launching a badly received v1 of a new product/feature is not the same as attempting to forcibly change or go against the culture of your platform.

A quote from Zuckerberg's post regarding the News Feed is: "This is information people used to dig for on a daily basis, nicely re-organized and summarized so people can learn about the people they care about."

This does not sound like they were trying to push back against users, whereas Google basically said "no, you do it our way".

Currently, for the side of me as a non-technology user, one service that is the most valuable is the free unlimited high quality Google Photos.

I don’t know who made the decision in Google to sunset that and start to charge for quota.

That stupid decision is going to destroy a very popular and useful social network service.

Then you will have a “G+” hindsight again.

Somebody might have already submitted a ycombinator proposal to take over that market when Google Photos is dead.

> "People who understand human beings" and the have the ability to synthesize those insights into compelling technology products are exceedingly in demand in 2021. Rest assured, we hire as many as we possibly can and offer them a lot of money/benefits. However, they can pretty much get jobs anywhere and many choose to start their own venture.

How is Google to address this issue? Buying more startup teams and then abandon their products? What about the talents who do actually want to work for Google but have no chance to pass the high barrier of engineering culture which is the symptom of lacking “understanding of human beings“?

These are serious social engineering challenges Google is facing and deserve serious discussion.

How much is YouTube Google's success though, rather than their own, and Google mostly rode on that success after buying them?

gReader also had a social network component...

I laugh whenever the tech futurists talk about AI, machine learning, automation, singularity coming in the near future.

Most of the important software, that run Trillions of dollars in online economy, have dependencies in ancient tech. No one wants to work in that jumble, and updating has failed in one way or another.

Google ACH and ask why it’s takes several days to transfer money from bank to bank.

> Google ACH and ask why it’s takes several days to transfer money from bank to bank.

Given that there’s approximately one country left in the world where that’s the case, I suspect the root cause is not a technical issue.

Google+ was very poorly designed.

Google+ was initially very well designed, and then Google proceeded to run it into the ground.

Google+ when it was about a year old, was still the best social network ever. But then Google tried to force Youtube integration, killed off Events and Hangouts, not to mention the questionable "no unusual name" policy.

If Google could have killed Facebook, it would have.

Reply on news.ycombinator.com