Why Is Facebook Not in the Cloud Business?
interconnected.blog
interconnected.blog
Their abstraction for job scheduling (Tupperware) is about 5 years behind - something like Borg or EC2/EMR. Something like that is a fundamental reason why fb can’t do cloud as it is right now.
Plus, the infra teams do operate like product - impact at all costs. Which to most management translates to short-term impact over technical quality.
It would be a true 180 in terms of eng culture if they could pull off a cloud platform. An example is how they bought Parse and killed it, while firebase at google is doing extremely well.
Having said all that, I think focusing on impact over technical quality is probably the right business decision for what they were trying to do at the time - drive engagement and revenue.
I think someone at Amazon in the mid-2000s would not have necessarily looked at their internal stack vs. other companies and picked them to become the breakout cloud winner, but when I was there (late 2000s), the AWS team was kept quite separate, and aside from things like S3 the retail side of the company wasn't really using much in the way of AWS infrastructure yet.
Source?
It's certainly popular but everyone I know (me included) is running away from it asap.
Any particular complaints?
(I work on Compute Engine)
My new projects will all be on Azure because Microsoft has pivoted to a company with reliable investment in UI, developer-friendliness, and long-term support. Serving businesses has also been a core of their company from the beginning, whereas Google seems unwilling to provide the human support required and AWS is clearly at odds with Amazon's primary culture.
It does seem like the underlying products in GCP are very good, but they mostly seem to replicate other offerings from Azure and AWS.
I agree in general, but specific services can be a problem. New server images have broken my apps, waiting for AWS to update support for languages or RDBMS can be painful, documentation is generally confusing (mostly because there are 10 ways to do every simple thing), and I bump into bugs regularly.
I now run a large application mostly built in GCP and also now replicated in Azure for various compliance reasons.
So although my experience with the 3 clouds is not equal, I've been exposed to them all, and actually massively prefer GCP for the UIs and CLI tools.
AWS stll has better support by third party tools, and Azure is lagging behind everything.
I'm generally fond of our console, particularly the tools that show both the API and CLI invocations for most UI activities. On the other hand, I work on core virtualization, so the things I want to express tend to be very simple ("make me a giant VM", "make me a gianter VM", etc. :)
The docs are always a work in progress -- anything you can recall trying to do with GCP that was particularly inscrutable (can't fix everything in one go, but one fix at a time is better than nothing :)
- Most frustrating are the outages we have no control over. It seems like every other day, the server running our firebase instance mysteriously disappears from the internet. We can ping `my-app.fibaseio.com` but receive no events from it. This lasts anywhere from 5 to 25 minutes. Usually there is no blip in the firebase status page. The main devs have given up trying to debug it.
- We are nearing the resource limits of the paid plan for a single realtime database. We're working on splitting our one database into a master and sharding what we can.
- The particular eventual consistency and transaction semantics require workarounds everywhere. If you want an atomic transaction, you have to perform it on some common parent, but you are also encouraged to keep your data model flat and normalize as much as possible. Our integration test suite is tiny and horrendously slow because we cannot rely on Firebase to be timely, so there are huge timeouts everywhere that regularly take ages. Every past attempt at tuning the timeouts to be shorter eventually causes spurious failures a week later.
- Unless your data closely models the kind of chat application Firebase was built around, you end up needing a real database eventually. Not just to perform real queries with joins and complex logic, but also to essentially maintain indices that anemic mobile clients can use because the firebase filtered query semantics are limited to a single key. Now you need some kind of daemon to shuffle data in and out of your real database. Unfortunately, your real database is missing tons of foreign key constraints because that's the easiest way to handle firebase's eventual consistency.
That's just my take a month after joining somewhere intimately coupled to firebase.
Well, now you have my interest! Is this a talent gap or fatigue in the organization? Sounds like a meaty challenge.
I'm joining as the second backend developer, so maybe while I'm still green and have the naive bravery, I will go give it another shot next time it happens. I don't know much about advanced networking, but we suspect there's some kind of partition happening frequently between our servers and our production firebase instance. Uncertain how to debug that and how to work around it. It usually affects all of our servers, which are located in a single data center. I suppose next time I will try to track down and inspect the socket being used by the firebase SDK, but I don't really know what to be looking for at that point.
Firebase lets you grow super easily at first but the bigger your project the more of a problem it becomes in terms of development.
The biggest problem is that databases are extremely inadequate for anything that goes beyond prototypes or very simple uses cases. There are no relations, very basic querying / filtering, no ACID, etc. your logic to interact with the databases becomes super tedious. Fauna is by far a much better serverless DB.
There is huge vendor lock in and their client libraries are huge. The JS library is like 200kB minified and gzipped. You can in principle use REST to interact with the backend but you lose most interesting features (realtime, sync, etc).
https://bundlephobia.com/result?p=firebase@7.12.0
Cloud Functions are extremely tedious to work with. Most of your functions cannot be tested locally and they take forever to upload. Sometimes even minutes. Sometimes deploy gets stuck and don't you dare cancel the upload otherwise it will take more time to be able to upload again. With Cloudflare Workers you can do all in local dev and deploy takes seconds. With Zeit Now you can also make all dev locally.
The Firebase Console is a huge piece of bloated JS made in Angular 1 IIRC. It's also very basic as far as functionality goes.
Firebase hosting is good. Nothing to complain but all other options are equally great (Netlify, Zeit, Cloudflare Workers Sites, etc).
Parse was ahead of its time, good Engineering, nice documentations and possible the first platform to make building backend/database system for an app seamless. Even when Facebook killed it, they did a great job at open-sourcing the platform.[1]
I had migrated my existing services to open-source Parse and even built a stateless application with >250,000 users with it few years back. But if I had to build the same backend now, I'll probably do it with Go/pgSQL instead of NodeJS/MongoDB as in Parse.
Amazon did it a long time ago, and when they started, they had no sales team. They had to build that capability slowly over time, but they had the first mover advantage, so they had time to learn. Now they are number one.
Microsoft already had the enterprise sales team, they had to learn how to sell cloud. They had some trouble with the shift, and with the technology, but now they are knocking it out of the park, and are solidly in the number two spot behind Amazon.
Google is struggling with their cloud business. While what they offer is technically superior to all their competitors, they don’t have the sales team, nor the experience in building one. They are trying, but they haven’t gotten the hang of it yet, and in the meantime, Amazon and Microsoft just keep growing.
This is the position Facebook would be in. They probably have the technology, or can at least attract the necessary talent to build it (I know that I would at least listen if they said, “Come build a new cloud with us from the ground up”). But they don’t have the enterprise sales team, just like Google.
Now, in Google’s defense, they have realized they are coming from behind, and as such have focused on selling their higher level services. If you want to do AI and Machine Learning, Google’s cloud is the place to do it! And then maybe while you’re there, if your core business is AI, maybe you’ll use some of their other services as well, since your data is already there.
Facebook could make a similar play. They could build a cloud and then require any 3rd party apps that work with Facebook to use their cloud. This would jumpstart their adoption, and could potentially help them solve their privacy issues. They could require that you never send any personal Facebook data out of their cloud, and could closely monitor all the outbound traffic. Then they could theoretically allow even more access to Facebook data, if they knew they could control what happens to that data after the 3rd party gets hold of it.
In other words they could launch a cloud that lets you run your 3rd party Facebook apps in a controlled and audited way, and even give you building blocks to do it quickly and efficiently. It would boost their bottom line, because there is good margins in compute and storage, and at the same time give them more control of their own data.
Paraphrasing OP: Google knew how to sell ads, not how to sell cloud...
- The Dynamics suite
- Visual Studio / TFS
- Windows Server family products
All of those were "cloud-adjacent" even in their proto-forms as on-premises–only offerings. Even if they weren't "real cloud infrastructure" they were big Capital-E Enterprise products and a lot of selling cloud is still B2B Capital-E Enterprise work at the end of the day.
(Though it hasn't seemed to work effectively for IBM. That may just be IBM incompetence as old school commodity-priced Mainframe mentality should be exactly the same as cloud sales. There's probably an amusing alternate world where IBM Cloud is using something like 1950s presentations only slightly modified to sell "cloud" in 2020, just because they could.)
Microsoft sales has a skillset of going into a company, talking to the CIO/COO/CTO/CEO and selling them infrastructure to run their business.
Google has the skills of going to the CMO or the marketing manager and selling them advertising.
Which of those skill sets translates more directly to selling cloud infrastructure?
Some media sales-people would be super comfortable selling enterprise stuff (because a bunch of them started at MS/Oracle/Sun etc).
The front-of-house salespeople would be fine, so you'd just need to hire the back-of-house sales-people (normally the more technical ones).
The real problem is that Marketing and IT are very different departments, and it's hard to change contacts in one area into opportunity in another (I think this applies to both Google and FB).
But mostly, to answer the titular question: it's about margins, nothing else. There's less money in cloud than in ads. So, like Google, they'll probably do something like this when investors start to worry about the ad market.
It's the opposite in enterprise tech where Google is the underdog and needs to do quite a bit of selling, and base it on the strength of the product and functionality. GCP is hiring strongly for enterprise/software sales experience but struggling with the leadership to use them effectively.
There's plenty of margins in cloud though. AWS has proven it and GCP has the potential to be bigger than their ads business, but I agree that it's not currently as lucrative today.
Google had experience in the former. Microsoft had endless experience in the later.
That sounds like an antitrust investigation waiting to happen. (Or it should; that Google hasn't been murdered by prosecutors suggests that the system isn't working as it should)
AFAIK, Google does not require third parties working with Google to use GCP
Oh no, I don't think so; I was thinking of their exploiting their search dominance to push Chrome.
Don't forget the support. I used to work at a Series B startup, and AWS provided excellent support. We probably tossed them a few million a year, so I can only imagine what kind of white-glove top-tier support a big enterprise spender would've received.
We got responses for anything ranging from general UI questions to highly-in-the-weeds Redshift technical questions within 1 business day. >90% of the time, the issue was resolved upon the first response.
Google on the other hand, has very poor support. Facebook is by no means a support paragon either.
For this reason alone, I see a Microsoft-Amazon cloud duopoly with anyone else being a minor player as the only outcome. When shit hits the fan (as it is now) you need enterprise support capabilities.
Google we can't even get them to answer an email.
Google is more important to our parent org than Amazon; Google is a strategic partner, Amazon is simply another tech vendor. We might be part of a 11-12 figure megacorp now, but Amazon hasn't treated us too differently from since we were a tiny startup with not much spend, and neither has Google.
Google "support" is horrendous. You cannot PAY to get them to help you. We were on google apps (a long time ago now) and there was some state issue with admin transitions - so you'd get stuck. 100's of begging comments from plenty of PAYING users about the issue on their forums. Calls got you zilch. Crickets. Finally 2 years later - oh, we noticed blah blah and this might work now.
I would never trust anything critical to google. They literally will NOT take your money to help you - we'd have paid $10K to have someone press whatever damn button needed pressing.
One of our engineers needed to send out a couple hundred individual emails to other people inside our company. So, being an engineer he automated it, wrote a little script to send the emails via SMTP. I guess some system within Gmail flagged it as suspicious behavior and froze our account. Oh hey, now no one in the company can send or receive email. Great. IT tries to contact support. They told him to send an email from our account to open a ticket. We can't send emails. Took several hours of phone tag to eventually reach a human who wasn't on a helpdesk flow chart. His response? Google can't/won't do anything just wait for the automated systems to eventually release the freeze in the next few days.
So yeah, no company emails for the next day or so. Google thought that was a perfectly OK way to run an enterprise software service.
I like their search / email products though.
GCP support tended to give more generic or vague answers, or would simply "unblock you to the next blocker". As a support expert, you'd hope they understood the workflows, didn't seem to be the case. Google searches seem to indicate this isn't too uncommon amongst cloud platform users. GCP is the technically most sophisticated product, but my experience as a user was stability was far more comforting when fires broke out, as they will for any cloud vendor.
As an aside, I once experienced a G Suite "circular lockout" issue where I had to request permissions from myself. I spent hours agonizing over fixing it, and never actually heard back on that issue at all from the support team. I'm sure GCP and GSuite are independent support teams, though.
Note: I haven't used GCP in a bit over 12 months. Maybe things have changed since then, I don't know. But it certainly seemed that if you're a small company, they don't really care about you. I'd assume that GCP's large enterprise clients received excellent support.
It seems like half the time AWS's tier-1 support sends the me a link to the documentation that I linked to in my ticket, the other half the time I spend hours, sometimes days of engineering time to get the logs they asked for, only for them to come back with "I talked with the product team and oh yeah that's a known issue". If I'm really lucky, I don't have to prove to them it's their fault that something's broken before they admit that there's a problem.
Frustratingly, it's all covered under NDA, which is where things really get ugly, because you can't even really talk about it. I'm sure support is awesome if you're Netflix spending however many millions of dollars a month. Maybe at that level there's a secret site that straight up says product X has limitations Y and Z, but after having to prove to AWS that the problem is on their end, multiple times, their enterprise support is worthless. It's a good lever to push on in a contact though.
I've never used GCP support so can't say anything about it.
I promise you that there is not. :)
Usually it went the other way. We (Netflix) would say "hey we think we found a limit in product X" and they would come back later and say, "huh you're right no one has ever seen that before". Then we'd get on the phone with the engineer who wrote it and we'd work through the bug together.
So in a sense, yes, we had amazing support, but only because we were their beta testers. It was a good relationship though, because it meant when we had problems we got a lot of help.
They were always willing to put the resources in to make sure we were insanely happy.
But talking to other customers, that part of the equation seems to be there no matter who you are.
One is that you can't trust them to stick with a product offering. They are driven by a throw things at the wall and see what sticks strategy rather than some deeper vision for the world. So I'm always suspicious that they will abandon anything they launch.
Two is that they don't believe in offering quality support. I had this issue with an Android app recently. It got flagged incorrectly for a compliance concern and I was not able to reach a knowledgable human under any circumstances, including getting it escalated by internal Google staff. I would have easily paid a support contract, $1k or more, to get access to a human. In the end, I had to guess at what their algorithm was flagging and just ended up tricking it through trial and error.
But why? Per OP - they'd be willing to pay for an enterprise support contract. Why should a mom and pop shop expect that their experience being a small fish will be any different in GCP than literally EVERY OTHER GOOGLE SERVICE?
It's one thing to tell people using a free service that their only option is automated support - it's quite another to tell customers you just flat-out refuse to offer a paid support model. That tells me that you are organizationally deficient at providing customer support. I've yet to hear anyone using paid g suite speak praises about their support experience if they ever have issues. Quite the opposite.
Well, for one thing, everyone you ask except apparently the HN comment section will tell you.
I've probably seen this exact conversation play out 10+ times now. Someone says that GCP has poor customer support by analogy to other, mostly free services. Someone who actually uses GCP customer support claims that this is not the case. Some third (or perhaps the original) person blows them off and insists that Google's behavior toward the statistically adversarial and free users of other services must be representative of Google's behavior toward an entirely different group of users.
It's baffling.
Not to mention that I have had rocky issues with GCP as well. There was documentation that lied about its caching behavior, cost me over $1000 of my personal money, they took over a year to fix the bug and offered no reimbursement. Maybe I'm a small fry and don't deserve support, but this is the kind of customer management that Google is absolutely terrible at.
It's easy to talk about Google apps that have been killed. But there's an entirely different category which just got abandoned or lost their ambition.
So given that, I can take in and acknowledge what you're saying about GCP having good or even great support. I just don't have any faith that it will last.
I saw their Android support through the lens of comparing them to Apple. I pay a small amount to Apple and have no problem getting my support rep on the phone and all support experiences I've had there have been amazing. So why doesn't the Android experience compare? I just don't see it in Google's DNA to fight to win. Android has given up and is trying to be the lowest quality second place they can be. And that's exactly what I expect to happen from GCP. Maybe GCP will fight hard at the beginning to establish itself, but after that, I expect them to do the minimal amount to maintain third place.
Maybe that's the case, but I can't see it. Cloud computing is such a huge business, it doesn't take more than a third-place position to make more money than say, Youtube. Some random article [1] has GCP's 2019 Q4 revenue at $2.6B compared to Youtube at $4.7B, and total revenue in the cloud market is definitely going up way faster than total revenue in the advertising market. Plus, the "Google sucks at products" narrative is based on a reputation that people at Google would much rather be building technology than products. Cloud computing seems like the perfect match for such a reputation. App Engine is almost as old as AWS, and looks very much like it started as an excuse to have something for Guido Van Rossum to do.
> I saw their Android support through the lens of comparing them to Apple.
> So why doesn't the Android experience compare?
It came out in the Google vs Oracle lawsuit that Android had only made Google something like $10B since it started. So GCP is already making something like 10x what Android makes, and given cellphone saturation this will likely only grow.
> But there's an entirely different category which just got abandoned or lost their ambition.
AWS has a whole pile of services that you can tell no longer get any attention, or in some cases, even have full time staff anymore. Services that still aren't integrated with CloudFormation after years and years because clearly nobody cares. A guarantee that the lights stay on and the service continues to handle requests is the most you can ask or get from any of the existing cloud providers.
[1] https://www.businessinsider.com/google-cloud-revenue-first-t...
That’s also a wrong assumption. Android devs may very well be paid GCP customers if they had a great experience with the Android platform.
I am not talking about only full stack solo developers, I am also talking about companies with mobile apps with backend needs.
You go to a restaurant, try a dish and it gives you food poisoning. Do you have to go back and try every single item on the menu? The other ones might be great, but realistically you probably go to a different restaurant after that.
Free Android app developers
You just described their problem. You pay them money, whatever they ask, and are shocked that e human provided support to you
Their support is downright AWFUL. Getting someone from Google to help is so challenging that they have decided that in order to work with them on their enterprise offerings you must go through a 3rd party vendor. Their third party vendors are all small companies with which my organization has very little trust for.
I will never knowingly try to do anything with Google again after the hellish experience I have had dealing with them and their vendors so far, it just isn't worth it.
We're not going to be the biggest fish, but our spend is approaching $1mio per year. If you spend more than that and have a TAM YMMV.
"I got a counterfeit product on Amazon, therefore I won't trust AWS".
This is actually a good argument. I can easily imagine that this could become a problem for Amazon in the future.
"Amazon continues to allow counterfeits to pervade. I cannot trust them with anything important."
Trust is trust, and if amazon can't be trusted to behave well in one context, over time, that erodes people's trust in the platform to behave well in other contexts. Thats what's going on with Google - they killed reader, and it's the same company, why are we magically expecting different behavior in cloud?
It might not be warranted, but that's irrelevant. Trust is earned, and Google isn't entitled to it, they have to earn it. Doesn't matter how they lost it, market forces are market forces, they need to get it back
https://medium.com/@serverpunch/why-you-should-not-use-googl...
I understand Google heavily rewards new product launches with promotions. Is that part of company culture not at all within the GCP org? I don't know, I'm asking.
People aren't conflating GCP with the rest of Google. They're just unaware of any markedly different promotion incentives in that org.
Of course it affects GCP, because due to their existing reputation, people think “Google has not cancelled any GCP product yet”
I feel exploited. I feel like I am the product. This might work when providing a free service to the public but we, developers, are not the general public.
As condescending and cliche as it may sound, we don't like our time wasted. Google's free services should be a magnet to entice, a funnel to capture and channel our hearts and minds, not the developer repellant they have become.
The fact that someone has to spell this is out is a testament to how out of touch Google has become with its traditional base - developers.
You have to be kidding me. Their payed support is terrible. We recently had a problem with occasional timeouts contacting the GCE container registry which causes our autoscaling groups to sometimes fail to start new nodes.
I shit you not but this one of the selected support answers (after many prior back and forth). Obviously, this issue is still ongoing after many days.
"Thank you for your information.
I have searched our internal documentation about any service outage and any network issues open recently, And so far I can’t find any explanation of your issue.
However, I found the following instruction [1] where describe how to run a local copy of the Google Docker Registry.
In addition, also attached the docs where described about how to set up a private Docker registry [2].
If above instruction does not work then please let me know and I will be happy to help you.
[1] https://stackoverflow.com/questions/27243294/unable-to-pull-... [2] https://www.digitalocean.com/community/tutorials/how-to-set-...
Seriously? Run a private registry? We don't run a private registry because we don't want to deal with that stuff.
You've really hit the nail on the head for me there. The people I really respect are the people who can look at idea a and say "that's failing because it's a bad idea" and look at idea b and say "that's a good idea, but we need to work harder at it". Google (from the outside) appears to say "These are all ideas, and they failed, kill them". There's no understanding or insight into the world, there's only 'experiments', which is as good as throwing a random number generator at an authentication system.
They also had to grit there teeth and embrace Linux, which would have never happened if Ballmer was still there. There is just no demand/need for Windows only cloud offering. You want to be in cloud? You better embrace Linux.
That tends to promote product churn.
Don't forget that Amazon is essentially a retailer, and their a certain amount of customer trust is essential.
https://telegra.ph/Facebook-Social-Services-FbSS-a-missed-op...
That does sound good, even for someone like me who has spent considerable effort ridding my life of FB's influence.
The killer product they have is their users and the data mined from them. Allowing third-party apps to leverage their infrastructure as a service would be brilliant.
I don't have exposure to their offerings but I've heard this and it is rarely qualified with specifics. What is it they have that is superior?
Edit: Amusingly I was downvoted for providing specifics. So I guess others don't agree with my assessment?
Serverless is a great example of this. GCP and Azure both have serverless offerings, but neither of them has the equivalent of Lambda Layers, which has been groundbreaking.
Even if we grant that Google's network is better, how can you point to that single dimension and claim that GCP is a better cloud platform? For most business's use cases it would be professional malpractice to recommend GCP over AWS or Azure.
AWS has a much bigger breadth of offerings, and I 100% agree with you that it would be malpractice to recommend GCP over AWS, for myriad reasons.
But by mentioning Lambda layers, for example, you're not making an oranges to oranges comparison.
For the actual functions offering, for example, the Google one is cheaper, has more consistent network access to the data stores, and starts up faster. Technically superior in every way.
But I would never use GCP's serverless offering unless I had to.
The fact that Oracle's main relational database offerings are still the most technologically sophisticated does not mean that Oracle Cloud is technologically superior to AWS (or GCP for that matter). It just means that they beat the other cloud providers at one thing.
You sure about that? GCP has hired (or acquihired - Diane Greene for example) sales/management execs from all of the big boys in the industry (VMWare, SAP, Oracle, etc).
1) I see no indication it's "failing", just lagging behind the others.
2) My theory as to why its been lagging behind AWS/Azure is because of trust. AWS was first and is now the "no one got fired for using AWS" of cloud computing. Microsoft is simply entrenched. Oh you want to migrate your on-premise Hyper-V Windows OS's to Azure, we'll gladly help you click this button. Google is notorious for lack of "can I call a person?" support and I think that's permeated itself into its enterprise sales.
And by struggling I mean struggling to keep up with Amazon and Microsoft, whose cloud computing units are significantly larger and growing faster in both absolute dollars and by percentage.
In other words they are behind and falling father behind.
Where are those specific numbers? AFAIK they don't split it out from all of Google Cloud.
EDIT: I did some quick googling and no one seems to share anything. So where are you getting your guidance may I ask?
https://techcrunch.com/2019/02/05/googles-still-not-sharing-...
No it's not...
https://insights.stackoverflow.com/trends?tags=parse-platfor...
The article is referring to IaaS from my initial read.
Rather, it appears a highly targeted backend specifically for mobile apps only. The article says "back-end services for data storage, notifications and user management".
They're some pieces of a cloud, for sure. But it's worlds away from the general-purpose offerings like AWS and Azure that the author is talking about. So not surprised the author didn't bother to mention it.
“... a hypothetical ‘Facebook cloud’ could offer very attractive, differentiating services beyond the basics of compute, storage, and network.”
Visa and Microsoft make a ton of money being a grease or tax on other business, a small shaving of money in exchange for massive efficiency, instead of trying to BE every business. Is anyone offering an out of the box Social Graph as a Service kit?
It would also be a huge growth opportunity for Facebook Ad Services, to be THE ad marketplace of all new social networks. And in a play right out of Adobe's playbook, buying some things like bigcommerce, ecommdash, shiptstation would position them to sell underlying services that allow their customers to compete directly with amazon, ebay, adobe, salesforce. There is surely a lot more money they could make selling saas and pass, vs becoming another generic iaas provider.
https://telegra.ph/Facebook-Social-Services-FbSS-a-missed-op...
If the growth knobs were turned off across all it's products and spare capacity became a thing, then it might make sense, but at the rate they are building there is no purpose to being a middle man.
Make a code optimization and save 1-2% of CPU on the web tier and you will be a hero.
I had a chunk of credit to spend on GCP, and it's been much better, and seems to be about the same price. (We went with AWS at work because, at the time, it was simply cheaper for us).
What would be interesting though is infrastructure that leans on what they know best in terms of architecture and development.
GraphQL APIs, React frontends, PHP, JS, MySQL... Stuff that works for them is widely adopted by the webdev world as well and webdevs want convenient, scalable infrastructure.
Oh and they also have people working with/on Haskell right? I bet tons of Haskell devs would love use FB infrastructure if Haskell support was provided. A niche market for sure, but I suspect it is one that longs for something like this.
The industry is closer to large airplane manufacturers (e.g. only Airbus and Boeing), search engines (Google and a little bit of Bing), and similar.
Shoes, jam, and hosting services all require minimal investment. (Cars require a lot of investment too, but not at the level of building a cloud. Which is why there are more than 4 brands of cars, but not 4,000 like shoes or hosting services.)
EDIT: In response to comments below, I'm not talking about physical infrastructure here, which does scale rather than being largely up-front. I'm talking about the absolutely massive software investment in tooling and reliability. Not only are you building 1,000's of API endpoints across dozens of services, but you need to also guarantee 99.99% uptime and virtually zero data loss when required. And sure, a company like Facebook is the type of company that can engineer that stuff -- but it still costs $$$$$$$$$$$$ to build.
Citation needed. If anything, it's the opposite, it scales up very linearly.
How does entering into Cloud Business align to the company mission? Wouldn't it be a distraction?
The main reason is because it doesn't align with FB's values.
How is building a cloud business going to connect the world?
(As fluffy as this sounds, mission was taken seriously -- all the way up to Schrep)
And they know it too, so they rather compete in fields where they are good at, namely social media and apps. Maybe its part laziness, part lack of interest on their part too. But to me it would be wiser to focus on some promising new areas, rather than start catching up 2 laps behind the competition. Yet it's not easy, like VR/AR market shows. Maybe in ML services they could challenge Google a little more.
The last thing you want in high availability cloud infra is a team of engineers who will sacrifice stability for innovation.
Even if they did do it, there's not a chance they could launch it under 'Facebook'.
I think your first-mover and competitive industry points stand, but I don't see how retail and search synergize better with cloud infra than social. The cloud infra business model is (as far as I can tell) just building general purpose computing-at-scale features in a reusable, billable way and selling those to consumers. VM/container/database/serverless/etc orchestration isn't more aligned with retail than other businesses as far as I can tell.
Search, yeah. Intrinsically maybe not, but Google has some very skilled engineers and researchers, who have pioneered a lot of the modern cloud applications/algorithms eg Borg/k8s, MapReduce, GFS and so on. So they have the know-how on how to build topnotch services, just not maybe as good business-savviness to sell them.
1) its hard to do well and still be profitable.
2) It would require a massive internal shift, The security tools internally for doing things like IAMs are far to primitive. It would take years to re-tool, stopping momentum on products that make money
3) its not where Zuckerberg see facebook in the future. That future is AR/VR.
Google stumbled with AR, It's structure means that any AR will eat at the mobile buisness.
Apple might make a splash, but its unlikely that they will be first to market.
Which leaves a whole segment for facebook to dominate, if it makes a decent product, and tackles it's image.
AR/VR will still take a while to be mass market. We need a lot more compute, better algorithms and hardware.
This is a myth that’s been disputed several times by high ranking Amazon officials. Amazon never turned their internal infrastructure into AWS. It was always a completely separate initiative. Amazon was not on AWS for years.
Google also only uses GCP for a few internal services.
Facebook’s hardware designs might generalize well, but perhaps not.
So the conclusion is that, whether you're a financial institution or Google, you need ECC, at which point running too hot doesn't have any appreciable effect anyway.
First it’s old data, but google having observed higher error rates suggests different choices. Location, power supplies, motherboard design etc can all play a role. Further, they should be optimizing for slightly different things. Finally, they got data for temperatures from running their actual production system at these temperatures while assuming it would cause even more problems.
Alternatively, it suggests better data from a larger, real-er world study.
>Finally, they got data for temperatures from running their actual production system at these temperatures while assuming it would cause even more problems.
I'm not sure what you're saying here. To be able to detect the effect of temp, they ran at both higher and lower temps and compared.
Right, but they aren't published in controlled studies, usually. The article mentions that prior to the Google study in question, the next best example used 300 machines.
So yes, compared to contemporary (and since then!) public studies of ECC faults, I'd say that the Google study is pretty darn authoritative. You're welcome to cite other recent examples to the contrary though (with DDR 1 and early DDR 2 RAM, of course. Modern sticks fault less).
> The ranges are simply for location or time specific variations not some uncertainty metric.
I'm not sure what you're talking about. Section 5.2 and 5.3 of the paper is about the effect of temperature and utilization. It shows that temperature has a negligible effect when controlling for utilization, while the reverse is untrue.
Supercomputers of that era had far more than 300 nodes, and you can find studies of their memory error rates. The Roadrunner supercomputer for example had 19,440 compute nodes with 4GB of ram each.
PS: The architecture was odd with 6,480 Opteron processors and 12,960 Cell processors + 216 System x3755 I/O nodes, but it’s still using commodity RAM.
This is simply a low effort type of paper on an important issue. So, there are plenty of them out there.
PS: If you want to compare here is one for DDR3 https://www.cs.virginia.edu/~gurumurthi/papers/asplos15.pdf
> technical infrastructure to support their original businesses[0], then turn those resources into services[1] to rent out in the form of cloud[2]
[0]: i.e. Borg
[1]: i.e. Kubernetes
[2]: i.e. GKE
Edit: and https://cs.opensource.google. It’s this product: https://developers.google.com/code-search
I'm not sure anybody's ever seriously claimed the software infrastructure was the same. The "origin myth" of AWS I've always heard is that many of their servers were going unused outside the Christmas season, so AWS was a way to monetize them the rest of the year. At a hardware level.
So all these tech companies have been able to leverage their technical infrastructure -- the hardware one.
It kind of doesn't even withstand a moment's scrutiny. What was Amazon going to do during Christmas? Like, if they lease all this spare capacity during non-peak, what happens to those customers during peak? There are spot instances now, but the first two AWS products, S3 and EC2, didn't come with some sort of, "Except during peak" contract. So Amazon needs exactly as much hardware to meet demand during Peak as it did before, and now it's got a bunch of customer workloads it also needs to serve during peak. I don't see how this would've solved any problems.
Most stories about the origin of AWS have a few things in common: (1) There was limited support for the idea of AWS at senior executive levels (2) The early AWS products were not used by for any significant Retail workloads within Amazon for quite a long time (as in, several years) (3) As recently as a few years ago, there were major organizational pushes to move legacy Retail workloads onto AWS (4) As recently as last year, there were still so many teams on Oracle that everyone with an Oracle dependency was required to have an OKR to get off it. I include this because IIRC this is public-- after Oracle was making hay with the "Amazon sells Dynamo/RDS but runs on Oracle" campaign, Amazon publicly released information about their Oracle Annihilation roadmap. So it supports the theory that it's taken Amazon quite a long time to shift certain Retail workloads to AWS, even in cases where an AWS product that's suited to the workload has existed for quite a long time.
I've never worked for Amazon, so I don't know, and it's impossible for all the stories I've heard from ex-Amazon people to all be true, but I think there's a pretty good argument against the "we should put this spare off-peak capacity to use" AWS origin myth.
Obviously they wouldn’t pull the rug from folks and spot didn’t exist, but that doesn't mean it didn’t happen that way.
1) being too late in the game, at this stage.
2) unable to rebrand themselves as a trustworthy service for enterprises. They're stuck with the "social" and "fun" label.
Their previous forays into PaaS, done at a time when Twitter was also in this space, have faltered. And Facebook's pre-existing services are too far up the stack to be relevant to any IaaS ambition.
I doubt the 'trust issues' contribute significantly in Facebook's decision to not offer this, but I agree they'd be a factor for potential customers once the offering came to be. Facebook's branding is odd in that they've doubled down on the Facebook brand despite acquiring services that flourished in part for being "not Facebook". They seem resistant to rebrand into a holding company, and their choice supports the view that the company's tech mission is to remain tightly coupled with their flagship social network.
Mobile was the best thing that happen to them.
Software is just part of it. It is a logistics problem fundamentally. And Amazon is not only a software shop, it is also the most advanced logistics company in US, both physically and virtually, and those two go hand-in-hand.
So no, I don't think FB can get into Cloud business anytime soon, and I also doubt they have that much of the cash to actually expand the business, it is going to be huge investment, globally. The market right now is not going to be forgiving to new comers.
Disclaimer: ex-AWS employee
Though I doubt that's the reason GCP is where it is, it doesn't really help your point.
Sure they have. The very same election as Facebook (and Twitter), in fact.
https://www.zdnet.com/article/google-russian-groups-did-use-...
Facebook would need a way to differentiate the search experience. I don't think personalization is enough since Google personalizes too.
I also don't think a company should build products for the _purpose_ of revenue. Revenue is the output of building a great product with a purpose for users. So I think it goes back to Facebook asking what unique purpose would it provide in it's search experience?
Facebook already has the analytics in place from their share buttons, ad platform and user engagement on the feed to create a great ranking system. A lot of people already use instagram to search for restaurants, fashion, travel and products, same goes for facebook marketplace and business pages, all they'd need to do is expand the search bars in their apps to include web results. Tracking search queries would give them true intent data that they could use to improve their ad targeting across their whole ad platform.
The only thing google has on facebook right now is the ability to let advertisers target search queries. (just did a google search and it looks like they're starting to add that option: https://www.facebook.com/business/news/reach-people-who-are-...)
I'm sure they already have most of the web indexed and search history would give them amazing information about their users.
> biggest hurdle is trust, given its problematic record on data privacy
It is? FB is a corporation - much like Apple, Microsoft, Google & Amazon. Each of those have had their fair share of issues throughout the years.
Too easy to be polarised with stuff like this. FB produced good tech and supports it too - which is more than I can say for Google. But I'll use Google for their search & docs products, which is top notch. There are no heroes/villains.
The websites that have been the worst at maintaining my privacy were those that primarily sold physical products or medical services. You'd be amazed how many organizations broker data now.
Ads are a high margin business. When looking at FB vs AppMicroAmaGoog financials, Facebook has the highest % profit. They also make the most per employee. Plus they are the king of social networks with Insta & Whatsapp as their brands.
What works against them is Facebook doesn't have a great reputation with developers, so even if they step into cloud, it will be very hard. Plus corporations will always be worried that Facebook will peek into their usage and power their ads engine.
Who on earth believes that GCP is in the same universe as even Azure, let alone AWS?
As long as Google keeps on positioning itself as reliable and does a few things really well, they'll win those segments. They're already doing that with K8S but their recent emails about per hour pricing was just stupid. It sends a message that their VPs just don't understand what people really want.
Would Facebook have any credibility with enterprises?
My sense: No.
What cluster management software did facebook open-sourced ? I don't see any. Maybe they use existing open-sourced techs.
$26.7B in 2019
https://www.forbes.com/sites/greatspeculations/2019/06/21/or...
Not obvious; Oracle is known for underhanded marketing and lock-in. Or if you prefer: Successfully selling to the C-levels doesn't mean that you provide actual value.
"Move fast and break things" is okay/great if you can control the deployment of code and use tests to make sure it still works how you expect it to. It's less great if the infrastructure keeps failing (it's not failing, it's changing!) and you can't test before the roll out.
You can't compare the two and their safety anf 'favor among developers', devs also fought against FB by having them change their licensing bc FB made the horrendous decision to essentially own any and all projects developed on it.
Devs don't trust facebook, such as no one should.
https://marketingland.com/amazons-booming-ad-business-grew-b...
https://www.searchenginewatch.com/2019/08/01/amazon-google-m...
Speaking of their infra, Netflix focuses on productivity, based on a fundamental value that everyone inside the company has freedom and responsibility. This culture empowers the engineers there to build amazing tools that make employees very happy, but in the meantime the stacks they built may not be suitable for serving individual enterprise customers. For instance, they certainly don't fully use the elaborate security infrastructure that AWS offers. They don't worry too much about being frugal, either. Remember the famous saying that "Netflix is a log company that happens to stream movies"? They do store everything, at the cost of millions of dollars. Their monitoring system is so easy to use and so expressive, that it blows out any known monitoring systems out of water, but in the meantime it requires thousands of machines (amazingly, it used to be only 4 engineers manning the whole stack. A perfect example that shows how much productivity good systems can bring to engineers).
So, yeah, great company and great stacks, but it does not mean they can compete in enterprise cloud business.
As far as I understand Cambridge Analytica used the publicly-available Facebook APIs to create a stupid web app (a personality test or similar) which asked users to log in and hand over a lot of data (both about them and about their friends).
While their end-goal was against Facebook's ToS, I don't think there's anything they could've done about it ahead of time as 1) removing those APIs would hurt legitimate usage (let's say for example I am building an alternative client or similar - I want to be able to see my friends' data on there) and 2) it is still the user that is willingly giving out their and their friends' data when logging into such an app (and Facebook makes it clear what data the third-party service will be able to access when you first log in).
In short it's idiots logging in and delegating their permission level (which allows them to see their friends' data) to a malicious third-party. Facebook isn't really to blame there. It's no different from someone telling me a secret and then I go around and tell it to everyone else.
So I decided to check on the article itself [1].
>"Adoption of public (shared) cloud services continues to grow rapidly as enterprises, especially in professional services, telecommunications, and retail......
Basically what IDC labels as cloud isn't what is being discussed in the blog, or in the HN comment section which is pretty much strictly IaaS business. The cloud from IDC includes SaaS. Things like ERP and CRM.
And they even said this
>Software as a Service (SaaS) will be the largest category of cloud computing, capturing more than half of all public cloud spending in throughout the forecast. SaaS spending,
Which basically makes the blog and ~60 comments on HN so far irrelevant. The absolute vast majority of growth are going into SaaS, aka Microsoft's Exchange, Dynamic 365, Office 365, SalesForce etc....
What about IaaS?
>IaaS...., will also be the fastest growing category of cloud spending with a five-year CAGR of 32.0%.
That is great. Until you look at the number closely,
>China will experience the fastest growth in public cloud services spending over the five-year forecast period with a 49.1% CAGR. Latin America will also deliver strong public cloud spending growth with a 38.3% CAGR.
What that means is that the CAGR, while impressive, will mostly be focused on market that are not current well served, China, India, Latin America, South East Asia. Of Course Europe and NA are also growing, but likely at a comparatively lower pace.
All these are 3 years from now, a very short period of time in terms of infrastructure building. Is this far too late in the Game for Facebook? Do they have a long terms sustainability plan? As we have been shown again and again, Enterprise Tech moves very slowly and tends to stick with big brand or long time partners. One can look at Google Vs Microsoft as example.
It seems the money right now is on HPC, AI, Scientific Computing, Deep Learning, all these seems to have infinite appetite of computation power that even with 1.4nm by 2026 the performance still would not have satisfy needs.
But what about normal Web Server usage? Are we close to its peak?
The growth in Cloud is partly ( or mostly ) driven by Smartphone, the world went from 1.5 billion internet user using it during free time to near 5 billion user using it All the time. Smartphone sales, and user growth has slow down, IaaS usually being lagging behind in supply, are we nearing the end of the growth curve? ( Excluding Scientific Computing Needs )
Remember a lot of these are talking in Dollar terms, 3 years from now the same dollar should have bought you 50% more computational power. And that is excluding any possible disruption from ARM.
ps: i think it will be public cloud someday
really.
those folks have a natural tendency to fatten the middle management layer and focus on buzzwords that shake the core business (mobile! vr!) and feature creeping until the product is so broken it dies. all while killing the few trully tech/innovative teams.
You can't actually buy anyone's data from Facebook, you can only buy ad targeting.
How about we, as a collective, say it for what it is instead of using words that mean something completely different and let people come to their own opinions instead of thinking for them?
As bad as a news outlet.
If you had effectively infinite money to make any idea you can think of into reality, would you use it building cool things that you are personally interested in, or would you use it to start another cloud company dedicated to prices so high it's practically highway robbery?
It would make sense if Facebook was ran by Larry Ellison, but money is pretty pointless as an end itself.
This is a term from the US Intelligence Community that refers to the 7 most prestigious universities in China that support the People's Liberation Army. This includes "Beinhang University" and "Northwest Polytechnical University". These schools are closer to the Army in China than say Westpoint is to the Army in the US. These schools guarantee some of the best paying jobs in all of China, and if you desire, a chance to reach the upper strata of the CCP hierarchy. For this, reasonably I think too, they expect a pretty large amount of loyalty measured in decades.
They have overruled consistently (there is nothing wrong with this I am just saying there were objections often) to massive capital infrastructures that make no sense in Facebook and that has always driven the finance people wild.
I don't think there is a conspiracy here. I do think that a lot of specific Chinese hardware manufacturers have gotten very very rich off of these decisions. Their architectural decisions are certainly really strange. They tripled-down on blueray DVD storage in 2009, even though massive cheap flash, then NVMe, then NVMe EVO Plus rolled out in quick succession.
I recommend this report and web app from the Australian Strategic Policy Institute: https://unitracker.aspi.org.au/.