Comment by thenewwazoo | Hacker News Reader

Comment by thenewwazoo | Hacker News Reader

I'm obviously not going to comment on anything internal, and I'm obviously speaking for myself and not the company, but it's worth bearing in mind that this migration was not from "on-prem" in the traditional sense. LinkedIn has its own internal cloud, complete with all the abstractions you'd expect from a public cloud provider, except developed contemporaneously with all the rest of the "clouds" everyone is familiar with. It was designed for, and is tightly coupled to, LinkedIn's particular view on how to build a flexible infrastructure (for an extreme example, using Rest.Li[1], which includes client-side load balancing).

There was no attempt to "lift-and-shift" anything. There are technologies that overlap and technologies that conflict and technologies that compliment one another. As with any huge layered stack, you have to figure out which from the "LinkedIn" column marry well with those in the "Azure" column.

I personally appreciate LI management's ability to be clear-eyed about whether the ROI was there.

[1] https://linkedin.github.io/rest.li/

Oof, I'm twitching just reading that because we're in exactly the same boat. The problem with the ROI is that any kind of not-self-run cloud is guaranteed to be more expensive in direct costs. This has been shown time and time again for any reasonably large enterprise. However, there is a long list of things that are hard to express in money that support a cloud move, mostly to do with keeping up with modern tech, hiring, DR, better resiliency, etc. and so the decision can be quite dependent on the particular execs in the chain of command and their subjective values.

This is based on the assumption that Azure has modern tech, hires well, DR, and better resiliency than LinkedIn's "cloud" for LinkedIn's needs. There's a bit of a problem around incentives here, where Azure is built to sell to Azure's customer base, whereas LinkedIn has evolved their own stack over the years.

The questions become:

1. Does it make sense to dump our special features in the stack, or move them to a higher level in the stack? 2. Does Azure have comparable capabilities for the LinkedIn stack? 3. Is LinkedIn worth it to Azure to sell to?

---

Often times, "at scale", you can support custom solutions outside of cloud providers that are purpose-built, and often times more resilient and efficient than the cloud providers.

AWS has taken a very interesting approach of building an incredibly wide set of solutions to support every customer under the sun, and their approach to being "customer obsessed" leads to them building super niche solutions if the deal is worth it.

I'm not sure how Google and Azure handle these engagements.

Google doesn't acknowledge that the customer exists or has wishes.

Microsoft will allow you access to the PM of the product if you are big enough, but his answer will be a not-exactly-committing "in 3 years" because their roadmap is full.

Every once in a while Google acknowledges the customer exists, but usually for a brief moment, either to close their account, or to tell them they are using the product wrong. They promptly go back to work after such brief outings, so as to not fall into the mistake of actually listening.

> 3. Is LinkedIn worth it to Azure to sell to?

Microsoft owns both, so that question gets convoluted. It's not really about whether LinkedIn pays Azure enough to care.

It's not really 'the cloud' as much as it's a managed mainframe you allocate resources from. Only it's actually quite expensive to allocate resources but it becomes more palatable with a monthly bill compared to setting up on-prem.

Costs more money but easier on the cash flow.

Telco is a really good example of how hard it can be to move to the cloud no matter how hard you want to. I imagine financial services are similar. Telcos built their own clouds because there was no other way to get the primitives that they needed (classic case: a floating IP that could appear in data centers on different sides of the country and be ready to serve clients in the middle of a call in under 100ms without dropping the audio). I mean, cloud just was not built for that.

The flip side is that it is freakishly expensive to keep doing that, and telco is not a business that is freakishly profitable.

I'm watching efforts like Azure Operator Nexus (https://learn.microsoft.com/en-us/azure/operator-nexus/overv...) which looks to be taking a stab at this, but even there, its more like OpenShift on specific hardware with some Azure dust, rather than a remaking of Azure primitives like redundancy for the telco universe.

Microsoft purchased tech and people from AT&T to help boost Azure's telco capabilities as a cloud solution[0].

I haven't followed what's happened since it was spun out, but I know that Network Cloud was heavily OpenStack based so I'd would be interesting to see how that may have changed but I no longer work in the Telco space either.

[0]: https://azure.microsoft.com/en-us/blog/improving-the-cloud-f...

Telco used to be all for openstack. They have all started the shift to containers and kubernetes

A lot of organisations were already doing “cloud” before the big migrations to AWS and Azure started in the EU enterprise scene. Around the 2010 it became a much better business case to get into the local cloud, which is basically where we bought our own hardware and put it in rented rack space at local businesses which were specialised in running it, instead of having it in our own basements. This mostly didn’t scale beyond the national, or even regional, level but then that is still not a big issue for most organisations today.

Then came the move to AWS and Azure, again because the business case made sense. A big part of this was Microsoft being good at selling their products in a bundle, so that getting into Azure sort of just made sense because of how well the IT-operations side benefitted from moving into things like Azure AD (now Entra ID for whatever reason), Azure Automation instead of scheduled tasks and so on. If your IT-operations infrastructure is already largely in Azure then your development will also be in Azure because it’s going to be such a tiny blip on your Azure bill that nobody will ever notice.

With the price hikes, and also just how complicated it is to actually run and maintain IT-Operations in Azure (I can’t speak for AWS) we’re now in a world where more and more organisations here in Denmark are looking to get out of Azure/AWS to reduce cost, and many have already made the move.

Of course no one is leaving Office365, so many of us are mostly just trying to broker better deals with Microsoft.

I am curious, which companies in Denmark are trying to get out of cloud?

Basecamp/HEY is one publicly famous example.

Another example is Ahrefs

Is colocation the cloud now?

Yes. And no. Depending on what you mean by "cloud" how much kit you have in colo, and how you manage the resources of that kit.

You can certainly use colo to provide the facilities most need from cloud infrastructure.

Just stuffing a server in someone's rack isn't cloud, but larger installations are often managed in similar ways.

I'd argue taking on prem applications and putting them in large (expensive) virtual machines on AWS ec2 or azure windows vm directly while postponing or deprioritizing any refactoring is not the cloud either.

At work, there is a push to move on prem applications to cloud but as much as possible without changing any of the code. We end up basically renting expensive instances for significant period of time and recreating all the on prem services on them.

So anyway, more like the reverse: cloud is colocation but expensive.

> With the price hikes, and also just how complicated it is to actually run and maintain IT-Operations in Azure (I can’t speak for AWS) we’re now in a world where more and more organisations here in Denmark are looking to get out of Azure/AWS to reduce cost, and many have already made the move.

It's an endless loop, I think. I've observed a very different behaviour in other parts of the world, where companies which either migrated away from the cloud already, or companies which never moved to the cloud to begin with, are now moving things into the cloud and other managed services in order to better focus on their core competencies.

Dropbox, for instance, shifted everything out of AWS in 2014 -- but are now moving a lot of their corporate infra back into AWS in order to focus on effective and efficient storage, which is what they're good at, without having to build everything else themselves. Likewise hedge funds - if you look at job postings for a lot of systematic trading firms (e.g. Jane Street, HRT, 2Sigma, etc.), you'll find loads of them are hiring extensively for people with cloud experience, because there's been a sort of wake-up moment where companies are breaking free of the idea that everyone can be Google, and realising that you can't build everything in-house if you only have 1000 employees.

So, interesting to hear of a move in the opposite direction in Denmark. I'm sure that ten years down the line, a new generation of engineers will be coming in and asking, "Why in the world are we managing all of this infrastructure on our own?"

> direct costs

> vs ""modern tech"" FOMO

Very tough decision, I'll need two MBA degrees to figure this out.

1000% Agree. So many ROI calculations I've seen in the past ignore key-person risks from internal systems, hiring premiums needed to keep specialized knowledge, and also carve-away infra human costs into other departments. Once you carve out enough stuff from the ROI, you can sway the ROI calculation heavily.

The folks who are too senior arent sufficiently technical to understand. The folks who understand benefit from the key-person risk and premiums paid, and thus are disincentivized to challenge the ROI comparison.

Key person risk should be understandable as redundancy to just about any exec…at least I would hope so.

> and also carve-away infra human costs into other departments

That's another great point; with the on-prem system where we run our own massive k8s/vm/data lake cloud, every team just throws things at it willy nilly and even though we know how to calculate total operating cost it's very difficult to set up backpressure to individual orgs and their budgets so there is quite a bit of waste I feel. With something like AWS the billing system is very detailed and is easy to attach to teams upfront. I'm not sure if it's good or bad that things are more efficient due to teams being incentivized to pay attention, but I kinda like it.

Yeah, based on my own experience with AWS and Azure (that has nothing to do with Linked In), my immediate reaction to the headline was, "well, you can be keen on Azure, but "stuck" on AWS for a myriad of other reasons". Reading the article pretty confirmed it.

I'm not sure how this has anything to do with AWS. But I guess you can use it to confirm your idea about vendor lock-in if you're bound and determined to convince yourself you're right. For what it's worth, working at AWS, I see a lot of companies who use Entra for AD and then federate into AWS using that.

I think you misunderstood OPs anecdote and took it a little personally?

They were simply saying that much as you can love one platform there are many valid reasons why your deployment on another makes it hard to shift that and recapture that configuration. Nowhere did they intimate it was a fault of the current platforms capabilities...

this describes github to a tee.

think very hard before you start building your cloud-native application on a specific cloud

From that repo:

> At LinkedIn, we are focusing our efforts on advanced automation to enable a seamless, LinkedIn-wide migration from Rest.li to gRPC. gRPC will offer better performance, support for more programming languages, streaming, and a robust open source community. There is no active development at LinkedIn on new features for Rest.li. The repository will also be deprecated soon once we have migrated services to use gRPC. Refer to this blog for more details on why we are moving to gRPC.

If anything, this highlights the dangers of building your app around the details of a particular clod infrastructure.

I guess in this case it was the other way around: building the cloud architecture specific to the needs of the app.

It’s not a good cloud if they built the wrong abstractions that aren’t generic

> I personally appreciate LI management's ability to be clear-eyed about whether the ROI was there.

Kind of confusing statement as they threw away tens of millions of dollars in a failed attempt to migrate.

At least they canceled the project when it became apparent things weren't going to work out as expected.

I've seen massive projects that have been train wrecks since day one pushed through until the bitter end because management can't admit failure.

We have that going on in Norway right now. Our state hospitals are switching several aging systems to some Epic-based monstrosity, and so far it's been a complete shit-show[1].

A major hospital has been running it for a year with lots of issues, some which has certainly affected patients gravely. It was rolled out despite heavy protests from doctors and nurses who had seen or experienced the new system at the smaller pilot hospitals.

Yet management is pushing on, and just decided the next big hospital should get the new system this spring, despite the many severe issues that are still unsolved.

[1]: https://www.nrk.no/emne/helseplattformen-1.15838445

That's not good management though.

Recognizing you fucked up and only cost the company tens of millions instead of hundreds of millions is "Not Terrible" management.

It’s not perfect, but it’s good. Or better than spending more.

Id like to be your financial manager.

The sunk-cost fallacy is a brutal one. Better to cut your losses than throw good money after bad.

Sunk cost doesn't necessarily mean good management.

It actually means the opposite. Instead of management identifying a boondoggle ahead of time they got the company into a mess.

It was MS buying LI and then wanting them on Azure. There's a huge Azure branding play here that - I'm guessing - drove this work. A distant second was some cost savings in the far future.

The hit to Azure's brand for this is probably significant. Who was looking at a migration to Azure before who's now discouraged?

> was MS buying LI and then wanting them on Azure.

LinkedIn's been extremely independent of Microsoft and still are just like GitHub and openAI.

This was a 100% internal LinkedIn management decision.

Could LI have selected a competitor?

LI is a monopoly in professional social networking.

there are no competitors

I meant a competitor to Azure

The only other option is their own internal stack.

I cant imagin being owned by MS they would be allowed to use Google cloud.

This kind of sentiment always shows up after the fact but the hard part is actually knowing it ahead of time.

Emphasis on knowing. I’m sure some people thought it was a bad idea ahead of time. But there are always people who think any new idea or big change is a bad idea.

The reality is that anything significant has to be tried out, for real, to find out. That’s the premise behind proven useful decision frameworks like agile or MVP. And as the GP says: good management shows up when they read the attempt clearly and resist the sunk cost fallacy.

Bad management sounds like “let’s never change anything” or “we’ve already come this far, we need to finish it no matter what.”

And to a large enterprise, tens of millions of dollars is not an existential amount of money to lose on trying something significant. Big sports teams lose that much on bad player moves all the time. “We can’t ever waste money on trying new things” is a great way to sink deeper and deeper into whatever rut you happen to be in at the time.

> Emphasis on knowing

Literally your primary job as an executive. Strategic knowledge and wisdom

If wasting tens of million on a bullshit initiative isn't an example of bad management then literally nothing is.

The only place these types of terrible managers can exist are too big to fail companies like LinkedIn and the Government where theres zero repercussions for poor management decisions because money keeps pouring it.

In smaller leaner companies where money is actually important executives actually have to be good at their jobs.

> "We can’t ever waste money on trying new things"

There's a difference between a good strategic decision and starting a "facebook for pets".

Some ideas are terrible at the outset.

And executives whole reason for existence is to make this determination.

Blueshift was an objectively terrible decision and whoever made that decision is objectively bad at their jobs.

Any amount of employee time spent exploring options that aren’t pursued could be framed that way.

I think it’s a pretty successful operation when you can make the exploration and come to the conclusion that it’s not a going to work, then pivot.

Sticking to the decision no matter what would be much worse.

Id like to be your financial manager.

All a matter of perspective. Did you lose $10 mil to avoid losing $1 billion in wasted effort?

Totally worth it. Your entire org also learned in the process.

Its like buying both Blockbuster and Radio Shack stock while their stock was in freefall then selling right before they declare bankruptcy and being proud at how good of an investor you are for not losing more money?

I feel like you’re reaching really hard to make this work.

Companies, teams and people do research and test projects all the time to determine direction. Making a decision like moving the entire LinkedIn data center is just a larger version of that.

a management decision lost tens of millions of dollars.

you're saying they're good managers because they didn't lose billions.

going by your "logic" its literally impossible for management to make a mistake because there's always some slippery slope they didn't slide down.

"""we put the company out of business but at least we didnt cause the collapse of matter into a black hole! we deserve some bonuses!!"""

this is how people who work at monopoly tier companies actually think lol

That's actually not what I said at all. What I said was...

> Any amount of employee time spent exploring options that aren’t pursued could be framed that way.

Employee time costs money. If 20 people are in a meeting for 1 hour, that 1 hour meeting costs the equivalent of 1 hour of each of attendee's salary. 20 people attend a meeting making $100 / hour, you just spent $2,000 for that meeting.

All time spent planning, also costs money. Every Agile "spike" to do research for a story costs money. Every architecture document. Every time you have somebody do a couple of prototype projects to test your options before you make a decision. Sometimes you can't easily find out what's involved until you really dive into the work. It's the nature of the beast.

For LinkedIn, purchased by Microsoft who also owns Azure I'm quite certain that during the process they assumed consolidation within Azure would be on the table. If anything it was probably expected to be a huge marketing push for Azure to be able to say that LI ran on Azure so it's likely a management decision that involved a lot of pressure to make it happen.

Being willing to abandon that decision after exploring to see what was involved is a big deal. In many companies you'd have just heard platitudes like "well we have to do it, find a way" perpetually extending the project while taking away from more valuable initiatives.

I used to work for a business telco that was acquired by a residential telco. They were so convinced they could roll this newly acquired company into their existing systems that they ran off all of the staff who built the systems. 2 years later, they finally realized that business telco was a lot more involved but by then it was too late.

I understand the thinking that got them there, but despite a lot of people trying to stop them they proceeded forward until their hand was forced. This LinkedIn decision could have gone much, much worse.

> being willing to abandon that decision after exploring to see what was involved is a big deal

Yeah but spending several years and many millions of dollars isnt 'exploring'.

I'd say it's a solid fail for the management.

But we can agree to disagree on this. I see you strongly believe that losing tens of millions is good management.

For what it's worth there's many posts nowadays on blind about how the culture of LinkedIn has tanked and I'm assuming management probably believes they're doing a great job in that area too.

You're really big on putting words in my mouth on this.

> I see you strongly believe that losing tens of millions is good management.

Costs are relative to size. There are a lot of people involved, it's going to cost a lot of money. Microsoft's operating expenses are estimated to be just shy of $350 million per day. At that scale, 10s of millions is not as impactful.

Could it have been done better? Of course.

Could it have been much worse? Yep. Significantly.

Reply on news.ycombinator.com