That sort of guy. Well he once told me something about executives and upper managers working for corporations that I have never forgotten. He said to me, and of course I am paraphrasing:
"Change gives the illusion of progress". I asked him what he meant and he responded with something to the effect of "They have the habit of changing big things every 5-10 years on purpose to make it look like they are productive, and to justify their own roles, one guy will come in and 'cut costs', the new guy after him will 'invest'".
IN PARTICULAR, at one Fortune 250 (which no longer exists), we implemented OneWorld to replace the mainframe. After 7 years, we still had the mainframe, AND a badly-implemented version of OneWorld. Then the company got bought by another Fortune 250, moves were made to "commonize" the IT systems, and then the parent company sold everything. But I'm positive that everyone involved in the project to retire the mainframe made the project look very successful on their resumes.
I don't know if the above fear would actually play out, nobody is willing to not make changes to find out.
High level management is fundamentally hard to stay on top of over time. It's about as easy as thinking chess is easy because you can theoretically know every move your opponent might make. There are so many moving parts to an organization that having visibility to them isnt enough to perform well. They have to influence most of the business pretty indirectly. If changing things gives you the confidence necessary to keep things running.. that's what you're going to do. Everything is a gamble, so doing nothing is kind of unacceptable leadership behavior unless they are actively taking up the mantle left by previous management and understand it very well already.
It's an area that is pretty tough to make meaningful criticism. its kind of a show dont tell type situation imo. If upper management seems under-qualified and overpaid to you, then maybe you have a calling to go perform better and get paid more.
Company management is in underdeveloped game-theory territory. Sure we can isolate one part of their job and describe how they are failing on it, but we dont know the trade-offs being made on a daily basis with their time and focus. a lot of which is going to be company secrets, if it even leaves their personal thoughts. Any criticism that comes down to saying "they should have done more" is likely out-of-touch, for example. Unless you can prove they were actually being lazy.. which is usually not the case, since they are often workaholics (ime). But we usually cant tell if something is a good move or not until it plays out on the market. So making criticisms based on hindsight is weak, as is making criticism that lack the full picture of the organizations goals and the time / energy they actually have on hand to accomplish them
CEO is a very general job role. I dont understand your point with the painters. That would be an operations issue, so I actually wouldnt criticize the CEO of the painting company at all for it. thanks to him/her, I was able to contact a painting company, they showed up, they painted the apartment, and left. I would think the operations team (the painters) deserve to be fired and held accountable for claiming they know how to paint a room when they clearly didn't. Nothing about their job is general or consists of trade-offs. If I said "you only have 10 minutes to paint the room and then leave" then yeah, it might come out like shit and the "excuses" would be valid. which is the kind of time pressure CEO's are often under with respect to things they are actually doing on a day-to-day basis.
i would hold the CEO responsible with respect to resolving the issue and refunding me, etc, since the CEO role is to be responsible for the outcomes of the company.. but he's not the one painting the room. Just like with any company, the CEO does not literally run the company. If service is poor, it is usually because people are finding their way past hiring filters to get jobs they aren't qualified for. Let's not forget that people everywhere are often advised to lie their way into employment, fake it till you make it, baffle them with bullshit, reword your resume to sound more impressive, etc. These people line the mechanisms that CEO's use to accomplish anything at any company.
Of course, the CEO position is no exception to this and I am not saying it is literally impossible to build a case against a CEO. I am saying it needs to fully encompass the position or else you're likely assigning criticism to the CEO for some culmination of lower level operational incompetence that they simply failed to overcome. If a director over-promises to the CEO and the CEO signs off on the basis of trust with the director, then when the bar is not met later of course the CEO will be held responsible but the reality of fault sits with the director, or maybe a subordinate to the director who convinced the director that the over-promise was doable. You then have to get into the weeds of whether or not there were signs the CEO should have seen as to not trust the director, or if they had reason to overrule the directors approach, etc. you then have to do similar things across all areas of the company to derive a valid criticism that the CEO is the common denominator in it all.
Leadership is significantly harder to criticize appropriately than operations. Personally, I would like to stop reading meaningless criticisms from people who want to complain and be heard but dont want to do the work necessary to make a valid complaint.
So if had a horribly painted room that you just paid extremely well to have completed, and the painters came up to you and gave you a laundry list of reasons that they failed... Would you hold off on criticizing them until you had a complete understanding of what it takes to be a painter?
trying to use painting as an analogous situation like that isnt transferable to the point i am making though. Putting the excuses in their mouth doesnt even make sense. We are presupposing that the painters did a horrible job.. while discussing how to decide whether or not a CEO did a horrible job. The only reason you know the painters did a bad job is because we are saying they did. the only reason we can use painting as an example is because most people can imagine a terrible paint job. i.e. we do have a full scope understanding of what it takes to be a painter. I am saying it is much harder to imagine the role of a CEO and what good results would look like than it is a painter.
Maybe my wording was fuzzy, but I am not saying you need a complete understanding of the CEO's role, but it does need to be of full scope. I see that reads near synonymous, so in other words it may be infeasible to account for the total depth of their role, but at the very least the entire breadth of the role should be looked at. If you default to "i gave you a lot of money to make it happen so it should be perfect" type logic; you're just being a "karen". the cost of something has nothing to do with the results, directly. Money needs to be converted into something that helps the work, and in that process we are all still limited by reality; diminishing returns, supply chains, quality of communication, availability of resources, etc. A CEO is at the focal point of all of this, and is human. Whether they get paid nothing or everything doesnt change how effective they can reasonably be.
But they do deserve criticism. it just needs a lot of work to do it right. you have to provide some sort of evidence that across all scopes of work the trade-offs do not make sense. maybe the CEO sacrifices on every front in order to provide the fastest service in the business and is successful in that. If you leave speed of delivery out of your criticism it becomes a meaningless criticism. "They charge a lot for poor quality". "These painters did a terrible job.. (even though I called them this morning, and they were done by lunch which allowed me to do a walk through with a potential tenant)".
All i'm really trying to emphasize is that we absolutely can criticize a CEO, but if you dont do it properly it is very easily washed away by the many unknowns of the position. however, if it is done right - it would be very damning as they cant default to company policy or directives from above as a scapegoat since they are the ones creating such things.
This is a bullshit reason based on jealousy, not reason.
> when those actions can have a negative effect on those below them and even to the external environment
This is the real reason it’s fair to be very critical of their maneuvers.
Reality is not zero sum, but neither are resources infinite. Is it really bullshit to critique more thoroughly that to which more of the finite resources are dedicated?
this kind of aligns with my point tbh. We give $10 critiques to million dollar positions as if they hold weight.
If this is trully BS, then allow lead developers to write checks for their whole team using the company's bank account.
they hold power in organisation, they can increase their own renumeration in a way that's rank and file staff cannot. Executive compensation has skyrocketed in the past 20 years.
If their management is ineffective, then they don't deserve top comopensation.
Management are contemporary clergy, spewing high minded ephemera, only to go home unable to point at anything net new left behind by their effort.
I grew up in farm land; we had no middle managers. Somehow food still got grown, harvested, and sold; somehow a Linux kernel and other wildly popular open source exists without them.
Post-WW industrialism needs to wind down. Militant minded people came home and forced their PTSD on workers. We spend a lot of resources equipping people to output nothing in deference to traditional economic memes. America of recent decades necessarily built itself into a production powerhouse to resupply a destroyed world. Such memes are outdated given automation and unsustainable given real material costs.
It's hard to keep a good (confident, ambitious) team from re-engineering. All the same dynamics apply: In your mind the disadvantages of change are small because you don't know them, but the advantages are large because you planned them. Making change gives you more control over your fate because you are executing your own plan as opposed to staying the course. Finally, how do you keep people motivated to show up every morning if you don't have a vision for change in the future?
I don't think its that different for managers and engineers. There's a lot pushing people to try something, even if the objective odds of success aren't great.
Satya: 1992, CEO in in 2014 Tim: 1998, CEO in ~2009 Sundar: 2004, CEO in 2015 Andy: 1997, CEO in 2021 Ted: 2000, CEO in 2020
They were all senior executives well before assuming their CEO roles.
That sounds ... dystopian.
That itself is based on the belief that (historical) progress is an improvement.
Mostly true over the last few centuries - aspirin is nice - except for those occasions where it was mass murder.
That's a very succinct way to put it. I think that observation also applies to consumer technology (e.g. regularly re-doing UIs to "improve" them when the changes are either in fact regressions or just different but not better). We've had it drilled in our heads throughout the modern era that new == better (e.g. the ubiquitous "new and improved!" marketing language), but that's not actually always true. Change for change's sake justifies itself through that misunderstanding.
In the recent past, we had a really high rate of genuine technological progress. But at some point we'll have picked most of low hanging fruit and will enter a period of slower progress, where faking progress will become more an more tempting for producers than the real thing.
Depending where, the real-estate and energy prices are nuts in most places. And, the engineers are expensive right now, and the services are cheaper.
It's not about giving it up, or change for the sake of change, it's about seeing the writing on the wall. These managers see the larger trends, rising energy costs, maintenance costs rising, hiring difficult, and retention of existing engineers impossible. Once you see those trends and a department is underwater and it's getting worse, you have to move. At another point in the market, when you may find engineers are less expensive, easier to hiring, technologies and space are less costly and easy to deploy. You move back.
But how will the cloud providers avoid these trends? They won't. They will have to do the same thing as anyone else: pay more. And therefore charge more. There are economies of scale, but those savings are logarithmic and a company like FedEx is already pretty far out on the X axis.
AWS and Azure probably run hundreds of Fedex-sized clouds and it's their core business. I don't think Fedex is very far compared to them.
Things that aren't Fedex's core competency.
Or, Microsoft's roofless datacenters[3], or locating datacenters in remote and colder climates, things the big players can do with economies of scale beyond buying things cheaply, they can customise the entire datacenter. Microsoft has experimented with underwater datacenters[4] and modular containerised datacenter extensions[5] which could be datacenters no human needs to be near to work on, or which could be dropped off somewhere with cheap land and power and internet, and picked up three years later and retired from use, or etc. Ideas which are not FedEx's core competency and need large scale and software clustering on top.
While FedEx would be hiring ordinary IT employees to work in a standard datacenter in cheap business park - not very enticing - Amazon could be hiring datacenter workers to work with Amazon's undersea cabling connecting their worldwide datacenters; more enticing work for skilled employees.
Google has been known/rumoured to migrate heavy batch processing workloads around the planet, following the day/night cycles to take advantage of regional cheaper night-rate electricity all the time. Something which reduces their energy costs but which FedEx may not be big enough to do.
[1] https://engineering.fb.com/2016/03/09/data-center-engineerin...
[2] https://engineering.fb.com/2019/03/14/data-center-engineerin...
[3] http://www.500eco.com/exhibits/microsoft-roofless-datacenter...
[4] https://news.microsoft.com/innovation-stories/project-natick...
[5] https://www.datacenterdynamics.com/en/news/microsoft-develop...
But The Facebook decided to make informatics their core competency, to the point of building their own servers with 12 volts running to the rack, same as Google before them.
There were surely industrial companies in 01922 who decided that management wasn't their core competency (though they used different words), and if they needed help with management they'd contract out to management specialists like Taylor or Gilbreth. They met the same fate that will meet companies today that decide that informatics isn't their core competency.
Innovations are more likely to happen if it is someones priority to fix a certain thing. I think they hope that savings from innovation and better methods at what ever company they hire out to are passed onto them. It is naive if they are unable to change clouds though that they will see these savings and as long as one relies on vendor specific features on is in that position.
So where are all these mainframe engineers going to go work? Our company has 3 admins for our two mainframes, and these guys while expert level admins for a mainframe, have trouble with Linux and Windows. Same with the developers writing code for the systems. When all you've worked on is a mainframe, then everything looks like a batch cycle...
That's deep and 100% makes sense, I can see how the cloud is both of these "things."
This is one of the biggest reasons many jobs are so miserable. You want to be in the job at the beginning of the"invest" decision. Unfortunately most people get little say in when they join because the information about this is kept internally to the company.
Well, six months later sales and profits were still way down and the new CEO was catching a lot of heat. He began to panic but then he remembered the envelopes. He went to his drawer and took out the first envelope. The message read, “Blame your predecessor.” The new CEO called a press conference and explained that the previous CEO had left him with a real mess and it was taking a bit longer to clean it up than expected, but everything was on the right track. Satisfied with his comments, the press – and Wall Street – responded positively.
Another year went by and the company continued to struggle. Having learned from his previous experience, the CEO quickly opened the second envelope. The message read, “Reorganize.” So he fired key people, consolidated divisions and cut costs everywhere he could. This he did and Wall Street, and the press, applauded his efforts.
Another year passed and the company was still short on sales and profits. The CEO would have to figure out how to get through another tough earnings call. The CEO went to his office, closed the door and opened the third envelope. The message began, “Prepare three envelopes...” "
In exchange I offer two relevant quotes from my quote file:
> The empire long united must divide, long divided must unite; this is how it has always been. (Luo Guanzhong)
> When cuffs disappeared from men’s trousers, fashion designers gave interviews explaining that the cuff was archaic and ill-suited to contemporary living. It collected dust, contributed nothing. When the trouser cuff returned, did it collect less dust and begin at last to make a contribution? Probably no fashion designer would argue the point; but the question never came up. Designers got rid of the cuff because there aren’t many options for making trousers different. They restored it for the same reason. (Ralph Caplan)
Do you think it's "don't change"? Cause that gives right up on progress anyway. So good luck on getting large shareholders to give you the reins with that message. They're looking for ROI, not the status quo - if you don't evolve, your competitor will, barring monopolies and such. And even there... Microsoft of today is much less dominant than the MS of 25 years ago, and arguably could be worth a lot more had they made better moves. If that's not a compelling example, maybe check out Sears...
Maybe most people in these positions know that change doesn't guarantee improvement, but they know that sitting still is the same as just waiting to be defeated. So maybe there's something less-than-stupid about these "short-sighted" "illusory" changes.
But maybe if you want to be a wise executive, the key is to recognize that change might not just fail to improve your position - it might actually actively harm it. So the good executive is the one who chooses to try to change things according to reasonable calculations about both potential upside benefit and downside risk...
I'm curious whether the folks claiming that have any data center ops experience.
Because, personally, I'd rather retire than deal with Dell, HP, Cisco, fibers, cooling issues, physical security, hardware failling... And that's just the hardware. Then you still need to pay VMWare for a decent virtualization platform, monitoring tools, etc.... Seriously, no amount of money would make me work in a DC again.
I believe companies selling bare metal as a service are a happy compromise of cost and convenience, though.
If the AC breaks at 3am it neds to be fixed. It doesn't matter if you have your own HVAC people on sight 24x7, your own people on call to service it, a local HVAC service to come in, or you outsource the entire operations and so you have no idea how that is handled. In the end the important part of this story is that whatever you are doing with the AC continues to work. Different operations demand different levels of service (I doubt that anyone keeps HVAC techs on staff 24x7, but if the AC is that critical it is mandatory). The only case where the CEO is up at 3am is if the CEO is the owner of the local HVAC service company, not the CEO of the building with the problem.
Once you realize that to management the cost is outsourced no matter what the only question is do it with your own people and HR, or outsource it. There are pros and cons to both approaches, but for most companies it isn't their business and so the only reason to do it in house is they can't trust any company they hire.
The downside for saving the costs is that you're losing control with every step taken away: as soon as you go into a datacenter of any kind, you simply cannot call up a HVAC company and offer them 100k in hard cash if they're showing up in the next 60 minutes and fix the issue. With a colo DC you can usually go and show up there to see if the HVAC, UPS and other systems are appropriate to your needs, but with one of the big cloud providers you have to trust their word that they are doing stuff correctly.
Now I'm deeply confused. Any company hired either has a profit margin (plus enough to fund an "Oh shit" fund in case times turn bad) or will not stick around longer than a few years. At which case why not just hire people directly and cut out the other company's profit margin? Assuming you hire similar people at the same rate, using your own existing and already-paid-for HR, how is that not cheaper?
In some cases you can even get a discount. Utilities are. Big customer of tree trimming, the companies doing that work can give a great deal because the utility doesn't care that they take a week off after a storm for high profit margin consumer trimming.
Larger hotels often have dedicated staff for things like HVAC, etc, because the importance of getting things fixed quick if possible is worth the cost of having someone onsite/available.
And you see similar things with colleges, etc; they often have a maintenance deportment that can be pretty large (though no doubt they've spun it off and brought it back in-house for the same "change is progress" reasons).
There is a cost to taking on things that aren't part of your core business too.
Similarly, Google doesn't ship enough stuff worldwide to justify drivers, insurance, trucks, jets, etc. - Fedex has the size and scale to make every package a couple cents cheaper, so it's just not worth it for Google.
The only other argument I can think of is the challenge of keeping every plate spinning, in good times and in bad. This is where your point of having a cost to take on something outside your core business comes in, but we seem to be in an era of mega-corporations - I'd expect lots of companies to snake tendrils into whatever will save them a fraction of a cent every time they have to do something.
Also, that's just compute. What about data? Sure cloud accepts your data cheaply, but they also charge you for egress of that data. Yes you should have your data in more than one location, but if you depend on just cloud then you need it in different AZ which costs even more money to keep in sync and available for training runs.
I think for simple workloads and renting compute for a startup, cloud definitely makes sense. But the moment you try to do some serious compute for ML workloads, good luck and hope you have deep pockets.
The part that’s always missing with these rent vs buy analyses on HN for some reason is that it’s totally ignoring the opex cost of operating your own hardware which is going to be non 0. Sure, it won’t be quite as expensive (no profit margin) but it’s not an order of magnitude. Additionally, most companies don’t run the HW 24/7 and, if they do, it’s not a level of people they want to hire to support said operations. Not just running it, but you have to invest and grow something that’s not a core competency to get economies of multiple teams loading up the HW.
If the next revolution in cloud comes in to cause companies to onsite the HW again, it’ll look like making it super easy to take spare compute and spare storage from existing companies and resell it on an open market in an easy way. Even still, I think the operational challenges of keeping all that up and running and being utilized at as close to 100% as possible and not focusing on your core business problem will be difficult because you won’t be able to compete with engineering companies that have a core competency in that space.
Effectively hiring, retaining, evaluating and rewarding competent staff is hard. Even at a big company the datacenter can be a really small world, which makes it hard for your best employees to grow. Things are especially hard when you don't have a tech brand to rely on for your recruiting, and the staff's expertise is far outside the company's core business, making it harder to evaluate who's good at anything.
Compute costs generally drop over time. Do you have any data points to confirm egress will soon go to zero?
I'm not sure why you think that. AWS hasn't budged on their egress pricing for a decade (except the recent free tier expansion), despite the underlying costs dropping dramatically. GCP and Azure have similar prices.
Fact is, egress pricing is a moat. Cloud providers want to incentivize bringing data in (ingress is always free) and incentivize using it (intra-DC networking is free), but disincentivize bringing it out. If your data is stuck in AWS, that means your computation is stuck in AWS too.
I think we’re going to put real pressure for traditional object storage rates to come down. Since Cloudflare‘a entire MO is running the network with zero egress. As we expand our cloud platform it seems inevitable that you will at least have a strong zero egress choice and if we do a good job Amazon et all will inevitably be forced to get rid of egress. Matthew Prince laid out a strong case for why either scenario is good for us in a recent investor day presentation (either we cannibalize S3’s business and R2 becomes a massive profit machine for us because they refuse to budge on egress or Amazon drops egress which is an even larger opportunity for us).
Products like Cache Reserve help you migrate your data out of AWS transparently from any service (not just S3) - you just pay the egress penalty once per file.
Anyway. I’m not saying it’s going to disappear tomorrow but I find it hard to believe it’ll last another ten years.
Early in my career I worked at a company and we had a DC onsite. I remember the months long project to spec, purchase and migrate to a new, more powerful DB server. How much that costed in people-hours, I have no idea. I upgraded to a better DB a couple months ago by clicking a button...
Don't even get me started with ordering more SANs when we ran out of storage or the time a hurricane was coming and we had to prepare to fail over to another DC.
The moment the model gets to be bigger than the size of any one GPU's VRAM, the higher by orders of magnitude of difficulty in the process of training that model.
Here's the list of ingredients for Excedrin Migraine:
Active Ingredients: Acetaminophen - 250 mg (Pain reliever), Aspirin (NSAID) - 250 mg (Pain reliever), Caffeine - 65 mg (Pain reliever aid) Inactive Ingredients: benzoic acid, carnauba wax, FD&C blue #1, hydroxypropylcellulose, hypromellose, light mineral oil, microcrystalline cellulose, polysorbate 20, povidone, propylene glycol, simethicone emulsion, sorbitan monolaurate, stearic acid, titanium dioxide
This is the list of symptoms that Excedrin Migraine claims to treat:
- migraines
And now here's the ingredients for Excedrin Extra Strength:
Active Ingredients: Acetaminophen - 250 mg, Aspirin (NSAID) - 250 mg, Caffeine - 65 mg Inactive Ingredients: benzoic acid, carnauba wax, FD&C blue #1, hydroxypropylcellulose, hypromellose, light mineral oil, microcrystalline cellulose, polysorbate 20, povidone, propylene glycol, simethicone emulsion, sorbitan monolaurate, stearic acid, titanium dioxide
This is the list of symptoms that Excedrin Extra Strength claims to treat:
- headache - toothache - a cold - arthritis - premenstrual & menstraul cramps - muscular aches
And while SOME places have normalized the prices between the two, they can be often found on shelves at two different price points.
If you are sophisticated enough to engage an ODM, build your own facilities, and put hvac and electricians on 24-hour payroll, go on-prem. Otherwise, cloud all the way.
That doesn't sound like it will save much money, honestly.
But i don't think they made even a initial load-/stresstest.
Unpack it, trow it into the rack, no checking of internal plug's just nothing...pretty sure about that.
I am beeing smug about not testing your hardware as you do it with software....shitty testing is shitty testing, counts for software hardware firmware and everything between. Even for your diesel generator ;-)
Load and simulated power failure tests all passed.
Then some time later there was a total power cut and that's when they realised the generator had an electric start wired to the mains supply.
> had an electric start wired to the mains supply.
But that's a good one, humans being humans...but it worked every time before today ;))
You can test your stuff and be still profitable henzer aws etc would make no money otherwise....you know they test their server much more (sometimes weeks/month)
And no, operators don’t disassemble to perform QC. And no, I could hire an entire division of people buying servers at Best Buy, and disassembling them, and stress testing them, and all of that overhead including the fuel to drive to the store would still clock in under cloud’s profit margin depending on what you’re doing.
You’re of course entitled to develop your cloud opinion from that experience. That’s like finding a stain in a new car and swearing off internal combustion as a useful technology, though, without any awareness of how often new cars are defective.
You can point at Google and speak in abstracts but it doesn’t address the point being made, nor that your rationale for your extreme position on cloud isn’t as firm as you thought it was. Is Dropbox the only time you’ve worked with hardware? I’m genuinely asking because manufacturing defects can top 5% of incoming kit depending on who you’re dealing with. Google knew that when they built Platform A. The lie of cloud is that dismissing those problems is worth the margin (it ain’t; you send it back, make them refire the omelette, and eat the toast you baked into your capacity plan while you wait).
Do you see how these costs all start to add up?
Is that like too much these days for companies that owb fleets of cars? is opening a server harder than checking whars wrong with a car? like a cable comes loose and that's gane over?
So you don't even test the car's, you just expect that the tire pressure is correct, tank is full?
Expect that something "just" works is exactly why pilots have checklist's.
Expectations are the main point for disappointments, you would never do that with software right?
Not true the point was you pay for it (cloud), or you do it yourself (but then do it right, and not like a amateur who build's his first "gaming-pc").
And if you do it yourself you can still be very much competitive vs cloud.
Again, still avoiding the point, but oddly enough proving the point. You assume everyone isn't an amateur and knows how to build and maintain server hardware. Furthermore, because the market doesn't have enough talent to support all of the companies that exist, consolidating this to a few vendors who do have the expertise is what makes sense (economies of scale) and is what the market already decided.
Please read, that was my comment:
>>Not true the point was you pay for it (cloud), or you do it yourself
>You assume everyone isn't an amateur and knows how to build and maintain server hardware.
Yes that i assume, correct. Otherwise i would not call it "maintaining", is a amateur maintaining your car? Your software? If you have just amateur's handling your hardware it's probably better to pay a cloud-provider or pay a integrator todo that.
Hilarious you used this as an analogy since software development shops are notorious for cutting corners when it comes to QA.
You facetiously implied that every company fully tests software before it gets to production. Oh boy, do I have news for you...
Note the word "fully" as the variations of what gets tested is so broad, I don't even know where to start to explain this to you.
>Oh boy, do I have news for you...
Nah it's ok, just happy that i have colleges with a much better mindset and risk-management understanding.
And i stop here, since you try to change what i really wrote.
There is a smooth curve between cloud and dedicated DCs, which has various levels of managed servers, co-location, and managed DCs. (A managed DC can be a secure room in a DC "complex" that shares all the heavy infrastructure of DCs.)
Primarily, the FedEx managers are committing the company long-term to Oracle/Microsoft platforms. Probably mostly to benefit their own careers.
Outsourcing hosting and management of DCs would have been something different, and probably healthier for FedEx and the industry.
Like most startups, our management of the data center was pretty scrappy. Our CEO liked that kind of stuff, and we had a couple of network engineers that could be on call to fix overnight issues. It definitely wasn't a burden at the 50 employees size of company (and that includes field techs that actually installed fiber, dragged cable under the street, etc.)
We actually had some Linux servers in the datacenter. I don't know why, to be completely honest.
So overall my thought is that maybe use the cloud for your 1 person startup, but sometimes you need a datacenter only and it's not really rocket science. You're going to have downtime while someone drives to the datacenter. You're going to have downtime when us-east-1 explodes, too. To me, it's a wash.
I imagine the entire sentiment of the comments is because FedEx is one that really should be sophisticated enough.
AWS almost certainly gets batches of bad hardware too. And if your services are running on the bad hardware, you can't have a peek inside and find the iron filings. For servers, this is probably not too bad, there used to be articles about dealing with less enthusiastic ec2 vms since a long time, and if you experience that, you'd find a way. AWS has enough capacity that you can probably get vms running on a different batch of hardware somehow. With owned hardaware, if it was your first order of important database servers and they're all dodgy, that's a pickle; HPE probably has quick support? once you realize it's their hardware.
If your cloud provider's network is dodgy though, you get to diagnose that as a blackbox which is lots of fun. Would have loved to have access to router statistics.
There's a lot of stuff in betwren AWS and on-prem/owned datacenter, too.
You bet it does! But as the AWS customer you'd never notice because some poor ops dude in AWS-land gets to call up the vendor and bitch at them instead of you. It ain't your problem!
Data center ops is ultimately a supply chain management problem, but most people don't treat it as such. That was my primary learning from doing data center ops at a few different companies. If you get the supply chain management right, and are technically competent, there can be a lot to recommend running your own data centers.
This is what I do. I rent bare metal from Hetzner and OVH. I also have some hosting hardware right at my place. It saves me a ton of money and no I do not spend any meaningful time to administer. All done by a couple of shell scripts. I can re-create fresh service from the backup on a clean rented machine in no time.
As for cloud - if I need to run some simulation once a month on some bazillion core computer then sure. Cloud makes much sense in this particular case. I am sure there are other cases that can be cost effective. Bot for the average business I believe cloud is a waste of resources and money.
If I was a manager at FedEx I'd definitely spend some resources to DYI and send some of those millions over to my direction instead.
I enjoyed server admin, "back in the day", when your servers were pets and not cattle. But of course we have to make tech just as expendable as our workers, business school demands it! What if your pet server gets hit by a digital bus?!
Of course, the newest idea is for creating and destroying machines automatically... that outside of the could is quite pointless but people want it anyway. I imagine seeing all that orchestration working must be even nicer than a machine autoconfiguring, but I am yet to see a place where it just works.
The promise is to be able to pay up front for a rack that will function as a highly capable VM, storage, and/or compute host, without any of the overhead that Dell, HP, and IBM bring. Just plug it in and start giving it workloads to do. All config can be done through the web-based management console or via the API, just like AWS.
This isn’t really a meaningful analysis though. It’s just “when you do things in house there are things you have to do”.
It’s like saying, “I would rather retire than clean the toilets, restock the toilet paper, etc” in a discussion about whether to outsource your bathroom maintenance. Doesn’t tell you what’s cost effective.
If your compute installation is big enough that payroll is a small fraction of the operating cost, then it's way cheaper than cloud. (that payroll has to include people who actually know how to build and run a huge compute installation)
The problem is that people come in integer units, you need a bunch of them to cover a bunch of different areas of expertise, and the particular ones you need are expensive. If you've got $1M worth of computers, you're almost certainly better off scrapping them and going to cloud, although the folks you're currently paying to run them might disagree. If you have $100M+ worth of machines it's a whole different ballgame; I'm not sure where the exact crossover is.
Note - that's assuming a single data center, and that you're big enough to build your own data center instead of renting colo space. If you need your machines to be geographically dispersed, you'll need to be even bigger before it's cheaper than cloud, and I'm not sure whether you'll ever hit crossover if you're renting colo space.
All of that can be handled by your colocation facility. In most cases you won't ever reach the scale where building your own DC makes sense.
> Then you still need to pay VMWare for a decent virtualization platform
Should still be cheaper than paying the AWS premium including for bandwidth, not to mention that you don't always need virtualization. If all you need the bare-metal for is a handful of machines to do a very specific task that's too expensive on AWS then running directly on the metal is an option (and leave on AWS the stuff that does require the convenience of virtualization).
> I believe companies selling bare metal as a service are a happy compromise of cost and convenience, though.
Agreed. Most companies shouldn't ever deal with hardware directly - just rent it from a provider and let them do the maintenance.
One of the visible signs of an unrecognized (and therefore, unresolved) dilemma is oscillation between two poles.
I got this from the late Eli Goldratt and the problem-solving tools he created. One of those tools is the "Evaporating Cloud," which is a way to visualize a dilemma of some kind.
I'd suggest that there is an unresolved dilemma here: Should we rent or own data centers?
Over a period of years or decades, you can watch these things flip-flop between two extremes.
It's interesting to me that this dilemma is kind of like a specialization (in OO terminology) of a more generic dilemma, which might be something like: "In general, do we want to own or rent the things we need to run our business?"
If your turn your head a bit, close your eyes a bit and squint, you can see how a dilemma like this can apply to things like "What should be our policy regarding employees vs. contractors? Should we try to hire and retain over a long period of time, or should we rent the people we need to get the job done and release them as soon as we are done with them?"
The overall point is that whenever you see flip-flopping between two poles, think "Unrecognized dilemma."
Sorry if this is vague.
If not for the rent seeking behavior of AWS and its cohort, the answer to ”own or rent” would be clear. The answer is “yes”.
Running a data center means you have your own sheep, which means you need shepherds. Shepherds are very useful to have when you have a question about sheep, especially when those questions are about how to manage or use sheep to best effect.
An approachable shepherd can save the rest of the company a lot of money on missteps and bad assumptions, but if you start getting rid of all the sheep, the shepherd will leave too.
We should be using cloud providers for DR, and for regional load balancing. But the company should be maintaining at least one data center of their own, in the same time zones as most of their developers.
I mostly blame Dell and IBM for this. IBM experimented with making server rooms easier to maintain 15-20 years ago and didn’t make it stick. Others ran with some of those ideas. Dell… I don’t know what Dell has done but I know nobody has been writing about it, so from a visibility standpoint they have done nothing.
If/when someone makes it easier (reduced labor) to manage your own servers, the pendulum will swing back.
It wouldn't take the most creative exec in the world to make a plausible looking case for changing in either direction even based on the same numbers. A bit of wishful thinking about which costs are included or excluded is probably more than enough.
Most of them were renting most of the infra anyway:
- Co-location (physical space rented)
- Managed service to keep the lights on (power, back-up generators)
- Leased hardware that gets replaced every 3 years
- Managed service for switch, firewall, etc. monitoring
- ISP, back-up generators, etc.
> but that will likely be many, many years down the road
They're going to explain to their future bosses that buying land, building a huge building, buying servers, figuring out cooling, setting up redundant ISPs, etc. etc. is somehow going to be smarter? Explain that one to me?
If you are big enough to have your own datacenter, you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc. plus funding Bezos' trips to space.
There's a justifiable niche where you're too small to justify running your own server/below the need of one full-time IT staff where the cloud makes sense, and startups temporarily benefit from the ability to rapidly scale on cloud platforms. But any Fortune 500 transitioning to the cloud is literally just taking their money and burning it, because the alleged cost savings of efficiencies of scale are being completely absorbed by the cloud provider's profit margins. And they're still going to end up paying their own IT staff to handle the cloud management in addition to (by proxy) paying the cloud provider's IT staff to manage the hardware.
So, then why even provide the distinction of rent vs own if thats the case?
> you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc.
And you're also sharing the costs with other people. Clearly FedEx is saving $400M/year while also funding Bezo's trips to space, no?
> But any Fortune 500 transitioning to the cloud is literally just taking their money and burning it, because the alleged cost savings of efficiencies of scale are being completely absorbed by the cloud provider's profit margins.
This is literally not the case without the details so stop speculating. Every F500 should (and probably is) be doing a rent vs own calculation and determining the TCO, and then making the decision from there. It's not unilaterally the case, it has to be assessed.
I wonder how much they'd save if they rebuilt their infrastructure with own DCs. There are huge savings in decommissioning mainframes, switching to free software and automating more. At the scale of Fedex, I don't think they'll spend much more on hardware and ops than AWS.
Big companies need this too. Some team wants to spin up some new service to try out some idea? In cloud-land they push a button. In "rack & stack" land they have to wait for Ops to purchase the hardware and provision it.
Cloud makes it cheap and easy to try out new stuff.
All transactions, no matter if in the cloud or on prem, go towards another company’s profit margin.
The cloud just offers an additional middleman that also gets a profit margin.
You get your support in one place instead of Hodge pogge of ... yeah we rent server from X but software that runs on it is from Y and in reality provider Z is support so now C has to agree for physical access to the server and now align stars to get all 4 working together instead of shifting blame around to fix anything.
And you hope the Amazon employee makes decisions that are favorable to your business, even though they do not care about it.
The problem with building out a new state-of-the-art datacenter - or several of them - is the enormous capital expenditure you've just put on your company's books, not to mention the operating expenditure of all the people that will be required to run it. Yes, it's true that as time goes on, you can claim some tax advantages in the form of depreciation, etc. on some components of this new huge outlay, but at the end of the day, when the company misses quarterly or yearly expectations from Wall Street and the stock price takes a 10% hit, the "blame" gets squarely laid at your feet - you are, after all, "the problem". You spent a shitload of money provisioning future resources for the company's (expected) growth.
Meanwhile, in Cloud Cuckoo-Cuckoo Land, your rival has migrated all or nearly all existing infrastructure to the cloud, thus including a significant op-ex, but not nearly as large as the enormous cap-ex you've just incurred. They're hailed as a hero. A goddamned visionary! Look at all that money they're going to save the company! Nevermind the fact that they explicitly instructed the IT department's head to provision only the necessary resources for current operations, after all, the "promise" of the cloud is that you can just spin up whatever you need in a few minutes, anyway. And besides, the cloud deployment of all the company's servers and the necessary expansion won't incur a significant turning of heads until long after our visionary executive has jumped ship to another company for more pay, a corner office, and a better stock compensation package. Best of all, he say that he saved the company $XX millions of dollars over your plan, and it can be said legitimately, even though it is, of course, inaccurate.
If you're huge, it's never cheaper to farm out the administration of your critical infrastructure to qualified experts. But because of the dominance of the Quarterly Report Cycle in modern business, this gets swept off by the wayside as "outdated thinking".
I say this, by the way, as an IT professional building out cloud solutions for companies. For a lot of small-to-medium sized businesses, and startups especially, the cloud makes sense. If someone at Federal Express thinks the cloud makes sense, they're looking out for themselves, not the company.
It's really just like the utilities these days, 98% people will not dig their own well and install some sewage system and buy a generator, but big companies can still opt to have them on-premise installed, even though some of those are just backup systems.
there is no return for the cloud for 98% of us.
Generating solar for local use saves the cost of transport.
High-voltage line losses can be as low as 3% per 1,000 km. That should be easily offset by the better location and better angle/tracking.
You mean there are fewer companies making large investments than smaller ones?
As someone who's been building an off-grid solar system this is largely the case. I've spent thousands on panels and batteries, my electricity bill is only a few hundred a month but we have frequent outages during storm season. It's definitely more of a feel good and control thing. If I consider the cost of running the electricity to our remote location, I'd probably barely break even in 10 years if I'm lucky.
I worked in a building during rolling blackouts in California. The Tesla Powerwall on the building was "screaming" (ultrasonic harmonics from piezo components) 24/7. The security guy actually came to get me to ask if the thing was going to explode.
Clearly, the building owner was load shifting and making quite a bit of money from it.
They shouldn't, though. If powerwalls are that great, it should be more cost effective to install giant banks of them in cheap warehouses outside major metros since you'll have economies of scale on installation and maintenance. Utilities are sorta getting into this game, but it's more common at the individual level, despite being more expensive.
This works only if the electrical grid has the ability to consume back your stored power. Most electrical grids in the US would have problems doing that.
For example, UCSD has their own generating facility; however, SDG&E was sufficiently backward that UCSD was unwilling to go through the grief necessary to put energy back into the grid. Therefore, UCSD only does load shifting or disconnects the campus from the grid during rolling blackouts.
Because of the poor energy grid transport, storage batteries make the most sense when you can consume the stored energy locally to minimize your grid consumption during high energy prices--ie. you have a high-rise office with lots of air conditioning or a manufacturing facility.
Luckily, it does work for most of the US, especially desert/arid areas.
> but I see fewer companies building collectors in places with cheap land and ample sun.
They are plentiful in Turkey and some southern EU countries like Greece/Croatia, I believe. Not sure about deployment in the US.
(The wise thing to do to mitigate financial impact may actually be starting the reintegration process right now.)
Sometimes tech debt can be used instead of financial debt!
Either way, I think you're spot on with the "training" part. FedEx was one of the first companies to raise significant capital and pursue a business dependent on technology. It also treated its people very well so relatively few left since the 70's. I think they just didn't train the next generation enough and they realized that those skills are disappearing rapidly because no college grads want to learn old & boring stuff and everyone who does know it costs a lot to keep (FedEx still heavily relied on COBOL when I left ~5yrs ago).
Of course they should make some effort to ensure there is competition. That means they are careful to ensure more than one provider exists even if it means taking a slightly higher priced option. Also contracts end early if the company is bought out. (that is if AWS, buys google cloud - see above about ensuring there are competitors in the market)
My guess is, we'll rather see some consolidation, like in every other segment. Which also means considerable expenses for the remaining players, who will experience increasing need to regain some by pricing. As theses players are sharing roughly the same boat, this will show quite naturally some characteristics of an oligopoly. At the same time, self-managed infrastructure will have become a rarity and costs of reintegration are prohibitive, which should allow for some elasticity in market prices. Also, any investments towards reintegration won't show results during the turn of current management, minimizing chances for this to happen, yet again. (This is not a level playfield anymore.)
With a good design, there is always that implicit threat that they could move back to on-prem with little effort.
How costly is it to maintain that level of talent? Do you think top server admin talent is screaming to work at FedEx or Amazon?
There is no grand plan, there's just evolution.
One of the things that bother me about AWS is that I still need to manage their risk of a data center going down by using multiple availability zones and managing the complexity of extra resilience. There is a conflict of interest: AWS has a perverse incentive in keeping a single AZ robust and unless there is a natural calamity, it should not go down.
I thought one of the major reasons of going to cloud is that you need not manage risk and offloading it from on-prem.
Now, they might find that software is a differentiator for them, but that's different.
HNers need to get their heads around this: the economies of the could represent a fundamental, secular shift.
Imagine if Fedex designed and made their own delivery vehicles. Then they 'outsourced' that to Ford/GM. You might say 'look at Ford's amazing margins, look at the money being left on the table' - but in reality, it's still more cost effective to 'buy vehicles' the to 'DIY' them.
In the 'long run' those Azure/AWS margins will erode - and/or - the long tail will creep up on them. Basically data centers offering less, for less, and companies realizing they don't need the complexity of AWS do do basic things.
What that will look like is hard to say.
With hardware it meant 'pivot to Asia'.
Will this happen again? Chinese companies creating large data centres in the US with minimal staffing, monitored and operated out of China?
If we were not in a giant geostrategic kerfluffle with them, then yes, I would say that is the future. But given security issues etc. it's probably not.
Can India do that? Maybe.
By contrast, generally speaking, cloud providers are in an excellent position to use price discrimination to extract the entire surplus value of every transaction. And the flexibility and reliability provided by FedEx will to a significant extent simply be that of their computers.
(As chrisseaton pointed out, Boeing is in a cloudier position.)
Especially to the extent that special services are not leveraged, then there are cheaper alternatives. A lot of IT is just 'instances' not 'fancy cloud queues'.
AWS pricing is very transparent and people can make the decisions they want. Generally speaking, AWS does not price discriminate on the basis of 'business model' and their margins are meaningless with respect to the overall business efforts of a Big Co.
I mean - unless you are doing big AI crunching, or 'free hosting content' - then frankly the AWS bill is not going to be a huge line item relative to overall operating costs.
And this is where people in my age range (20-30) will get to swoop in if we invest the next 10-20 years learning about and tinkering with server hardware.
I highly recommend it for everyone.
https://www.reddit.com/r/AskHistorians/comments/vqr30e/jack_...
"Jack Welch extracted record profits from GE for 20 years, but left it a hollowed-out "pile of shit," according to his successor. What exactly did Welch do that was so damaging, and how did he get away with it for so long?"
from the post reply, in case it's not available from the URL:
alecsliu · 2 days ago · edited 2 days ago Gold2Eureka!Bravo!Today I Learned
Welch took over a GE that was at the time, a major company. At the time, he viewed GE as bloated and needing to change. While he might've been right about that, the approach he took was perhaps less ideal.
One of the worst things he helped make commonplace among American companies is the concept of stack ranking. The way it worked was like this: people were divided into three groups: A, B and C. A's were the top performers who needed to be rewarded generously, B were adequate performers who should be allowed to stay, and then C were those who needed to be fired.
So far so good right? Well, not exactly. For Welch, these three buckets could be separated into the top 20%, the middle 70%, and the bottom 10% (20/70/10). Based on the above points, the bottom 10% would thus need to be fired annually.
In the short term, this helped in making the company lean and look more productive, increasing the bottom-line and portraying an image of success. However, the long term consequences of such a change was cultural degradation and the introduction of new bloat and waste (running all of those performance reviews and firing and rehiring so many people takes a lot of money.) Consider the case of a perfectly adequate team: the entire team has achieved their targets and has contributed to the company as per their job description. The issue? In stack ranking, 10% of this team would still have to be fired even though everyone did their job. Unsurprisingly, the introduction of this competitive atmosphere where it isn't enough to succeed, one must be better than their peers, results in backstabbing, competition, and a host of issues which eventually weaken a company's competitive edge.
This was only a part of Welch's general treatment of workers as numbers rather than humans. Aggressive cost-cutting, offshoring, etc. were the norm under Welch's regime and he would destroy entire divisions. This again, was great in the short-term but bad in the long term. Welch clearly had a dim view of company culture and believed it to be unimportant.
The other major issue of Welch's was the acquisition of hundreds and hundreds of businesses, as part of Welch's goal of acquiring his way to the top. On the surface, this is what Welch did; on paper, he didn't destroy the main profit makers of GE so much as create new ones, primarily in the form of its financial arm.
The result was that this allowed GE to play with the numbers in a way that allowed him to make sure that GE was always meeting targets set by Wall Street; in order to generate the right earnings, simply buy or sell certain assets, write-off others, etc. and when your company is an acquisition machine, it's not too hard to find the right numbers. This process helped expand GE into a mega-conglomerate but again, ultimately left the company in a weaker than expected state. In particular, on paper the core business of GE became its financial arm, as that was where all the funny business with the numbers was happening (nothing explicitly illegal though). Ignoring the damage the Welch did to GE's profit centers through his horrible business practices, this was a huge part of why GE declined so rapidly in the years following. GE's valuation was based on an inaccurate picture of its profits and value, so when the truth started coming out (especially with the Great Recession collapsing financial services profits), GE was quick to follow.
As for why Welch was able to get away with it all? Well, the answer was because he was delivering. He hit the earnings targets, he made the board and shareholders very happy, by all accounts GE was the paragon of success and everything was going right. What Welch did was unprecedented, and it's hard to really understate that. For all of his faults, he had America tricked into believing that what he could do was unique and that he could avoid the realities of the economy and cyclical markets, that no matter what was going on, GE was special. Welch died a rich, rich man and many, many people profited greatly off of GE during its 20 year bull-run.
Edit: My off-the-cuff writing is always horrible wrt to grammar and structure so I'll probably edit it later haha