Actually the cost savings hinge on the massive cost savings from moving to a commercial cloud, laying off workforce would save more, but would prefer to save a bunch by moving to cloud and keep the workforce for other things.
I see two possible explanations (both of which may apply):
(1) You're getting the cloud servers for a remarkable discount, which, as other commenters mentioned, may not last if you don't put in the engineering effort to be vendor-agnostic (though I'm not sure I agree it's as huge a cost as otherwise implied).
(2) You're grossly over-paying for your current environment. Given what I know about the purchasing habits over merely large (nowhere near the size of F50, mind you) companies and, more importantly, the pricing of "enterprise" hardware and support, this seems likely.
If the second is true, then I expect you could gain the same savings, if not more, by implementing your own "private cloud" by emulating the purchasing and ops habits of larger-scale startups or established tech companies (but not yet so huge as to be too custom) who run their own hardware.
You'd still have to retrain and put an end to manual configuration, but I expect that would ultimately be welcomed. Some might be able to keep their jobs almost as-is.
What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect. You'd probably have to hire someone who knows how to be frugal like that from an engineering/ops standpoint for architecture/purchasing and is willing to work for such a large company and isn't already employed by FAAMG.
1.) No specific discount, this is standard, public pricing on major commercial clouds, with some tricks applied like reserved instances, which is part of the reason I am not worried about low prices first and then ballooning pricing later,
2.) Our operations and maintenance and sustaining are in fact more expensive that anyone would reasonably expect, I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices, I've walked the data center floors and coded with the ops teams.
It is #2 that is really hitting us. I have personally moved several apps as a PoC and you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs and autoscaling.
I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. Thank you for the response.
I would be surprised if this isn't the vast majority of it, with the rest being the over-provisioning you mention later. Mostly, it's tough to gauge, because each effect is multiplicative.
I'll go so far as to say that, somewhere between 10 and 15 years ago, that any hardware support you've been paying to vendors can be considered as complete waste [1]. Worse, it's often a percentage (i.e. multiplier) of original purchase that goes one indefinitely, even though the value of that gear falls rapidly.
Also, (for that 90% you believe you can move to the cloud), if you have any proprietary hardware, you're paying an eye-watering multiple of its commodity equivalent (even considering additional labor, if any).
The above is easiest to observe in storage. There's not all that much mystery around how much the components cost and how reliable they are, so cost estimation for an equivalent commodity solution is easy enough. Today, even commodity/free no longer means "no frills" in terms of software features, but I expect that's less relevant when comparing to cloud providers, anyway.
> you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs
I very much would. Just remember you can smaller VMs yourself, cheaper than cloud providers, bringing your staff up to speed on more modern skills, so long as you don't fall into a vendor's trap.
VMWare's model where the software ends up cost is the same order of magnitude as the hardware is such a trap, but so is software that's free but requires expensive professional services to set up and maintain (OpenStack, historically, though I'm not sure if that's still true).
> and autoscaling.
This, actually, is the rare case where cloud providers can save really save money, if your duty cycle (for lack of a better term) is under 10% of so. Other comments in the thread mentioned this, as well.
Unfortunately, if you never autoscale to (approximately) zero, then you're grossly over-paying for that base load, which can wipe out those savings.
The "hybrid cloud" model addresses this, as well as avoiding all-or-nothing thinking.
Ultimately, if you're optimizing (not necessarily exclusively, of course) for cost and want it to last, you'll have to change the whole organization's way of thinking. Avoiding vendor lock-in is key, be it proprietary hardware, software, or cloud providers. Also, encourage good engineering practice/culture, without getting overly focused on the latest fashionable tool. (A trite example being saving on labor through automation but buggy automation causing an eye-watering AWS bill).
[1] There's some non-zero value to any actual replacement hardware, but that's negligible if considering what you actually got, net of any markup for being propreitary.
The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/co...
Let me guess... you're using kit from the vendor at the bottom of that list? :)
No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct.
The commentator your/this reply is to points out an interesting aspect:
> What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect.
This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types.
Well... the landscape is indeed changing, as clarified by the 9-year old article above.
So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor.
(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)
And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings.
If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses).
I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is.
Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing...
> I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally.
I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on.
There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here.
This was what I was getting at with the "the offer will be what the buying team wants to hear" - at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.)
Cannot keep writing as I must go, may continue later
I suspect it's this kind of over-reach that ends up scaring decision-makers off from any in-sourcing (for lack of a better term) solution. To me, the key is in sticking with commodity solutions.
Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. In the worst case, you create interdepedent vendor lock-in and indefinite delays. Is there an example of any large company, other than Google, who has mastered this?
I found it interesting that you would mention Backblaze since, even back then, I questioned (I think even here on HN) their decision to go with a custom chassis and the non-standard, low-performance technology of SATA port multipliers. Well over a year before that blog post, there was enough competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro, SuperMicro) that I was using them at a cash-strapped startup instead of used Dell MD1000s.
What we don't know, however, is if those backplanes were actually a commodity yet, at their scale. Maybe Backblaze couldn't actually get enough cheap SuperMicro chasses soon enough. I do recall some resellers charging what seemed like comically high markups, but maybe they were banking on a scarcity I was blissfully unaware of, at my low volume. Still, the fact that the option is conspiciously absent from their bar chart tends to make me suspect their summaries and conclusions, even if I always appreciate their raw data (e.g. disk stats).
To tie this all back to the original topic, sticking with commodity solutions, be they hardware or cloud, and re-training your staff on them will help them avoid the potential career dead-end of over-specialization.
Hmmm. Good point. (Maybe I scared OP away, heh.)
> Is there an example of any large company, other than Google, who has mastered this?
Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s (2015).
Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable].
Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.)
As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on https://about.gitlab.com/ (the one from Forrester) doesn't show GitHub :D
Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the amount of cheap iteration that can be done is restricted. There are probably similar cost-benefit disparities as well.
I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc. Admittedly this question is based on my own complete naivete and OP probably has at least some info on this sort of thing available or accessible.
I looked just glanced at that video, it looked familiar. What I'm reasonably confident of is that no meaningful definition of "from scratch" applies. AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just slightly tweaked (if at all).
Now, it's certainly possible that Quanta offers customization that SM doesn't, like deleting extra on-board parts, but I'd be surprised if they allow even something like a choice in BMC/IPMI platform. Certainly nothing as extreme as exotic power options.
My point being that, if Quanta suddenly decided to jack up the price, CloudFlare wouldn't be over a barrel due to a custom solution. Worst case, they'd just have to pay (presumably higher) SuperMicro prices. Best case, they find a different white-label vendor, because what they're looking for is a commodity, nothing special, readily available.
> Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior
That seemed true 2 years prior (2006) when high-port-count SAS expander chips were mainly available super-expensive standalone devices, presumably targeted at the "SAN" market. It took a while for manufacturers and then customers (other than me) to twig to the fact that one could attach a cheap SATA disk to each SAS port on an expander, thereby making it relevant to the lower-end/frugal market.
> I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively
Before I answer, I'll pick a nit.. I'm not actually convinced that this business needs truly massive scale.
Especially given the OP's mentioning somewhere in the thread that the greatest savings in a POC cloud migration were from a smaller VM with auto-scaling. That suggest that even the current environment is over-sized, and that can easily get worse as computers get more powerful for the same size/cost. Remember, it's a traditional F50 business, not a web-scale tech company.
> (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc
That said, the job of vendor selection is pretty easy. Since it's commodity, it's pretty much about price.
At times, that has meant that, with a steep enough discount, you could consider Dell (and I've heard HP, but I never had such a discount) for servers, just not for storage. Supermicro seems to have been low-markup all along, and there are always white-labels, though that may not make sense at low scale.
Networking gear isn't quite commoditized, but we're close, with open-source/SDN type platforms.
One thing I've learned is to be open to different vendors for different commodity parts. Sometimes this means considering actually doing some of ones own assembly work, such as mounting disks in arrays. The modern knee-jerk reaction is to reject anything involving that much of the customer's labor [1] on the grounds that it's not scalable or somehow categorically too expensive, but that just leads to an insidious form of lock-in (which is, at least, temporary and limited to the transaction at hand). The integrator may be giving you a great deal on the server but soaking you on the disks (or maybe they're getting soaked on the disks and merely passing it along).
[1] Which can be contracted out, though that's fraught with its own problems, not the least of which are cost inflation and, perhaps more importantly, less "invemstment" in the quality of the final product. Really, though, I think the amount of labor is grossly over-estimated, especially working in anything approaching assembly line fashion.
And realistically, you need most of those people to maintain and configure your cloud service.
It's hard to imagine someone Fortune 50 scale gaining any cost efficiency from going to a cloud service, is this a temporary pricing promotion you're looking to take advantage of or have you done all your homework on the savings? How confident are you on what your cloud expenses will look like?
Google, AWS, Azure would all love a f50 client and I don’t think they’d care too much about breaking even on the costs of the service or even losing some money because of the amount of social capital they earn from the “partnership”.
There are a lot of benefits to migrating to the cloud, in my experience, cost isn’t one of them. Unless of course you are mucking around with accounting tricks by cutting employees and moving that cost to services or cost of goods sold, I’m not a expert in this, but it works and will save you money because you can control these expenses more easily and because of how payroll vs services are accounted for.
This is one of the other risks of moving to a cloud provider: You add a new dependency on a single company. If you don't have a multi-cloud strategy that allows you to force Google, Amazon, and Microsoft to fight for your business each time you expand, you're going to eventually end up paying top dollar.
That’s one of the things tools like Terraform, Kubernetes or Mesos were designed for.
If the lowest common denominator is K8S/Mesos, while not using any of the cloud services, you’re doing it wrong.
However, you do bring up an important point, which is a criticism I've always had for the term "infrastructure as code": Unless the underlying hardware is identical in performance, that code isn't really describing/commanding anything deterministic.
This is especially true for databases or anything I/O-sensitive like message queues or stream processing.
It would certainly make "cloud agnosticism" a much less trivial exercise.
At the risk of sounding ad-hominem, I generally find that it is software engineers whose experience is primarily at higher levels of abstraction who under-estimate how much it matter (or, conversely, over-estimate how much software or "code" can just solve such problems).
See also https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
Some cloud providers will be better than others. AWS is still very expensive for I/O intensive workloads. The tendency to revert to mainframe area commitments because a vendor provides a short term discount without factoring in true cost required to maintain the same QoS provided by on-prem is common for executives.
A new layer of abstraction is a major benefit of the cloud.
Have you already virtualized your own server and applications in the DC you have?