81% of IT teams directed to reduce or halt cloud spending
venturebeat.com
venturebeat.com
Meanwhile, how many are giving their teams the resources (i.e., people) necessary to do the analytics on the bill required to find & cut costs? Every cloud I've worked with has been terrible at billing in a way that could be reasonable investigated. At one point we literally pulled all the billing data into a SQLite database. (And that was a great idea, as it made queries against it much faster.)
Further, most higher ups seem to have a sense of "it won't be invented here", and so a lot of stuff ends up in vendor land; need more people then to pull those separate bills into one spot, and start attributing costs against the revenue they might help generate.
I've also worked a fair number of places where people were cagey about telling eng how much they're spending where. If you want your IT dept. to cut costs, they have to know where the money is going.
And then there's the yak shaving of it all: we've had a support ticket open w/ Azure for 9 days now asking "where did this $1k go?". Our Blob Storage "Cool" tier costs upticked, and I'm 99% sure the bill there is just wrong. Cool storage should be (price * amount stored) and the "how much are we storing" graph is not proportional to the cost graph. The only gotcha is an early delete penalty, and we are also pretty confident we're not hitting that. (We know what's being deleted when, and none of it should be subject to it.) But how does it take a over week to answer the question?
I don't get time for that - I've got access to Cloudability but no one has ever showed me how to use it.
- run on demand instances (maybe mix in spot)
- run fargate instances (maybe mix in spot)
- reserve them.
Every company hates #3, but it's usually the best choice if you can't go spot - you can swap them within the family for size, use them for baseline, it's fine. You're incorporated right? So if you fold, it's just another debt on the pile, stop worrying. Go no upfront if you really want to avoid the cash outlay, or go spot if you can defend limited outages.
If you have a 1-year project that will have a 1-year extension, then get a 3-year reservation (with some amount upfront) that you can sell.
Your second best option is a Savings Plan.
- Engineers may want to switch sizes or go serverless.
- Long term vendor lock-in.
- Miss out on price cuts.
There’s a reason the vendors push them so hard. (No spot market on Azure either)
Surprisingly enough we have 3 people working on it right now.
Sadly they’ve found that it was extremely easy to save several thousands of dollars a day, and has been so for the past two years.
Terrible for who? You or the cloud provider? I'm a bit of a worrier (and not a perfect coder), so infinite scalability with utility billing combined with a mistake on my part could result in an epic bill. All I'll say is that I won't practice cloud development on my own credit card.
In my experience it takes azure support over a week to answer any question. Unless the question happens to be which person without a substantial update is taking over the case. You’ll get lots and lots of those answers.
Someone else's computer has been painful and very expensive for our organisation so far. Escalating costs aren't even the worst problem.
Wrote to support, explained my situation, and they removed the bill without any hassle.
Azure forced me onto invoice billing against my will - and every month without fail they don't automatically match my payment and I have to endure increasingly threatening emails that my account is pending termination while their support takes days to reply to my tickets about it, and when they do reply the advice is usually just 'wait a little longer'.
I've moved critical services off Azure because the risk of wrongful termination is just too high.
Check your partial / interrupted multi-part uploads. If its like amazon the uncommitted blobs won't show up in storage graphs but you'll be getting charged for them just sitting there.
AWS in particular is handsomely engineered so you can't answer this sort of question, as well as making people spend more without need.
- Per resource cost is turned off by default. Turning it on costs money. How much? ¯\_(ツ)_/¯
- Services like S3 will bill you per API call. How do you know who is calling your APIs? Using Cloud trail, which is turned off by default. Turning it on for this sort of analysis might cost a lot $$$ for a reasonably used bucket.
- CloudWatch will bill you per custom metric and adds up quickly. Why do I have so many custom metrics? Good luck finding out.
- Many costs are tied with stuff that's really hard to control e.g. egress traffic.
- Using many accounts is by far the best way to segregate unrelated projects. But then, you will end up paying for many criminally underutilized NAT GWs, VPN connections, Load Balancers, etc...
- Many services are ha, multi az only. That makes AWS unnecessarily expensive for dev environments / non critical services.
The list goes on. I know a fair number of consulting companies making the money of their lives reducing AWS bills.
Not if you use shared VPCs.
And you can have business support only on one account... obviously you can go Enterprise and have the same support level on everything but pay a lot more $$$
It just always seems a bit salesie to me. Bring your network switching to the "cloud".. yadda yadda. Why? Everything is working great.
The problem is that mostly anywhere else in the world, you can get personell for much cheaper.
For highly volatile workloads, ephemeral workloads, anything where go forward capacity is fuzzy or unknown, cloud is awesome. For anything where your margins aren't robust, or your workload is known and consistent, you're better off going through the schlep of internal IT management (as long as your org isn't dysfunctional and how you operate this is a business enabler, not a hinderance, forcing shadow IT attempts). If you can't generate more revenue or margin, you've gotta look at cost savings, including the cloud premium.
The thing is, I've yet to see any startup even with VC money that has a product which has this need. Most startups nowadays factor away common crud, like booking a service or providing organisational tools. These high pressure, intense workloads are often incredibly unrealistic.
The scalable cloud is useful for scalable workloads (AKA how many times is you peak demand larger than the average?). If you want managed infrastructure, you are looking for renting a managed server or VPS.
Sounds like the more basic problem is that we keep ending up working in companies where the "higher ups" are constantly shoving decisions down our throats.
Which, ultimately, is probably the root cause of most of the caginess, the yak shaving, the compunction to use cloud databases and cloud-everything, and the silly support tickets that stay open forever and ever and ever.
If you don't fund, hire, and staff your support org with good people at multiple tiers, don't be surprised when you get crap. There just aren't enough good minds to service the incoming workload.
Oh, and for god's sake have a decent "to engineering" escalation path for the support folks.
Unfortunately, the support function is almost always a cost center and beholden to only, if anything, response time related SLAs. Instead of customer satisfaction, etc.
I was considering writing to Azure support about that...
My last billing issue with them took ~120 days to resolve, so I don't have much hope.
You are also right that it is across storage accounts, and access tiers: I've now looked at some of our other SAs, including ones in the Hot tier, and yep, they see the same bizarre uptick. Given that Hot tier is experiencing it, that rules out the possibility of "human error" w.r.t. to it not being a Cool-tier early deletion.
> Is this similar to what you're seeing ?
You've nailed it precisely; it sounds exactly like what we're seeing.
(What a small world HN is sometimes!)
¹the numbers on our end wiggle a bit, so I'm not sure about how precise I can estimate the uptick. But 70% is as close as any guess.
One of the most effective ways I've found to drive change in how product management and execs treat engineering was getting average salary numbers by type of position for my team out of HR, and attributing costs toward every major feature request + meetings in a weekly report.
My manager at the time, the CEO of a VC funded startup, did not see the point. We were after all already paying for the engineers, and they were always fully loaded. Until he'd seen a couple of reports, and saw what it told us about prioritisation.
Our project and product managers got a lot less meeting-happy after seeing one regular meeting they were very fond of cost an estimated 10k/month due to the number of people attending, and they got a lot more thoughtful about feature request and how to prioritize after seeing the real costs of supposedly "trivial" requests that had previously been slipped past normal processes.
Same applies on the cloud front, yet a lot of what has driven the adoption of cloud is that it often helps engineering escape cost analysis and efforts to optimize costs.
And so you get organisations which often have no idea which decisions contribute what to those costs.
I don't believe a baseline exists. The cloud is a new thing, and the fashion on how to use it is still changing constantly.
You also can not use this number as a proxy for any other thing. It's just what it is, you can't look for deeper connections.
That feels like it should be 100%.
An engineer should be as aware of their $ costs as they are their CPU load, filesystem I/O, or memory usage. An engineer should know how many instances they scaled to, and when they're going to scale down.
In fact, the $ cost should be calculable (to a reasonable indicative amount by napkin math - precision not necessary) according to the factors the engineer knows.
Not being aware of the costs feels irresponsible on the part of the engineer. There was a time when we had finite capacity (a single server) and perhaps the engineer had to care less about the cost then (but perhaps they should've cared about utilisation by some measure even then)... but now we're in an age of your application being able to scale beyond reasonable bounds of your companies purse strings, a solid understanding of what your application costs is an essential and core skill. If this is not the game you wanted to play, stay on a VPS or VM.
Anyone possessing a solid understanding will also be looking to reduce or halt their spending... hence, I am surprised the number in the headline is not higher, even ignoring the macro-economic picture.
In theory most places are running with slack. In practice they run under capacity, and engineers aren't going to worry about things out of their control.
Reminds me when I worked for a company. Tight with raises but spending half a junior dev salary on some monthly service they never used. And a couple of dev salaries per year on the company party!
It’s a matter of priority. During periods of revenue or user growth, reducing infrastructure cost may be a premature optimization. Now may be a time to focus on operational and logistical inefficiencies
Even during hyper growth you should be aware of your costs, because you should have an idea of your margin.
I'm not sure any engineer that has awareness of costs, CPU, memory, storage, network, etc... is not trying to reduce some of it - even during hyper growth.
Never once has a dev asked about storage costs, of the differences in storing a file in a filesystem vs a blob in a database,
Never once ask about mem optimization could reduce infrastructure costs
never once
The most common response (39%) was the C suite wanted spending to remain the same.
But my real gripe is that this survey is being presented as doom and gloom when you could equally summarize the results as "about 60% of IT leaders directed to stay the course or beef up cloud spending"
People often say this, but it's just not true. First, any company with significant spend has custom deals with custom pricing in exchange for a commit. Second, even if you don't have that, there's often reserved/"savings plans"/whatever where you commit in exchange for a lower price.
Third, the only time any of the big 3 cloud providers has increased pricing (it usually only goes down) is when GCP increased bandwidth and storage costs a few months ago. AWS, which is what the majority of the market uses, has only decreased prices, hundreds of times.
Regarding VMware, you know you have to pay for licenses and support in an ongoing manner and they are known to jack up prices, right? Furthermore, VMware's software is a dumpster fire that gets you around 5% of what you get from a serious public cloud. Maybe 30%-50% if you go in with NSX and vSAN and the vSuite where you lock in yourself to terrible software with the wrong abstractions, lack of APIs, at enormous inflexible costs. How is that any better?
Probably your savings will come from beign able to get bigger machines. After 4 years in the cloud I am still surprised to see how performant physical servers are.
1. Broadcom, if history is any indication, is about to make VMware much less of a bargain, likely exceeding the cost of the cloud, if not matching it.
2. Often time when a company moves to a cloud model, they lay off the traditional administration staff in favor of "DevOps", many many devops engineers would not have the first clue how to manage the low level infrastructure of Storage SAN, FiberChannel, vLAN's etc... let alone managing vSphere or hyperv, etc..
3. Often times when a company moves to a cloud model they decommission their Onprem Hardware, Lead times right now are crazy still so any org wanted to move to OnPrem quickly is going to be hard pressed to do so.
2 and 3. Sounds like the decision to go to the cloud is going to hurt companies twice in the long run. Glad whatever CTO made that call got his bonuses and moved onto the next place already.
It is also unclear what the long term path is for OnPrem Windows Server, many people believe in the next 5 years it will be replaced with some Variant of "Azure Stack" and will be billed on a subscription basis
from hardware vendors, to Microsoft. The future is clear. Everything as a Service is coming for OnPrem, your network, storage, and hosts will all be some subscription model just like "cloud"
Granted, I have no idea how the costs compare. In every company I have been at, the higher-ups tell us in tech we have to "learn about the business side of things", yet the business side never wants to share information.
All of the large enterprises who have transitioned to cloud have really good sweetheart deals, often with additional ties to other business initiatives between the cloud provider and other enterprise. They know the savings they're getting is extremely short-term, and presumably they expect to re-acquire or retain the talent to leave eventually.
Bear in mind, if you want an idea how little established enterprise values the cloud, in the year of our Lord 2022, the first automaker to transition to AWS is happening, announced last week: https://www.theverge.com/2022/10/13/23401350/bmw-amazon-web-... The way you can translate this is that if it took over a decade for the first automaker to transition to the leading cloud provider, few to no other auto manufacturers are outsourcing their data to the cloud. And I'm willing to bet BMW got a really epic deal to convince them to do it.
The reason things like multi-cloud are so derided is that these companies' goals are to get you entirely dependent on their proprietary service design, so that it's too hard to ever leave. You can't go on-prem, you'd need to do a whole bunch of engineering to transition to a competitor's platform, etc. So you just suck it up and keep handing them a blank check every month.
The graphic doesn't even have a category for being directed to halt spend and less than half are being directed to reduce it in any way so I'm gonna say I still improved it ;)
Wait, what? There's no way 29% of companies noped their way out of cloud provider lock-in. Not saying it doesn't happen, but 29% is a huge number.
Unless of course they're startups with no idea how they want their infrastructure run or legacy companies dipping their feet (?) in the cloud for the first time.
The GKE equivalent of EKS IRSA is GKE Workload Identity.
It's pretty much the same user experience:
* Enable Workload Identity on your cluster
* Create a GCP service account
* Grant your Kubernetes service account permission to act as the GCP service account.
It's a bit more seamless because you don't need to upgrade your client libraries. Instead there is an on-node metadata server that provides access tokens to workloads.
Disclosure: I work on this
We're a rather simple company though, less than a dozen services, we use Postgres and Redis for storage, everything deployed on Kubernetes.
Watch how fast the bill will go down
One guy turned off staging at night and saved $600k/yr.
I believe we did keep an official staging running, but the 11 bento environments were the ones that shut down at night.
I can guarantee you that they spent more than $50k/month on all staging environments.
Any other similar platforms out there?
Funny enough, for all the problems with studying business administration, they do teach this as part of the principal-agent problem. If the principal wants something done, and the agent has no incentive to do it, you need to give the agent some of the incentive so their interests are aligned with yours.
At the same time it's something you rarely see implemented by BA types, because they don't like to share.
Google "Hanoi rats" for an incentive blowup. Then there's myriad other incentive issues related to taxes and subsidies, all stuff that had the good intention of promoting good things and reducing bad things.
Sometimes its a meta-game: if a team sees a "share the spoils" scheme being trialled with another team, maybe they shouldn't fix their issues until there's a scheme in place for their team? What about the shape of the payoff? If you fix two issues in the same period, are you better off or worse off than splitting them over a boundary?
Even in sports, there's issues: there's an incentive payment to break the world record. Say you can pole vault 10cm higher than the previous record in your practices. But why break it once when you can break it 10 times? I read about this some time ago, should be findable online.
- Corrosive effects on team cohesion, when separate ICs compete to see who can find cost savings first.
- The "cobra effect", where engineers will deliberately introduce cost inefficiencies so that they can fix their own bugs later.
- (Related to the previous point) Prevention of cost overruns is even cheaper than fixing leaks after the fact, but is very hard to properly incentivize.
Any class on principal-agent problems will hopefully include a chapter warning against asymmetric incentives and how they will be ruthlessly exploited by the agents. Engineers are not some exception here, in fact because of their training they are often extra good at spotting loopholes in the rules.
Commissions, metrics, and goals and heavily used by sales across many companies. I’m sure there is some abuse but it is still heavily used.
Mangers at one of my previous workplaces were very vocal with "everything is running with slacks" so whenever I gave them an estimate on how long a feature would take to implement or a bug to be fixed (bugfix estimates are BS to begin with but...) they took it and multiplied it by 0.75 at most. Then bitched about it when it was over the time they assigned. Guess what happened.
Exactly this would happen with a cost-cutting incentive as well - if managers told us "reduce cost by 30% to the end of the year" then the cost would creep up to begin with. Incompetent management making stupid rules and stupid incentives is what drives me nuts :) .
The devil really is in the detail: "you need to give the agent some of the incentive so their interests are aligned with yours". The vast majority of employers don't pay SW Engineers enough and don't incentivise them enough so they give a shit. This really has to be a company culture, not forced but lived especially by managers, then it would work with engineers as well.
Building a new system that doesn't have a baseline cost? If the cost saving bonus is based on a reduction in existing cost then you don't get rewarded for a new implementation. The business don't have a baseline cost yet. So what do you do? Build it to run at a higher price, run it for a few months and then cut costs.
Also, is the reverse true? Deny bonus if costs rise. Because if it is, then the engineer has to prove that the increases are directly related to increased customer activity. But what if increases aren't relative to customer spend? Etc
The industry is in a weird spot right now where Finance doesn't understand Engineering well enough to provide proper direction on where Engineering spend is truly wasteful (versus necessarily large), and Engineering quite frankly doesn't care, as long as Finance keeps paying the bills. It's rare to find leadership that appreciates both the underlying financial concern as well as where best inside their tech stack the financial waste lies and how to tackle it and prevent it in the first place.
Then engineering cares, and the needed talks/concern about resource usage can happen. Part of engineering is optimization, cost is one metric to optimize.
Budgets are not about what things cost, it is about what you or the business can afford.
For example, I do not need to know how much Steak is this week to know I have $100 for food.. If steak fits in the budget then great, if not I get chicken... or beans...
The steak example is different because the steak has a price tag on it.
IMO this is the easiest way to drive cloud adoption: tell PMs that the cloud is instant but on-prem takes weeks.
What's wrong with letting the specialists handle logistics so you can focus on business logic instead of basic piping?
Growing food is more like designing and manufacturing servers.
Cooking is more like taking these goods and putting them to use.
There are plenty of places that host their own servers, including people that have these servers in their home or office. But far fewer produce these machines.
You could also say that cooking food is dumb. What's wrong with having professionals chefs do it? But that also doesn't take into account the added costs which makes it unfeasible for most.
Facebook does not use cloud providers and designs their own servers.
The very large compute uses and cloud providers have incentives to grow their own food. Doing so can save them a few percent at scale. That savings is both not passed on to you and even if it was is not significant at small to medium scales.
The economies of scale only get passed to the consumer if another provider is willing to pass the savings on, which only happens in commodity markets (like oil, wheat, etc.). Low-end server rentals are a competitive market. "Big 3" cloud is not: they use differentiated products and egress fees to lock you in to their oligopoly.
They don't have to pass on their economies of scale to you. They don't pass on their economies of scale to you. They charge you slightly less than your cost to leave them, and that's it.
Unless you're an infrastructure or network logistics company, it's a waste of resources to build your own. Cloud services are so commoditized already it's unlikely to be any sort of meaningful differentiator for your company. Especially if your field isn't even tech proper.
Plenty of mid sized data centres make sense too. Something I see in common with many of these businesses is either the requirement to store a lot of stateful data or to have data locality and to not be concerned with things like being able to serve international customers or being a plodding business which will predict at a steady predictable rate. Some people run businesses that run headfirst into cloud companies entire pricing strategy because their business doesn’t work like a VC funded tech startup.
The “but somebody can figure all this stuff out for you” cuts both ways. Why can’t a skilled company make it easier to staff and operate an on premise data centre? Running a datacenter only gets less labour intensive for what you get every year. You know that you can hire people for pretty much any aspect of building or maintaining a datacenter that you can imagine? The datacenter is commodified so why is it special that the cloud is?
I just feel this analysis is a spurious oversimplification and what you’re far more likely to see is a sort of Pareto distribution of datacenters and the cloud is never going to swallow the whole market.
The trick is you need skills, most of which are not in demand anymore and difficult to fill. If you pretend like you can hire and manage: the results are clear.
There are opex/capex aspects which I don't understand, I amortized servers with an average server lifetime of 3 years.
It's easier for most people to spend money with one vendor, though, and that's where AWS/GCP makes sense.
Sure cloud is easier and if you need things that aws offers then go for it; we run well on a few $100/mo. Our most sold/popular product costs (I did this calc because of another ‘cloud is great’ HN post a few months back) less than $1/mo to host and maintain and has 50k+ active users.
I like and use aws but it makes no sense in many cases; I would rob myself running most of our products on it because they don’t require remotely anything that it offers. I think the ‘make a unicorn’ mindset messed up people’s bottom line understanding; making 10k-100k$ more profit seems pathetic to companies that raised 100m$. Thing is; that is real money and I definitely rather have dinner with my colleagues than send it to Amazon.
Being able to avoid most of the costs and training and staffing you’re talking about has been a thing for a LONG time before the rise of cloud. There’s long been plenty of companies that will run the datacenter infrastructure for you or ever servers and let you hash out the details. People still made their own datacenters and continue to do so to this day. That is not why cloud has been so successful, the biggest thing cloud did to be successful was pioneer a new pay per use business model and the tools to enable it. Later on they created a bunch of new business innovations like EverythingAAS. If those innovations don’t come into play, AWS isn’t more attractive than Hetzner.
I’ve built and decommissioned smaller datacenters for different businesses a bit under 1MW. I’ve seen datacenters make no sense, I’ve seen datacenters make complete sense.
I think it's kind of "golden handcuffs" type of situation - if you want these people to come and work for you you have to compete on salary with Amazon, and Bezos has much deeper pockets than you.
The result will still be orders of magnitude cheaper than AWS.
That's a mistake.
You can get deeper into the weeds and look at Zinc Whiskers and such but that's a niche you just have to generalize.
Many companies who buy their infrastructure learn that they can run servers for 10+ years if they are properly cooled and maintained. Parts that wear out (fans and disks) are cheaply replaceable.
Engineering a web app in a way that autoscaling is effective is a lot of work. It is very well possible that renting a beefy dedicated server will do just fine and cost a magnitude or two less in renting and engineering both...
Also, people tend to say I lost my mind when I say complex and expensive high availability solutions are often not worth it: it's cheaper to be down. Because you will be down, no matter what you do, there are a multitude of services you will rely on -- and some downtime is completely expected and accepted in this industry. So adding a few minutes every few years when you need to manually fall over to a slave database versus using a complex HA database solution? A cost-benefit analysis is in order.
If you do need the cloud, yay but so many people no longer even consider whether they do.
That's plain false. Cloud companies are charging significant markups.
> except taking much longer and usually ending up with worse service ... it's a waste of resources to build your own ... Cloud services are so commoditized
All incorrect generalizations. Plenty of companies have capable engineers running their own services for cheaper. Do the math.
- Ability to scale up and scale down in a matter of minutes.
- Reduction in variable capital costs required to meet peak demand.
- Ability to test and prototype products or services at scale without significant investments.
- Ability to have, on tap, products and services that in a traditional sense, would take a lot of time and manpower (A company can invest in equipment and manpower to spin a few services. But as you add more services, it costs more and more to build and maintain such systems. With cloud services, you can use, test and experiment before finalizing)If I understand your point correctly it is the opposite - cloud not only lets you trade capex for opex, but also allows for opex variance. Yes, clouds do let you scale up and down (if the application is designed to do that) easily, however how many businesses do really have high load and geography variability?
That is the core point behind cloud vs on-prem: capex vs opex and opex variance.
If you skip economies of scale that is. AWS, Azure and GCP allow you to use stuff that has cost billions of man hours and matériel to develop and deploy. How much would it cost you to build a global network? A globally distributed DB? Custom ARM/Tensor chips which come out much cheaper than mainstream alternatives for some workloads?
If all you need is to host a few apps in a narrow geographic area, or do compute/graphics intensive stuff on an inflexible scale, it will probably cost less over a long period of time (3-5 years). But it's still much more inflexible, time consuming and hard. You need to hire the right people and they need the time to order, configure and deploy all that infrastructure.
There's no real alternative to cloud-based hosting if your application needs to be reachable from more than one geographical region or has wildly volatile usage patterns.
My last company had all the AWS goodies, spending thousands per month...on a site that got around 50 hits a day.
>90% of companies are overspending on cloud resources. On average, 30-40% depending on provider (AWS is typically the worst).
>75% of IT/Engineering teams have overbuilt their environments to leverage the latest & greatest tech stacks / toolsets with no justifiable business need.
>50% will confidently state they have “optimized” their environment and in almost every case there is money being left on the table. We have only engaged with one customer that we couldn’t offer savings. They spend $30M+ per year on AWS and had a full time staff of 3 engineers solely focused on cost optimization.
>50% of IT / Engineering leaders became incredibly defensive when presented with savings analysis which almost always included reduction in technical complexity of stack. We have since pivoted and marketing/engaging almost exclusively to CEO/CFO as first contact.
>50% of companies had one of more cloud costing/monitoring tools in place that was not being leveraged, or worse, gamed to produce inaccurate reports.
My favourite find was a database backup someone configured where it was keeping full uncompressed daily backups forever. Fixing just that cut their spend by $100K annually, and then we kept going.
The worst offenders are governments, especially in some countries that haven't had sufficiently brutal economic downturns recently to force anyone to do some belt tightening. Australian government is spectacularly wasteful, burning piles of money in the cloud for services that only malicious bots ever visit.
Like you said, these places love to pad out there resumes at the expense of the taxpayer. I regularly see multiple Kubernetes clusters deployed for one web app. Or architecture diagrams that look like a spiderweb of connecte d systems for something that should be running on a single VM.
Previously, having two years of runway was considered plenty since new rounds were raised every 18 months or so.
The word around VCs is that there won’t be as much bubbly investment as there was in the last decade, so now a four year runway is a common target. So you cut costs till you get to a four year runway.
A couple months ago, folk were grumbling about a non-deterministic failure in the main build CI plan, for the N'th time without attempting to do anything about it. So, I proceeded to queue up dozens of jobs at a time, every 15 minutes for several hours, until I had several failures worth of interesting logs. I also had the interest of several other software teams, wondering why the entire CI pipeline was clogged up.
It turned out someone years ago had written a cron job that would asynchronously delete every file on the build server older than a week, every hour on the hour. At some point some build artifacts got added that came from a cached copy on a remote server rather than locally built, and their modification timestamps were preserved. Any jobs that happened to be running while on the hour would have those files mysteriously disappear without a trace. Oops!
Reserved Instances are a really quick and easy way to blow a few million with one click.
Europe (Frankfurt), 3 year, all-upfront, Windows w/ SQL Enterprise, Dedicated Tenancy, u-12tb1.112xlarge instance at $5,926,378 up-front.
The discounted $225/hour rate for it disappears into the background noise.
(No, I'm not willing to test it to find out if it actually works)
[1] https://aws.amazon.com/ec2/pricing/reserved-instances/pricin...
(should have read the licensing issue - still you can buy 2 and the consultant)
An iPad Pro is faster and cheaper than many cloud resources. I don’t advise to run production workloads on an iPad.
Question 2: does exist some kind of optimization-per-usecase service for such a fixed cost scenario? E.g. "Optimize for Website" or "Optimize for Database" and you can download a Terraform file?
AWS lets you provision some service though (I think DynamoDB has three different ways to be provisioned now) but it's not per use-case, it's based on what you think your needs are.
https://aws.amazon.com/ec2/pricing/reserved-instances/pricin...
2. You can chose which type of machine you want to reserve as your servers. Some of them are optimized for memory (good for databases). Some for CPU, etc
An example of this is s3 storage. You tend to find your storage cost will increase over time as you put more stuff into s3. S3 also charges per api call so you'd have to somehow fix that.
You have egress fess also which you're not in control of so variable customers usage will net a variable cost.
In terms of reserving compute power (a popular answer) you can also achieve fixed price with on demand if you don't auto scale. So reserving here doesn't remove that variability. Reserving just make it cheap if you have constant use. So you can reserve you DB server as it's always on and you'll save over on demand.
Many of the small cloud providers offer much less variability for their services with things like fixed networking cost. I tend to use those as I want a fairly fixed bill. The only thing that's variable on my cloud bill is backup storage and GBP to EUR exchange rate. The backup fee's are < 5% of the total cost at the moment.
When cash is tight you just shut them down. Had the businesses bought expensive servers and hired specialized personnel they would not be able to wind down that effectively.
Instead, I find myself being slowed down greatly: "Having a log shipping solution would be nice here, but I cannot set it up, because there's not enough RAM. Static code analysis? Not enough RAM. Another back end API instance for that new front end we're working as a replacement for the old one? Not enough RAM. Wanting to test how well scheduled processes run overnight with larger amounts of data? Too few CPU cores. Want more container images stored in your own Nexus instance? Not enough disk space. Want another database instance for CI or testing destructive migrations? Not enough of anything."
Which is the problem that the cloud largely solves, at the expense of way much money spent than necessary. But hey, when you have to deal with red tape to get more resources on-prem, that's just a way of managing the overall process and manages risks that you might not care about in the moment when you "just want to get things done"; albeit that's not to share that you shouldn't care about them.
So there are definitely problems with either approach (even cloud setups might have to go through someone and Jira/Redmine issues, as opposed to self-service), albeit at the end of the day most of those feel social/economical - after all, those large corps need their cloud service profit margins.
Archive: https://archive.ph/jBg8z
These companies need to think in terms of improved cost efficiency and reduced waste _relative_ to the system throughput. If you incentivize your employees to reduce absolute cost absolutely, you'll lower throughput and build a culture where scaling up is a fearful event. Scaling up should be seen as a triumph - more work = more revenue - unless your business model is a total failure.
There is an initial setup cost, but maintenance is usually cheap if you put things in place correctly, automate everything and put proper monitoring system (not the cloud version, of course).
Being frugal with cloud resources forces you to consider your tech choices. That’s a good thing!
500 decision-makers isn’t tens of thousands of companies moving to the cloud. Worst poll ever.
For about 4-5 users... You can't imagine the effort of configuring around and keeping up with the latest updates. Saddest part is the frontend had so much glue code, it was not fun to look at.
Is anyone having a good multicloud experience? Any advice to share about keeping workloads portable across clouds?
* Start with your architecture, KISS is king. Monolith using as few pieces as you reasonably can. Do you really need two DBs? Do you really need a message queue? etc. Trim, cut, remove. Scaling is mostly accomplished by using larger/faster systems.
* Only use the basic services universally available (VM, S3). You need your own provisioning and backup scripts and routines.
* Use your own VPN, nebula has been good for me. Makes your multi cloud infrastructure look like a flat network internally.
* Be very aware of egress costs and how your provider calculates them. Keep intra-cloud comms down. Don't do obviously bad things like host the DB at a different provider than the app server.
You can also mix on-prem/colo with cloud fairly seamlessly if you've built up this way. Cloud makes sense for really small, and for quick changes. Own your hardware when it makes sense. Make the switch when the finances line up with no need to change any software.
Edit: spelling