Cloud costs are in a bubble
the-investing-desk.com
the-investing-desk.com
In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year?
I think the meme that developer's time spent on cost control is useless needs to die. Engineers should be encouraged to control costs so that they can take better decisions early on. They should be nudged to pick more cost-efficient choices. This won't only reduce costs, the chances are companies will also end up having dramatically simpler architecture.
[1]: https://www.cnbc.com/2019/03/01/lyft-plans-to-spend-300-mill...
That has a minimum cost of their hourly rate. The maximum cost is the cost of not doing what's most valuable to the company.
> In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year?
They didn't spend that in a single year.
It also depends what they were doing, I was recently very shocked by the data costs of a major hardware store in the US but after breaking it down it made more sense. It all depends what they get for that $300M over 3 years.
> Lyft has signed up to pay cloud market leader Amazon Web Services at least $80 million per year for the next three years, totaling at least $300 million.
You can launch a massive project, hire dozens of people, buy server and colo space across the globe, rebuild their infra from scratch, and get costs down to – what – $60 million a year? Lyft is closing in on $4 billion in annual revenue. A few million in savings is meaningless, especially when balanced with the massive risks of such a project.
Also when you run the DCs yourself, usually it's much slower for other engineers using your DC infra than using AWS or similar. It gets complicated!
Because they aren't a priority until they are.
Execs will either turn a blind eye or simply won't understand that it's a problem, until a massive purchase order lands in accounting and gets escalated to the highest levels.
Now it's suddenly a problem and it is up to the operations/devops/SRE/name of the month/SecFinDeVOps to figure out how to reduce costs. Which is completely backwards as they are probably not the group who requested the resources in the first place.
For better or worse a lot of folks are allergic to this. A framing that I’ve seen is to just understand the cost drivers from a technical perspective, for example large materialized views in BigQuery consume a lot of CPU. That’s technically interesting and it’s the reason they’re expensive.
Maybe all those computers are doing something that someone thought was important, but it is not serving the core business.
Dude's got a screenshot. I doubt he faked it for the attention - he probably actually got that bill.
He feels it's expensive and he should pay less for the value he's getting considering his alternatives (Azure VMs in a 1/3 of the price).
Perhaps refer to the actual price of a billed hour there? That feels like the core theme here.
MongoDB Atlas runs on the big 3 clouds, and of course you're going to pay more for a fully managed service, as opposed to hosting it yourself directly on the big 3, but racking up $2,500 in serverless compute charges in 24 hours seems very very hard to do, and especially by accident.
And then I read your first sentence - why is it always the assumption that people are going the hyperbole route? This is a well-upvoted link, with enough comments to merit a good discussion, and what appears to be a nicely-thought out post after the link.
Also, I've been following Snir for quite a while - dude wrote a massive Data Engineering roadmap which is heavily used by people learning how to ETL and has a very interesting investing blog. He's an engineer turned investor, and looks at the market from this perspective.
The derogatory quotes are unnecessary.
TBH I know how to spend that much and more with a single query (and ~30 minutes of compute time) in BigQuery. Serverless BI analytics workloads over large public datasets can do amazing things... and are also exactly the sorts of things a naive user would use to test + benchmark the capabilities of serverless data-warehousing systems.
For all of this to stop, not only does the bubble need to burst but people would need to be far more intentional about what resources they use where and when. People would also need to have longer software lifecycles, more QA, and so on. This would only happen in an environment with far higher interest rates and far lower monetary velocity. In essence, there'd have to be a lower tolerance of risk to stop the constant churn of hardware, software, and development methodologies on both. There are tradeoffs to such changes.
No, they don't 'just add more complexity and overhead'. They are tools. Properly utilized, they are great.
There's a reason why you are (probably!) not running a power station and home and have, instead, hooked to a power utility company. They have the expertise and can amortize the costs among many customers.
Similarly, containers actually brought efficiency – they replaced many(obviously, not all) of the use-cases that usually required virtual machines. They also simplified some very difficult deployments.
The problem arises when you are doing things for the wrong reasons. For example, using 'microservices' just so you can try to avoid teams from having to talk to one another and end up shipping your org chart.
I do agree with a more generalized statement: more often than not, monoliths are what you need.
For the same reasons you outlined above: complexity.
Is this just legacy applications that are stuck with Mongo? Certainly nobody is starting new projects on Mongo?
Serious question.
Having used both, I personally would never choose mongo over Postgres, there are just too many conveniences Postgres offers that are a pain with Mongo.
Serious question.
E.g. around the 300GB mark I had to constantly remind myself and others to be careful of it because it might have normal relational columns you think are fine to query, but it also has a fat json payload on each row and if you're careless the query planner would basically do a table scan for things you wouldn't expect.
This isn't so much a problem with json in an rdbms but more the false sense of query performance on non-json columns in the same table...after moving it to mongo never had the same problems because people knew not to query it in an unsupported manner.
I don't use mongodb in particular but I never want to go back to worrying about what a SQL application instance is up to, either.
because that is how they compare feature wise.
postgres is not easily comparable to mongo, these are totally different db engines with different tradeoffs made for different purposes.
They very obviously have a significant degree of feature parity.
The mongodb pricing page says one can setup a M10 instance for $57/month. [https://www.mongodb.com/pricing]
Racking up a $2500/day bill is several of magnitudes off - there’s nothing in the article to help put the workload in context.
I have zero interest in mongodb, as a stock or a technology, but this article doesn’t help to actually determine anything about cloud costs. (Lack of data points, irreproducible method and conclusion)
I used the "Serverless" product, not the "Dedicated" product. That's the gap you see here.
But your point is important, I'll add it to the post. Thank you.
That is priced quite clearly on the bottom page at 102.49 USD an hour.
So in this service the actual cost of the hardware is a rounding error. It also needs power, cooling, a roof and some maintenance crew, but that still doesn't come close to the rental price. So what are customers actually paying for?
New era of cost-cutting, running systems yourself and distributed computing will bring far cheaper prices soon. If we do well energy and computation/intelligence will be trending toward 0 generally speaking
Most of the time is more expensive than paying for the infrastructure and personnel to manage it.
Also, while apparently cloud reduces complexity, it will end up adding lots of other complexities. And dealing with complexity means more money.
Also, the more you invest in a particular cloud technology, the more locked in you are, the less alternatives you have and that means paying up more.
If Stack Overflow does well by managing few dedicated servers, I don't get why smaller operations really "need" cloud.
And to add another perspective, the more layers of abstraction and indirection you add, the more complicated and ineficient things will be.
With the right offering and the right software you can build something just 10% as good as some of the hyperscalers and have an amazing company.
Competing on price is not a sin, and it's what I'm working on/betting on.
That said, I think the real big threat to the hyperscalers is cheap chips and more people being able to treat datacenters like parking lots (nice ways to use capital to generate cashflow) -- the commoditization of data center operations.
Of the large salary companies - Amazon, Google, and Microsoft are in the cloud space. Google doesn't make most of its profits from cloud. Netflix and Facebook don't sell these services at all. Dropbox and Digital Ocean (and sort of Cloudflare) are also in the cloud space and aren't known for such salaries.
It's possible everything is in a bubble from the low interest rates. My prediction is Adtech is far more likely to be in a bubble than cloud computing. The productivity gains and small number of people who can fill the role for cloud are going keep it dominate probably for a long time even in a recession.
I only ask because it is has been around for a few decades now, has steadily been on an incline while old advertising methods has slowly been on a decline. Those trends are more likely here to stay…
The pandemic fueled weird economics everywhere but I don’t think we will see falls below pre-COVID
Interest rates go up -> VC Money goes down -> Advertising budgets go down
Online advertising is "theoretically" better than other advertising. You can track exactly how it converts to sales or not. It is still the best way to advertise in my opinion so will beat out any other method for sure.
Just one company, sure, but it did make me wonder about adtech in general.
The margins are so insane that it just doesn’t matter. They are trading engineering time for time to market.
It's always so hard to nail down AdTech, to the outsider it seems like a world filled with bots and bad data and fueled by FOMO. Probably a good chunk of Google's revenue comes from companies that don't really need to advertise, but do so anyways so they won't loose their "competitive edge"
In the end major financial decisions have many tradeoffs, but the micro optimizations are less important than having a high savings rate. If you make 150+k and make reasonable choices more money hits serious diminishing returns.
If you believe we're in a low-inflation economy, you may want to have an argument on that subject with folks that don't. :)
Central bank interest rates don't really matter much for investors or borrowers. Typically investors would use the 10y treasury yield as an approximate risk free rate and apply a risk premium to that calculate their required ROI. Central bank interest rates only matter to banks.
For the last decade 10y treasury yields have ranged from around 1.5% - 3%. And as recently as 2018 they were around 2.8% - 3.1%. Today the 10y treasury yields around 4.2% which is meaningfully higher, but I think people are vastly over stating the impact of this. It's also quite possible the 10y yield is currently peaking and will move down over the next year. The long-end of the yield curve typical moves with inflation + economic growth expectations. I'm not committed to any side of this argument but I will say there is good reason to believe inflation pressures are likely to trend lower globally over the next several decades because of global demographic trends and slower rates of productivity growth. This is why interest rates probably haven't been as recklessly low over the last decade as many people in 2022 like to suggest - they've needed to be as low as they have been to prevent deflation and economic contraction.
Similarly to investors, companies don't borrow at the Fed funds rate, they borrow at a premium to what the treasury market yields. So while corporates will need to borrow at a slightly higher rates to fund growth going forward, I again think people are vastly over stating the impact of 1-2% higher borrowing costs.
That said, it's totally reasonable to expect margins to fall as industries mature - especially when there is a lot of competition in that industry. I do believe cloud margins will contract slightly over the next decade, but I'm not convinced pricing is in a bubble either. Firstly, investors (collectively) are far from stupid. If you think the market is wrong and that you're the only one to see these risks then you probably haven't been humbled by the wisdom of markets enough. Cloud company will be growing for many years to come and valuations today imo do already allow for some margin contraction. But we've seen this all before... As OP mentioned SSL certs used to be ridiculously expensive, now you can get them for free. I personally remember having to pay several hundred dollars a year for a VPS, dedicated IP and SSL cert back in the mid 2000s. Hell, it cost about a $100 a year just for a domain + decent webhosting back then. I mean if pricing is in a bubble today then you should have seen it a decade ago...
Finally, when it comes to mortgages obviously it's important to ensure you can afford your repayments even if the unexpected happens, but as a long-term borrower inflation is your friend. Even if real wage growth for software engineers decline the likelihood of significant nominal wages declines for software engineers is practically zero. As an example while the real wages of cleaners and factory workers may have declined in real terms over the last few decades, in nominal terms they're making more than ever, and that's really all that matters in terms of servicing debts.
Plan like everyone else in your country who has to deal with this uncertainty on significantly lower salaries.
Budget. Save. Trim expenditure. And enjoy your career because you chose it as a career, not because it allowed you a lifestyle.
This whole "its your calling" thing is how the abuse and low pay in adjacent industries, like game development, are maintained, and frankly it's BS.
I'd go back to school and into something that will maintain my lifestyle.
And then, when you are prepared, you can be more calm about a lot more things. Yes, it's possible for the Zombie Apocalypse to hit and make a mockery of all my prep... but because I'm more-or-less ready for more realistic scenarios, I'm also more chill about the possibility of them happening.
And they may not, in which case frankly I won't have wasted many resources anyhow, any more than it's a waste to buy an insurance policy that you never get to collect on.
But that also means that you need not be too alarmed. Even just one year of making the aforementioned $300,000 allows you several years of not working at all before you are making less, on average, than a typical low risk job. As you've already priced in the risk, you'll be fine.
And if the market is wrong then you've gained a nice reward.
That, from an employer point of view, is certainly what makes the difference between offering a high risk premium and deciding that the work isn't worth doing.
> which is partly because of the value of experience, which takes time to build
All jobs value from experience. The question the worker has to ask himself is: Where am I going to allocate my time to gain experience? One can't do everything. They have to make a choice. Tech is a risky place to allocate your time, but compensates for that by offering a premium to compel you in that direction.
Tech is risky because it is all about exploring unknowns. You get periods of fruit, but you are also likely to get periods of dead ends, and when those dead ends start to accumulate people start to back away. That is quite unlike, say, road maintenance where there is a strong belief that roads will be around for the lifetime of one's career. That isn't a guarantee, but it is lower risk. Tech is known to disappear.
Tech has crashed many, many times before after too many dead ends. It feels to me like we're in a dead end period. When was the last time you were excited about new tech? The WFH period during the pandemic brought some small excitement in tech emergence, but it seems we're trying as hard as possible to back away from that.
But who knows? This time might be different. And if that's the case you've got your risk premium as a reward for taking the risk.
That said it's always a good idea in general to live below your means, and to be specific, not get the biggest mortgage your lender thinks you can afford.
You're not exactly risking your life and limb by taking a job at a cloud firm. Demand for software isn't going anywhere, and no hiring manager at any future job will give two cares as to whether or not your previous employer's revenues were inflated or not.
You are risking opportunity elsewhere.
Let's say your options are working as a developer for a random Fart App startup or a government job sweeping floors that has been around for centuries. Which are you going to choose? If they both offer $50,000 per year, the choice is clear: The government job. It is the one that is almost certain to have more longevity.
But if the Fart App startup increases their offer to $300,000, well, now it's a harder choice. The startup still isn't likely to last, but with $300,000 in your pocket you're not so concerned about it disappearing into the void on a whim. And if on the unheard of chance they do make it as a lasting company that spans centuries then you're in a really great position.
> Demand for software isn't going anywhere
Thing is, it does crater approximately every 10 years, as we've observed since software first became a thing. It is more entrenched now than it ever was, so there is some reason to think this time is different, but the market is pricing in the risk regardless. The market can be wrong, but the market likes to be prepared.
It slows down, along with the rest of the economy during recessions, because it, like everything else, is driven by the same market forces.
My point is that in the average case, there is ~zero opportunity cost or damage to your career, because you worked at MS Azure, instead of... Somewhere else.
You cite fartapp startups that run on rainbows and free VC money, but cloud providers are about as dissimilar from the fartapp as you can get, in terms of their economics and prospects.
You may be bearish on them, that's fine, I can't predict the future, but I see no reason to believe that you're going to have a better life working at literally any other random software firm, during an economic contraction. There's always a few workplaces that tend to be counter-cyclical, but they are, by definition, a minority.
... Also, many government jobs are a political football, that are only as secure as the next election. The best part is that after you worked one, and had half your compensation deferred to retirement, some politician will come along, and campaign on reneging on your pay, after you've given your thirty years of service.
Tech has been the darling of late exactly because it hasn't followed the general economy. In 2008, when virtually every other industry was about to sound death knells, developers were making bank producing apps. Similar story through the pandemic, when people were locked out of work during lockdowns tech was making a killing selling them things to do with their free time. Not having a good reckoning like everyone else only makes it appear more risky as time marches forward as it is assumed its day will come.
> but cloud providers are about as dissimilar from the fartapp as you can get.
I can't imagine Firebase, which is supposedly the fourth largest cloud provider according to a recent article here, is used for anything much beyond fart apps. As the fart apps die, where does that leave their business model?
Providing tooling to the gold prospectors is a good business to be in... Until people stop prospecting for gold. Firebase is Google at the end of the day, but if that division dies, that's still significant.
I appreciate the concern though :).
Cloud computing is pure scam. you rent 1 vCPU and people often think this is as good as a real 1 hardware core, while in reality that physical core is being sold twice/three times to different customer. Your 1 vCPU is maybe 0.5-0.75 of a real hardware CPU, perhaps even less, depending on how greedy cloud provider is.
It is almost like instead of driving your own car and making stable car payment - you decide to exclusively use Uber/Lyft to go around. Sounds good if you are in NYC/SF, but not so much outside of these perfect use cases. Also doesn't make sense if your primary job is pizza delivery, all your margins from delivering pizzas will just transfer to ride hailing company.
I'm saying this as someone who manages a 50 TB cluster for a startup (which I know isn't a HUGE amount, but definitely not small either)
Early days I remember looking at Cloud vendors that were offering FISMA virtualization capabilities and I did an analysis of alternatives around running a set amount of CPUs, RAM, i/o and disk to support VMs. Terremark was the first offering that could meet our reqs, at around 10k a month per for a server's equivalent capability.
This put in my mind from day one that I was trading off High Capex with low Opex, for No Capex and High Opex. Given our trades, it was clear that high Capex was preferable to high Opex, as our high capex would have been equal to one month's opex.
When we later on saw more mainstream vendors like Amazon move into the space, their offerings were also coming with insane overhead, $2000 a month automatic surcharges for a FISMA regulated environment.
The thing to understand early these days is COGS and whether the business can scale its service with good margins. That is a significant change of concerns for investors from the past years where growth was the only thing that mattered.
The other thing that I think is a little off in this post is the idea that startups not caring about costs are a big driver of hyperscaler revenue. I suppose that may have been true a few years ago but AWS, Azure, and to some extent GCP are having a lot of success moving legacy on-prem deployments to the cloud. It's not uncommon to encounter 50 year old enterprises with a billion dollars of cloud spend. One example of this is the dedicated SAP instances that AWS has, some of which cost $100K per month.
Some of the cloud provider compute (VMs) and storage (disk/object store) has very low margin. With it highly unlikely a company could provide equivalent offering in-house.
At a glance, Atlas pricing includes the typical ~ 20% premium if we compare it with AWS (Azure/GCP) VM setup. 20% is very far from *100 price difference as per article.