Hacker News Reader

Comment by neom | Hacker News Reader

Bryan Cantrill is a million times smarter than me so I'm probably missing something. I thought people bought into the enterprise cloud ecosystems because of cost savings elsewhere? If you go into a platform like AWS/GCP/Azure shouldn't you be designing your applications to take advantage of the flexibility in the wide range of services offered? If you don't need that and you're not buying into all the abstractions they provide, you could use any other cloud-lite (Vultr, OVH, Linode etc)?

Maybe I'm stuck in 2015 but are people really out here rolling their own clouds these days?

View on HN

Our big belief is that elastic infrastructure (that is, cloud computing) should be orthogonal to the economic model (that is, own versus rent). Today, that is not the case: if you want cloud computing, you more or less have to rent it. But we know that there are many who wish (or need) to run cloud computing but on an owned asset. There are many factors that deter that today, not least the odious per-core licensing that we highlight here. (For a particular brazen example of this, see VMware's infamous "AMD tax" ca. 2020.[0])

[0] https://news.vmware.com/company/cpu-pricing-model-update-feb...

Why aren't there prices on the website? (Seems weird that there isn't given that blog post, where costs are pretty central.)

(not Bryan, not actively part of the decision to put prices on the website or not, just my own take.)

Different audiences have different expectations. In the place we are, that is, enterprise sales, the expectation is not that you can go to a website, get a price, and click a button. The expectation is that the two organizations will communicate over a period of time ("the sales cycle") to work through everything that goes into a deal, and eventually come to an agreement or not. This includes so many variables that putting the price on the website wouldn't make sense, as you're never going to end up at that exact price.

I used to find this attitude frustrating, as an engineer, but the longer I have been a professional, and worked at various organizations, the more I come around to that being a good thing, not a bad thing.

All I really want to know is "ballpark" - are we talking four, five, ten figures?

I feel bad burning salesman time for a solution that is way out of our range, but I can't learn that until after a meeting or two.

I hear you, but I can assure you that this comes with the territory for sales people. Qualifying leads is part of the job.

It's between a half million and a million.

Oxide is offering you radically different -better- terms. The price they mean to charge will almost be based around what they need to a) fund more development, b) grow, and probably also relate to what their competition charge. The actual price you would actually negotiate surely depends on other factors, like how badly you want those better terms. If the better terms aren't enough to make you talk to them about pricing, then Oxide surely isn't for you.

If the price isn't listed, I always take it as a sign that nobody is buying it for price reasons ("if you have to ask, you can't afford it"). Outside of the blog, I don't even see a claim that it's cheaper so that's consistent.

That is understandable, but you would be mistaken.

That quote (which may not even have been said by J.P. Morgan) is talking about luxury consumer goods, which is a completely different market than business to business sales.

I wonder how many startups are overpaying for things like cloud because they are using consumer thinking instead of business thinking.

A huge number. I do a lot of due diligence of companies, and the over engineering on the cloud and costs have to be seen to believed. A common theme now is people use lambdas and step functions in replace of function calls. So a page load or transaction may involve two dozen serial lambda invocations. It is madness.

By way of contrast, a payments provider we explored had four moderate sized boxes running everything. They were about six years old and fully depreciated, but more than adequate to run millions of transactions through a month.

"If you have to ask, you probably can't fit it in your house."

As someone that often has limited time to research viable options to present to leadership, I expect to be able to negotiate pricing at the enterprise level if we want to move forward. What's deeply frustrating that keeps me from sometimes mentioning an option is having no idea what the rough order of magnitude pricing is. Some kind of pricing is deeply appreciated.

If I don't know whether something will fit within the approximate budget for the project and can't quickly get an idea from other research I'm not going to mention it as an option. I'm used to spending about a million per rack but that's for a complete ESX cluster with storage and networking, if I have no idea how alternatives stack up against that it's hard to put it on the table.

I will echo this. A product with no pricing information will often just get no further exploration, especially if a competitor does have it. Don't underestimate the overhead of just having to talk sales before you have a price.

You are right that this sort of sales style has its downsides. I am also not saying there's no value in having something on the page, or that Oxide is always going to be this way. I am just talking about why it is a norm in many parts of the B2B world.

I’m surprised there is anything that you can spend a million dollars on where nearly all the alternatives aren’t “please contact sales”. And even if there was, you couldn’t do any real analysis without contacting sales, because any displayed price would be highly negotiable.

Where I’m frustrated by this attitude is when I just want to buy 10 seats of something and it doesn’t have a price, not at the seven digit level.

I actually find it interesting how many things we (which, in part, I mean I) got wrong about cloud computing early on. Various other things too.

> Our big belief is that elastic infrastructure (that is, cloud computing) should be orthogonal to the economic model (that is, own versus rent).

I’m probably missing something obvious, so take this as a genuine question rather than an attempt to debate, but why aren’t they fundamentally connected?

If you sometimes need 10 servers and sometimes need 100 (elasticity) then with renting you can always have the proper number. If you own, then you have to own all 100 servers.

(not Bryan)

> why aren’t they fundamentally connected?

They are connected in the sense that capacity planning is always a thing. But that doesn't mean that the ownership model is inextricably tied to the deployment model, which is how I personally say instead of "elastic infrastructure." "I am making an HTTP call to request a new VM" is very different than "file a ticket with IT to procure me a new server, install it in the data center, and then send me the keys." The slogan "pets vs cattle" is an advanced form of this change in thinking. In some sense, it is hard to own your hardware yet treat your servers like cattle.

You basically have four basic options:

* deploy elastic, rent your hardware: this is the default today for many startups

* deploy to hardware directly, buy your hardware: this is the Old Times

* deploy elastic, buy your hardware: this is Oxide

* deploy to hardware, rent your hardware: this is a thing, though not nearly as popular as the other three

(I call them basic because hybrid is a thing: you can own usual capacity and then rent more for bursts, or as a fallback, etc)

Now, Oxide is not the only game in town when it comes to "deploy elastic, buy your hardware": this is what many IT departments do. You could, for example, buy some servers, toss OpenStack on there. Oxide's thesis is that doing this is less than ideal, and we can make it significantly better. In fact, it is so suboptimal that many people choose "deploy elastic, rent your hardware" because it is so much easier. And now we're back to Bryan's statement.

Does that makes sense?

It does make sense, thanks.

I'm not Bryan, obviously, but part of the answer here is that 100 servers running at 100% capacity is an absolute upper bound, but most of the time you're nowhere near that. Most of the time few things are at full capacity, which means that you can multiplex your physical hardware resources to increase utilization efficiency.

Even taking into account the cost of skilled staff, AWS was never price competitive with in house, at least in my experience. I simply could never make the numbers work.

In my industry (telco) we had two teams: my team ran our own hardware, the other team ran less than 10% of our workload on an AWS stack that cost as much per month as we paid per year - including annualised capital costs.

They also had double the ops team size (!!), they had to pay for everyone to be trained in AWS, and their solution was far more complex and brittle than ours was.

Assuming Oxide would have been price competitive with what we were already using, I would have jumped at the chance to use them, I could have brought the other team on board, and I think it would have given us a further cost and performance advantage over our AWS based competitors.

This has been my (admittedly small and small size) experience - it's hard to make cloud competitive unless you have something like vastly changing requirements, huge burst needs, etc.

Right - we made a lot of use of spot instances for building and testing. It’s great for that kind of fractional use, for sure.

Where you using cloud scale style purchasing in house or were you on enterprise servers with enterprise switches with enterprise storage?

The cloud is good for many users, especially if they migrate to cloud native system design, but as a telco you would probably have facilities and connectivity which helps out a lot.

Companies like Mirantis, choosing a technology completely inappropriate for distributed systems (puppet) put a bad taste in the mouths of many people.

I implemented OpenStack at one previous employer to just convince them that they could run VMs, intending it to allow for a cloud migration in the future.

As they ran a lot of long lived large instances it was trivial to make it cheaper to run in our own datacenter. Well until I moved roles and the IT team tried to implement it with expensive enterprise gear and in an attempt to save money used FEXs despite the fact I had documented they wouldn't work for our traffic patterns.

Same thing during the .com crash. I remember our cage was next to one of the old Hotmail cages with their motherboards on boards. We were installing dozens and dozens of Netras and Yahoo was down the hall with a cage full of DEC gear...we went under because we couldn't right-size(cost) our costs.

A lot of the companies who save a lot in cloud migrations were the same, having decked out enterprise servers, SANs, and network gear that was wasted in a private cloud context.

Enterprise _ is often a euphemism for we are fiercely defending very expensive CYA strategy irrespective of value to the company or material risk.

> in an attempt to save money used FEXs despite the fact I had documented they wouldn't work for our traffic patterns.

What is a FEX? Feline Expedited eXchange?

Cisco Fabric EXtender. Like a TOR switch but dumber.

We provided SaaS billing services to telcos. We were very successful in our market, but not very big. Just a couple of racks of gear.

Our production workload was pretty homogenous, and we were super cheeky - we’d use previous generation servers to keep capex down. We didn’t even use VMs, just containers; docker-swarm was good enough (barely). Our bottleneck was always iops, so we’d have a few decked-out machines to run our redundant databases. It worked fine, but I have subsequently enjoyed working with k8s a lot more.

We did use enterprise gear, but previous gen stuff is so much cheaper than current gen. So our perf per watt was not great, and we’d often make the decision to upgrade a rack when we hit our power limits, since we rarely had other constraints.

As mentioned elsewhere, we did use AWS spot instances for fractional loads like build and test. It’s not that we didn’t use cloud, it’s that we used it when it made sense.

All of that said, I do suspect the equation has changed - not with AWS, but with Vultr. I’ve deployed some complex production systems there (Nomad, Kafka+ZK, PG) and the costs are much closer to in house. They have also avoided the complexity of binding all their different services together. They also now provide K8s and PG out of the box, charging only for the cost of the VMs - as opposed to the wild complexity of AWS billing.

So maybe I’m coming around.

At my previous employer (which was telco adjacent in a way) we basically came to the same conclusion. Big plans were drawn up to only locate specific hardware in physical data centers and move 90% of the load to the cloud. It never left the planning stage because our VP could never get the numbers to work. Once you cross a certain threshold of services, scale and reliability you’re paying a premium to be in the cloud.

Yes. Example: oil&gas, and telcos. Some of them believe that the public cloud doesnt do what they need it to do. Super-high performance networking for example, IPv6, SCTP protocol, whatever. IIRC Schlumberger built a private cloud on OpenShift and Verizon are a decade into rolling their own cloud built around OpenStack and OpenShift (they have blogged/presented about it quite a bit). AT&T built a private cloud and then tied up with Microsoft to make Azure Operator Nexus (see https://learn.microsoft.com/azure/operator-nexus/overview).

I honestly don't know who is on the right side of technology and economics here. One glance at Oxide or Nexus says these systems aren't cheap. But if those industries are right, and regular cloud genuinely won't cut it, then maybe these systems can find a niche.

One thing that would determine the fate of these platforms is if the major vendors, that everyone in a given industry uses, commit to it. For example, in telco that might be Ericsson, Nokia, and Huawei. I don't know if they have sufficient incentives to do so yet. They would, I guess, prefer to wait for a clear leader to emerge and then commit to only that rather than bet on three or four speculative horses?

There are alleged cost savings elsewhere. I haven't seen a company that's actually realised them. Every company I know of that's gone all-in on cloud is paying a fortune and still has a big team to manage everything.

My company alone has over 150 case studies that document specific positive ROI on cloud adoption. Not every can or will run their own; I’m pumped for Oxide and I think they are going to smash, but some teams really really benefit from Public Cloud.

I think kubernetes has partly been about not relaying directly on all these cloud services anymore. Sure AWS and friends have 200+ services that you could use, and if you do use all of them, it would be hard to move to oxide.

Many applications don't depend on all that stuff, just vm, storage and db. Partly that is by design so the application can be deployed in different clouds. Also older application that moved to the cloud.

Reply on news.ycombinator.com