Hacker News Reader

> I'm constantly fascinated

To call a halt to your constant fascination: they don't all have that problem. They still get the complexity of cloudy things regardless when they use one.

They also get some of the complexity of cloudy things when they run their own datacenter. In the end you find stuff like OpenStack which becomes its own nightmare universe.

YMMV, but more and more I see people moving to k8s to get away from OpenStack, to varying but generally positive success.

There are at least five shades in between on-prem and a managed k8s cloud.

Could you mention three?

colo racks, rented dedicated servers, ec2 / managed vm offerings?

And colo providers solved those hurdles for decades. Let’s not act like the only options are cloud, or build your own datacenter.

My startup hosts our own training servers in a colo-ed space 10 min from our office. Took less than 40 hours to get moved in, with most of the time tinkering with fortigate network appliance settings.

Cloudflare zero trust for free is a huge timesaver

I’m constantly fascinated by people who think they need on prem global infrastructure when the vast majority of applications both have very loose latency requirements (multiple seconds) or no users outside of the home country.

Two datacenters on opposite sides of the US from different providers will get you more uptime than a cloud provider and is super simple.

While some of the complexity goes away when it's on prem in to parts of the US, having to order actual hardware, putting it into racks, hiring, training, retaining the people there to debug actual hardware issues when they arise, dealing with HVAC concerns, etc is a lot of complexity that's probably completely outside of your core business expertise but that you'll have to spend mental cycles on when actually operating your own data center.

It's totally worth it for some companies to do that, but you need to have some serious size to be concerned with spending your efforts on lowering your AWS bill by introducing details like that into your own organization when you could alternatively spend those dollars to make your core business run better. Usually your efforts are better spent on the latter unless you are Netflix or Amazon or Google.

Why is it always public cloud (aws, gcp, azure) vs. "bring your own hardware and deploy it in racks".

There are multiple providers that offer VPS and ingress/egress for a fraction of the cost of public clouds and they mostly have good uptime.

I recently rented a rack with a telecom and put some of my own hardware in it (it's custom weird stuff with hardware accelerators and all the FIPS 140 level 4 requirements), but even the telecom provider was offering a managed VPS product when I got on the phone with them.

The uptime in these DCs is very good (certainly better than AWS's us-east-1), and you get a very good price with tons of bandwidth. Most datacenter and colo providers can do this now.

I think people believe that "on prem" means actually racking the servers in your closet, but you can get datacenter space with fantastic power, cooling, and security almost anywhere these days.

>I think people believe that "on prem" means actually racking the servers in your closet, but you can get datacenter space with fantastic power, cooling, and security almost anywhere these days.

That's because that is what on prem means. What you're describing is colocating.

When clouds define "on-prem" in opposition to their services (for sales purposes), colo facilities are lumped into that bucket. They're not exactly wrong, except a rack at a colo is an extension of your premises with a landlord who understands your needs.

It's a spectrum:

On top is AWS lambda or something where you are completely removed from the actual hardware that's running your code.

At the bottom is a free acre of land where you start construction and talk to utilities to get electricity and water there. You build your own data center, hire people to run and extend it, etc.

There is tons of space in between where compromises are made by either paying a provider to do something for you or doing it yourself. Is somebody from the datacenter where you rented a rack or two going in and pressing a reset button after you called them a form of cloud automation? How about you renting a root VM at Hetzner? Is that VM on prem? People who paint these tradeoffs in a black and white matter and don't acknowledge that there are different choices for different companies and scenarios are not doing the discussion a service.

On the other hand, somebody who built their business on AppEngine or CloudFlare workes could look at that other company who is renting a pet pool of EC2 instances and ask if they are even in the cloud or if they are just simulating on-prem.

There's also a good point here (at least at the top of the stack) about reliability: the top of the spectrum goes down relatively frequently due to its dependencies, but even plain old boring EC2 has much better reliability than services like Lambda.

I think the question people are really interested in is usually "What percentage over my costs would I pay to outsource X?" (where X is some component of the complexity stack)

Which, first order approximated, is a function of (1) how big a company you are (aka "Can you even afford to hire two people to just do X?") and (2) how competitive the market is for X.

Colo and dedicated VMs are so reasonably priced because it's a standardized, highly-competive market.

Similarly, certain managed cloud services are ridiculously expensive because they have a locked-in customer base.

Which would suggest outsourcing components that have maximum vendor competition and standardization, as they're going to be offered at the lowest margin.

> it's custom weird stuff with hardware accelerators and all the FIPS 140 level 4 requirements

What kind of weird stuff are we talking?

Servers for an API for https://arbitrand.com

Essentially very high security and throughput TRNG servers (with cryptographic signing and the like).

Because their arguments are disingenuous.

It reads like propaganda sponsored by the clouds. Scaremongering.

Clouds are incredibly lucrative.

But don't worry. You can make the prices more reasonable by making a 3-year commitment to run old outdated hardware.

> having to order actual hardware, putting it into racks, hiring, training, retaining the people there to debug actual hardware issues when they arise, dealing with HVAC concerns, etc is a lot of complexity that's probably completely outside of your core business

Vertical integration is a widely known and understood business strategy - running your own infrastructure will help you reclaim the cloud margins back for yourself.

You can do it as a one man band or a huge multinational.

I use hotels and other rental offerings, including the cloud. But I when it is advantageous to do so, I buy and own. Even though it comes with maintenance burdens.

You don’t deal with hvac complexity. People are so blinded by the cloud they don’t even realize CoLos are a thing.

“What alternative is there to living in a hotel if you aren’t a carpenter!?!?!”

There are tons of examples where low latency is good for business, even small businesses. I'm sure you've seen the studies from Amazon that every 100ms of page load latency is costing them 1% of revenue, etc. Also everything communication is very latency sensitive.

Of course there are plenty of scenarios where latency does not matter at all.

So you can trade off 300ms of additional roundtrip time (on anything non-CDNable) at a cost of 3% revenue and reduce your infrastructure complexity a lot

Not every business is based on impulse buys. Amazon is a pretty biased sample there.

Is this agreeing with "Of course there are plenty of scenarios where latency does not matter at all." or are you trying to make a point?

That latency is correlated with revenue is not exclusive to Amazon.

And many people who aren’t impulse buying will not stick around on slow sites.

Disagreed, once we're not talking a worldwide shop for non-critical buys like Amazon the picture changes dramatically. Many people on local markets have no choice and will stick around no matter how slow the service is.

Evidence: my wife buying our groceries for delivery at home. We have 4-5 choices in our city. All their websites are slow as hell, and I mean adding an item to a cart takes good 5-10 seconds. Search takes 20+ seconds.

She curses at them every time yet there's nothing we can do. The alternative is both of us to travel by foot 20 minutes to the local mall and wait on queues. 2-3 times a week. She figured the slow websites are the lesser evil.

Your mind is stuck in low cost retail shopping. Even setting aside the likely self-serving nature of that “study”, most interactions are not latency sensitive like that.

When I go to book my colonoscopy on my hospital’s reservation system, I don’t bail out and look for a new doctor if it takes me 10 tries.

There are very few businesses where UX latency at the sub second level matters and the ones that do are not the ones you want to be in.

I've spent a good chunk of my career in realtime audio/video conferencing and am now working on APIs for B2B SaaS, thanks ;)

I also know that user experience of any application suffers a lot when latencies are high. Your point seems to be that there is lots of software that doesn't care about its user experience (mostly because the people making buying decisions are not the people suffering from those decisions) and that's a fair point, but I don't think that is a great business strategy for any software business.

Of course there are lots of scenarios where latency literally doesn't matter at all.

> Vast majority of applications have no users outside home country

Any evidence to back this up. Because on the surface seems like a ridiculous statement.

Most software is internal software used by a handful of people in a certain department of a company. Even in large multinational conglomerates the applications used by every single country subsidiary can be counted using your fingers.

I would say that a large portion of b2b or internal software tends to come into this category. If you are building something for a single business that only operates in one jurisdiction, and you don't do i18n, why bother with global distribution? A lot of b2b stuff covers process that have legal stuff baked in, like tax handling or other assumptions.

Why does this seem ridiculous? There are more national companies than international companies, by a large factor.

Are the majority of applications even developed by "companies"? I'm honestly not sure at all, or even how to go about measuring that. By the numbers, most games on steam are developed by individuals or small teams, even if the bulk of sales are driven by games produced by larger companies. I'd imagine the same is true of app stores, too.

Do you know how many businesses software is used for? Look at every local business you go to and enumerate the things they use software to accomplish.

When they pay their local utility bills, is that international? How about paying their rent? How about filing their state taxes? How about ordering from local suppliers?

Very little of the world is international business. That tiny slice that is just dominates the zeitgeist because it’s international.

For some examples of things well known that absolutely don’t need global data centers:

- airbus and Boeing

- Coca-Cola

- Marriott and Hilton

- the entire US federal government (apart from some maybe military applications)

- McDonald’s

The list goes on forever because it’s literally nearly every business. Unless you’re in real time markets or operating store fronts globally where latency hurts sales, putting up regions all over the world is a complete and utter waste of money.

Making global regions as easy as a click of a button was one of the greatest marketing ploys of cloud providers to date.

“Of course Good Will needs a Singapore data center!? How will we meet our P99 goals otherwise?”

McDonald's absolutely does need global data centers. Even now the kiosks are frustratingly slow here in Europe, can't imagine what a round-trip to US servers would do.

My siblings work there and their work information system is not so bad but definitely it'd be totally frustrating if it didn't run on AWS in EU.

If McD do need a global data centers to show the menu in the kiosks then they should fire their whole IT dept and start from scratch.

There is absolutely zero reason for a kiosk to touch 'a global data center'. No, not even for a payment, because it's just asks the payment terminal if the payment succeeded or not.

I absolutely disagree. The restaurant managers really shouldn't need to manage servers too. They get the kiosk as a service that they don't need to care about and that is correct.

So McDonald's should be dispatching kiosk admins all around the world? That's very much not eco friendly. And of course, expensive... And a total nightmare to manage. K8s and AWS is a night walk through the rose garden compared to that.

> No, not even for a payment, because it's just asks the payment terminal if the payment succeeded or not.

Yeah sure, works great as long as the kiosk doesn't crash during the payment.

> The restaurant managers really shouldn't need to manage servers too.

And they don't. Why the hell do you pull this nonsense?

In case you don't know (looks like you don't) these are Windows machines[0]. Any Windows machine is capable running IIS or even Apache on it. No need for the servers for managers to manage. Just effing serve it locally, if you can't provide each McD with a mini box what is managed by the central IT.

> should be dispatching kiosk admins all around the world

LOL

> K8s and AWS is a night walk through the rose garden compared to that.

You need a sysadmin to teach you how to build robust and resilient local applications and services without being Eco-unfriendly and without racking billions in AWS bills.

> Yeah sure, works great as long as the kiosk doesn't crash during the payment.

Do you understand what it is the payment terminal which is processing the payment and you can't offload reading CC/NFC to the 'cloud'?

And finally, there is already 'IT' infrastructure in each McD: Ethernet switches, UPSes, wireless AP and maybe a controller, menu and order screens at the counter, networked PoS and whatever else. If you claim what restaurant managers manage all these then you don't really know anything about nor retail nor IT. Or just bickering in a bad faith.

[0] and if in your case they are Linux ones... do I even need to continue?

What sys admin? They don't have any. The kiosks are entirely blackboxes managed from the US central. It works if it has a working network. The store manager can manage it from their iPad lying down on a beach. The card terminals send data directly to a bank and it doesn't matter whether the box burns down while it crashes, the payment is recorded.

I built similar kind of self service box (self checkout) for a local store chain. I know the problem well. Main problems are costs, system administration and management. Our solution is just a simple Chrome kiosk mode browser window too. It's a smart solution.

We run it on AWS because no reason not to - simply pushing the SPA to S3 behind Cloudfront beats any other kind of deployment/hosting method. Some backend stuff runs in Lambda, some runs in ECS containers. All data in a managed RDS DB. Easy to use, easy to maintain, easy to upgrade, easy to deploy, easy to scale from 0 to several thousands kiosks...

I used to be a Windows admin (though admittedly the last Windows I managed was 2003 R2). Just the word IIS makes my neck hair stand up.

Especially if it should be holding payment data... Huh, damn. Wow.

You'd need sync to a global service anyways - it's a multi store chain. Your suggestion just makes everything harder and more convoluted - and it's just wrong, the payment terminal has its own network connection and doesn't need the kiosk to work at all. The kiosk just initiates the payment but the rest of it is handled out of it - and the kiosk waits for central service to confirm the payment. It's bullshit to connect the kiosk to the bank, we support many banks and keep adding support for more, and what if it's stolen, should we always be ready to rotate the certs on 1000s devices - the bank doesn't have appropriate api, so it's either that or again, a central service? Nah, we just turn it off in our admin panel.

I also used to be a Linux sys admin way before clouds. I built one of the first cloud services in my country to solve the problems of that. The company ultimately was out competed - but not by traditional hosting. By the big public clouds. Apparently the problems are real.

BTW coincidentally I'm just about to launch a new cloud platform. Nothing fancy really, but it's built on dedicated servers for performance. Last 3 weeks I spent working on stuff I could've done with 100 lines of Terraform/Pulumi with AWS. Maybe a proper sys admin like you could teach me? I am not happy with it at all, it's a major headache and I'm considering a hybrid setup because I just don't want to lose sleep over customer data and site availability.

You're absolutely right.

The only motivation would be latency, but you could have specialized services that run at the edge, if that's so important, for example if payment verification should take 500ms instead of 3000ms.

But you could also just rewrite the protocol to have less back-and-forth sequential data exchange, which is a smarter approach.

If their kiosks need to talk to a server, they fucked it up. This is the problem with idiot cloud architects.

Not sure how how you could jump all the way to running your own Asian datacenter from my post. A bit amusing though :). I even wrote that it's worth running the LB/DB in the Cloud?

Oh it was more of an addition to your point about "reducing complexity of managing the most frustrating things like the load-balancer and the database, albeit at a price of course". There is a whole mountain of complexity that most software engineers never think about when they dream about going back to the good old on prem days.

Alright, just feels like taking a bit too far into the exceptions. Even back then only large companies would consider that. Renting servers, renting a server rack (co-location), or even just a in-office server rack for what would be a startup today.