Move Back to Datacenters
rakkhi.substack.com
rakkhi.substack.com
The security points, which the main points hinges on, seem hyperfocused and in most cases misguided.
Redshift is not internet facing. If your Redshift is internet facing, you've messed up somewhere.
The CPU attack example given is for AMD ZEN, which the most common AWS workload CPUs aren't. Further, the benefit of using a cloud provider is that they put mitigations in place for most exploits, whereas running your own stack means it's now on you; running your own stack does not excuse you from having to put mitigations in place.
In the examples again, the speed to market's problems are a reflection of your organisation, not the cloud.
Stepping back a little, I'm thinking (as terrible as it is) that this is a case of blaming the tools, but never yourself; a lot of the problems the author is seeing seem very specific to their observations but are a poor use and poor understanding of AWS in general.
Overall not a great article, with a headline designed for people who already dislike AWS/GCP/Azure.
I am hugely in favour of AWS for web startups I am involved with because 1) I know AWS very well, 2) they provide hundreds of thousands of euros/dollars of credit to startups (I think all the big cloud vendors do at this stage), 3) it's useful to have architectural flexibility, and 4) they can take on a bunch of the security and availability burden that can otherwise be very labour intensive in a small company.
Also you can pry S3 and RDS Aurora out of my cold dead hands!
If you are a startup and/or have a limited/specialized business scope* Cloud can be the best choice.
For organizations of 4000+ people that manage 100+ applications (from different vendors) with a 90+ people IT department this is maybe not the silver bullet. At this scale you need to keep complexity and security manageable and a more traditional redondant datacenter architecture might be less of a headache than the cloud.
*PS: What I mean by "limited/specialized business scope" is for instance Twitter/Meta. Huge scale but heavily specialized around their main products and develop most of theirs application in house.
This completely misses the business case for cloud, and is why these sorts of analyses don't convince that many business folk, and why cloud use continues to grow fast.
The business case is that someone else is doing the infrastructure management for you, at commodity prices, but you still have complete, direct control over "your" resources and ability to easily automate everything.
This tends to save much more money than any % of your CPU, memory, and disk budget could possibly achieve.
It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies!
For many companies, the "just host it in a datacenter" argument is no less silly than these scenarios.
Which is great if you have limited capital (bootstrapping, new startup, etc), or are the CEO of a large company that wants to show better cash flow to earn a larger bonus.
In reality, the idea that the cloud somehow means you can outsource your infrastructure team rarely works out in practice.
At best you’ll get much more expensive developers having to spend time on infrastructure which might look like a couple of fewer salaries to pay, but you’re likely now paying for it through dev time anyways.
Many business people believe that moving to cloud can reduce the headcount needed for managing the infra, but that is usually not what's happening. You will still need more or less the same amount of people to patch the OS and configure the networking, but with a slightly different skill set -- instead of Cisco IOS commands, they now need to deal with AWS transit gateway.
Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you. There are also third-party companies that offers this service. Redundant power and air conditioning are the responsibility of the colocation provider and those are covered by the SLA. Those data centres do have generators on-site.
That said, configuring you top-of-rack switch is usually not covered by the co-location contract. This need to be done by a network engineer, usually the same person that would manage VPCs.
But we don't ever "patch the OS." Our cloud provider does that for us. Upgrades and patching are automatic within the maintenance window. We deploy containers, the OS is just a platform layer that the cloud provider manages.
As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.
At larger companies, it can be harder to make this comparison, since you'll typically have a mixture of cloud and onprem. But if you look at teams with products deployed in the cloud, they'll often have much more frequent deployment, with better automation, and much less dependence on the always overcommitted infrastructure team(s).
> Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you.
That's not very compatible with infrastructure as code. You're essentially saying "keep doing things the old inefficient way, it's not really that bad."
There are all sorts of failure scenarios that a managed cloud system can cover from completely automatically within minutes, that would require frantic calls to remote hands in the old world. And even when recovery isn't automatic, it may still require nothing more than some pointing and clicking to recover.
> instead of Cisco IOS commands, they now need to deal with AWS transit gateway.
> This need to be done by a network engineer, usually the same person that would manage VPCs.
These aren't comparable at all. Unless a company has some sort of very special networking requirements, they can set up VPCs with some pointing and clicking in the cloud provider's web UI. This is a far cry from the level of knowledge required to manage networks at the hardware level.
If the PaaS works for your scale and budget then it’s perfectly fine. I just want to mention that it is probably no longer the case for scaled-ups.
But IaaS has been around for nearly 20 years now, and containers for at least 10 years, so I tend to assume people have got that message by now, especially on HN.
Re scale, you can run pretty high scale on managed Kubernetes clusters. E.g. GKE supports up to 15,000 nodes per cluster, hosting over 3.8 million concurrently running pods. Multi-cluster ingress is also supported if you need to run multiple clusters.
In addition to the servers.
In addition to patching the routers.
Our build pipelines rebuild container images for every deployment. All you need to upgrade some dependency is to set a version number in the Dockerfile.
There's no "too" because that's the only place we deal with that kind of version dependency. (Not counting dependencies internal to a service, like libraries.)
This is absolutely not true. Unless your cloud services are limited to function-as-a-service and serverless/containerized applications,you are indeed responsible for "patching the OS".
This kind of thing is explicitly covered even by AWS' intro to AWS courses, namely in its shared responsibility model.
> As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.
Sorry, your claim is outright unbelievable. I've never seen a company who owned any web application that only had "one or two guys, possibly even part-time, dealing with their cloud requirements." Unless the whole company is only "one or two guys, possibly even part-time", your claim simply is extremely far-fetched and blatantly unbelievable.
Yes, I'm talking about containerized applications, deploying to fully managed environments like EKS, Fargate, GKE, Cloud Run. If you're not using containers then yes, it will be more difficult to achieve what I'm describing.
I've been the one part-time guy for three different SaaS startups with funding in the $20-30m range, and globally distributed dev teams. I'm a software architect and dev primarily, so setting up the cloud platform is just a side activity.
For one of those companies, I was literally just a part-time contractor. Once it's set up properly, it should be easy for regular admins to operate, and devs can just follow the templates for configuring a service. It's not rocket science.
All the stuff you're imaging is so complex is exactly the stuff that the cloud lets you delegate to the provider. But you have to be willing to do it, you can't stick to the old way you've always done things and expect the cloud to make anything easier for you.
It's also a gross misrepresentation of the initial statement. Having a part time employee setting up a few containers to run in A container orchestration system is absolutely not what "setting up the company's cloud requirements" means. It's a blatant attempt to oversell a couple of clicks worth of work as covering a company's whole cloud requirements.
I think this is an unfair analogy.
It would be more akin to saying instead of hiring Ubers and taxi's and spending several tens of thousands a quarter, buy a couple of cars for the company. But instead of needing a single person to onboard people onto the company's taxi account and paying the invoice each month, you instead need someone to buy the cars, ensure they're taxed and certified, create an internal hiring system, etc.
For some companies it does make financial sense to have your own fleet of cars, in others it makes more sense to get Ubers.
It's exactly the same as cloud vs not-cloud. If you're not taking advantage of the cloud (e.g. serverless technologies) why pay that premium when a small number of dedicated machines and a simpler architecture/stack will do the job better and cheaper? They're only as difficult to maintain as complex cloud environments.
Infra engineers needed to run your "managed" cloud stack cost much more than generic sysadmins for commodity open-source on-prem solutions.
Where are the people with the skills to do this work? I wrote a whole post about it, but it is not like people with that knowledge exist in troves just waiting to be hired.
There are no path to learn this stuff either. Do we really expect every single company out there to build teams of sysadmin and operators from a non existing pool of talents?
It is easy to say that people can move to onprem, but orgs can barely hire the talents they need already. Sure let put even more pressure on a limited supply.
E.g., there's probably three orders of magnitude more engineers familiar with Postgres out there in the world than with DynamoDB.
Nope. Hence you learn the RDS api and use it.
The amount of engineering needed just to migrate droplets live is enormous - it’s an art (according to a friend in hypervisors), and that ignores everything else regarding hypervisors and security.
Scalability is also a thing you get “for free” with serverless computing.
Even if you feel that is a serious obstacle (which I personally don't) you can buy a managed solution (such as an Oxide rack) and use that on prem.
I wouldn't often advocate to do colocation or on-prem since that indeed comes with a whole set of headaches, but renting dedicated physical servers like we do offers a lot of flexibility, with very little overhead, at a price/performance ratio that makes AWS look like pure extortion.
I think if you try to host in US to have lower latency to many customers, and with some less cheap and more reliable server grade network/hardware, it is likely cost will be higher than for hetzner.
* Some software is only available as SaaS anymore
* Jacking up prices to ridiculous levels for on-prem licenses (to favor their SaaS offering of course)
* Intentionally knee capping on-prem software feature wise
* Stifle development of the on-prem product
* Force you to use some parts of their cloud services to make other things work
* Dark patterns and endless nags in software to push people to use some part of their cloud services
* Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex
* Slower response from the software vendor in case of security issues
* Exporting data from the cloud and importing it on-prem is impossible
And I can make this list go on and on and on... My point being, for small time firms that don't have the resources and solely rely on commercial software, moving back might not even be an option anymore.
Or unavoidably. We have a SaaS product that's also available onprem - because we have some big enterprise customers who won't accept anything else - but there's a bunch of stuff we take advantage of in the cloud that we can't easily replicate in arbitrary onprem environments. So the onprem software ends up "kneecapped" just out of sheer pragmatism. There's not a single missing feature onprem that isn't because it would be too much effort to implement and support it.
> Stifle development of the on-prem product
Similar issue here. We have finite engineering resources, and onprem is a fraction of revenue.
> Force you to use some parts of their cloud services to make other things work
See first point.
> Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex
See second point.
In short, many of the things you're saying are indeed disadvantages of onprem products today, but it's often not the result of some sort of "intentional kneecapping" or "dark patterns". In my direct experience, it has never been the result of that.
In any of those cases, we either have a choice to implement and support some onprem alternative, require the customer to provide the necessary service locally (which is not particularly effective for all sorts of reasons), or simply disable features onprem.
In the cloud, we get significant benefits from using managed services. Onprem, we not only lose those benefits, we have to pay an additional cost in terms of making our code support both environments and building out the deployment and support of the services in question.
It's really exactly the same issues that make cloud attractive to businesses in the first place.
Turns out in the two decades I’ve been in my industry everyone else has gone to ‘the cloud’, and the tooling has gone with it. Every product or project I looked at was skewed entirely toward physical host agnostic and/or single service reliability agnostic management.
So not only is in an issue with going back on prem, there’s entirely unserviced customers that cannot adopt the tooling given the direction it’s been developed.
This reminds me of the idea of "Service as a Software Substitute" (SaaSS): https://www.gnu.org/philosophy/who-does-that-server-really-s...
> What does it mean to say that a given computing activity is your own? It means that no one else is inherently involved in it. To clarify the meaning of “inherently involved,” we present a thought experiment. Suppose that any software tasks you might need for the job are implemented in free software and you have copies, and you have whatever data you might need, as well as computers of whatever speed, functionality and capacity might be required. Could you do this particular computing activity entirely within those computers, not communicating with anyone else's computers?
> If you could, then the activity is entirely your own. For your freedom's sake, you deserve to control it. If you do it by running free software, you do control it. However, doing it via someone else's service would give that someone else control over your computing activity. We call that scenario SaaSS, and we say it is unjust.
It sounds a bit preachy like some of the FSF/GNU stuff sometimes does, but at the same time I can't shake the feeling that they're somewhat right and that those arguments have also fallen on deaf ears most of the time.
For my own personal stuff, I just self-host everything I need in containers and mostly use free software, so I can (and have) easily move hosting providers or host things myself in my homelab, have full backups and so on. But at the same time, it definitely takes a bit of work to figure everything out and not everyone wants that, especially in environments where you can point a finger and go "the vendor is having some issues, we're off the hook".
Plus, I don't think there are many alternatives to certain software out there, like MongoDB (maybe JSON in RDBMS if you really need to, but the API would be different then), or S3 (MinIO has a license that's too strong for some, Zenko feels abandoned, Garage is the same as MinIO, maybe SeaweedFS but that doesn't have full/mostly full API compatibility from looking into it).
Honestly, it sometimes unironically feels like building boring software that relies on MariaDB/PostgreSQL and maybe Redis/RabbitMQ is the right answer. Not quite sure about blob storage, though.
That's a fair point!
I mean, there are some options out there like Odoo, or stuff like Nextcloud/Collabora, OpenProject and you can run your own mail server etc., but those will increasingly feel like bad ideas for someone who needs something that "just works" and also has deadlines to manage.
It's a bit like "Nobody got fired for choosing IBM", except nowadays with Microsoft/Google/Amazon cloud offerings, the Jira's of the world and so on.
Secure computing, but in the way that's always been done: secure from the customer. Not secure for the customer.
The rest is a bunch of FUD - I spent years going through these points with some of the worlds best security teams to secure some of the most systemically important workloads. These arguments are fairly tired.
I’ll tackle another one - speculative attacks. First, you certainly can get bare metal exclusive access to hosts. But instances move around the broader infrastructure of an AZ, even if you’re using something like placement groups which only assure a local affinity. The chance a bad actor can colocate in the same physical device as your workload and successfully attack through side channels is vanishingly low in larger regions. To target anyone specific you would need to do such an enormous fishing expedition that it’s impractical. Further cloud providers aren’t insensate to such attacks and accounts that are doing that sort of topological mapping are easily detected. A better solution is to simply cycle your instances periodically to migrate your workloads around. For very sensitive workloads where the extraordinary unlikelihood isn’t sufficient, just get a bare metal instance.
I don’t dissuade anyone from running data centers. But I’ve yet to find anyone running back.
Big claims require big evidence. How did you end up with these superlatives?
Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it?
Not all internet exposure is equal. Moving out of cloud often makes sense, but security isn't the right motivation for it.
Is that AWS security engineer looking specifically at how your systems interact with DynamoDB? And how you have specifically configured your DynamoDB instances?
And someone rolling their own infrastructure isn’t building out their own custom software to do it. They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software.
Now, it may be true that in recent times with the amount of money made in the cloud a lot of these vendors are using carrots and sticks to push you to their cloud offerings (often by making their on Prem offerings worse), but let’s not pretend that on-prem software is two cups tied together with a string or that moving to the cloud somehow alleviates the need to have in house expertise to ensure your cloud software is running correctly and securely.
>They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software.
Of course. Even with well designed software the risk is much higher for the end user though. Deploying a cloud load balancer is pretty idiot proof. If I do the same via local nginx...configuration files, local firewall, file permission, maybe my distro repackaged it, did I set up the user right, am I updating it regularly, what about my underlying operating system security etc. And on top of that it's a once off configuration...it isn't battle tested by thousands running the same load balancer.
It's not that it can't be done, but you do pay the price in complexity and engineering hours.
>And how you have specifically configured your DynamoDB instances?
Well yeah - I'd expect them to have thought through the possible permutations in reasonable detail, which is feasible given the limited customizability. If you're self deploying the possible combinations are endless.
It's a bit like testing on android vs iphone. You can reasonably test against the 30 or whatever common iphones, but you're never testing against the 1000s of funky android hardware combinations out there.
.gif^$domain=*.substack.com,image
.gif^$domain=*.medium.com,image
This prompts the question of "So, what are you selling this year?"
I'm sympathetic to on-prem and datacenters, but maybe all the reaction GIFs are distracting the CIO/CTO from the new fine print?
We switched to Azure in 2018 and never looked back.
Sure you trade security (do you really, though?) in exchange for:
- not needing to head to the DC because a power supply failed and a rando who was in the cage never plugged in the redundant one
- not having to be way over-capacity in scalability, or suddenly under-capacity and emergency ordering some more 1Us
- not sifting through eBay to buy a spare hard drive that one of your boxes from 2011 needs but dell no longer makes
and the list goes on ad infinitum.
This problem has been solved. It's time to move on. Any time a company spends tinkering with their stack is less time spent delivering something of value.
Remote hands set up the hardware for me and replace failed hardware. The service is so rarely needed that it's basically free.
In recent years I've started to rent the hardware I want as I want more frequent changes. I tell my host what hardware I want and they'll acquire it and install it.
If you're running very old hardware that you can't even buy replacement drives for then you are doing something very wrong.
Need to scale fast? Then I don't have to wait for servers to arrive, I use their cloud offering that's in the same network.
I agree that it's a solved problem, but the solution to anything somewhat permanent is not the cloud. It's not really any harder to run your own hardware.
Also if you're renting your hardware... for a subscription ... that sounds an awful lot like the cloud with extra steps.
They’ve all drunk the kool aid and are cloud true believers.
The cloud makes a lot of sense for startups for a variety of reasons. Low capex, easier to get started, you’re willing to pay a premium to not have to deal with certain issues, at least in the beginning, and your costs are likely not very high anyways because your usage is relatively low.
You have a lot of business travel wherein your employees currently stay in hotels whenever they visit major cities? Don't use hotels, they are expensive! Build and operate your own company owned full-service apartments in those cities.
For most companies their software is their business. Take away the software and the company stops functioning.
Their employees aren’t just visiting their software every few weeks or so. They’re using it every day from morning to evening.
There is no travel involved. Your software is your home.
So really a better analogy would be renting vs buying your home. The rental comes with the furniture and for several things that might go wrong the landlord will come and fix it, whereas when you own the house, you have to buy the furniture and arrange it, and if anything goes wrong you are responsible for fixing it.
Of course, the analogy breaks down in many ways because when you’re on Prem you are buying established software with a support license so if something does fail the support license should provide as much, if not more service than you would get on AWS.
The entire analogy is completely pointless because it adds a layer of indirection for no reason other than for some reason tech people absolutely love making analogies to physical objects.
Having worked on these for years, building solutions, diagnosing problems and so on I'd risk saying we have reached the point where the cloud is slowly getting more complex than traditional setups. In theory, it should be simpler, right? A Transit Gateway is a virtual device so easier to configure than a Cisco router, right?
The problem is, as the cloud providers offering gets richer, with time it has to be more complex in response to various customers' requirements, so basically you need to learn both: traditional Linux operations and networking, and a whole new class of interactions between dozens of services with hundreds of API calls. It quickly becomes overwhelming. IaC obviously helps but anybody who has maintained a large repository of CloudFormation code will tell you its not a panacea.
And then a million small quirks that basically come from you not owning the infra. E.g. Amazon is pushing the Control Tower as a model for new setups especially for larger orgs. Guess what, this thing has almost no API. Last time I checked it allows you to maybe check a state of a control. This means you need to set it up manually! In 2023! After 5 years of development! And if you add an OU, you need to register it (you guessed it - also manually!). Say you want to create a dozen of VPC endpoints - they won't have names because there is no API, you need to name them manually. And so on and so forth. Why? because you are living in someone else's place, so you you are totally dependent on them.
I am lost in trying to start off a sensible / official / approved AWS setup but I can use some years old bash and salt s riots to setup my raspberry pis sitting there in front of me.
It's not just me (maybe mostly me) but there is a new layer of complexity - I can learn and understand the FOSS but trying to learn AWS means piercing the layers of marketing that are creeping in - one of the huge advantages of a FOSS readme is it comes from the head of the person who designed the system i want to use. not a product manager who probably misses the technical point anyway
So yeah. I am wary.
CloudFormation is one of the worst IaCs out there, so it's not really a good example.
One really nice model is managed Kubernetes with IaC/GitOps via tools like ArgoCD or Flux CD. This eliminates the need for things like CloudFormation, Terraform, Ansible, Puppet, OS version and patch management, etc. - most of the stuff that those tools are used to manage is replaced by the cloud provider's management of the cluster nodes. This lets you focus on the application layer, which is where the business value is. IaC in this context is pretty seamless. It also makes you much less dependent on the cloud provider's specific APIs - Kubernetes becomes your interface to all that.
This is just romanticizing things. In every deployment - whether in it's "in cloud" or anywhere else - there is always the quick on off change that someone makes. Probabably for a valid reason, e.g. solving a production problem quickly. Chances are high that will go unnoticed, until another problem manifests (and that is hopefully not a security issue!).
I would argue that in cloud setups the chances for that to happen are actually slightly lower, because teams are incentivized to use immutable and declarative infrastructure. And that there might be an audit log in place which tells what changes have been made - although that still requires people to look at it, which again happens only in the case problems show up.
The article pretty quickly moved off that point. And it seems to at points be saying that making it hard to setup infrastructure is a feature, will help the bottom line. Which has some truth but at enormous emotional cost to your teams.
I do think this shift needs to happen and I appreciate such lengthy sets of concerns being brought out. But I'm pretty lukewarm on this analysis.
While I agree in principle, these functions don't exist in thin air. You will often want to store the results somewhere. And then protect them. So with time you are almost replicating a lot of infrastructure. And if you are a big org you will probably want to enable SecurityHub and other security/governance services like AWS Config and things get expensive again.
Allthough I get the idea of paying others to run your servers (I do so myself), I am somewhat puzzled that people so readily want to move on from the option of selfhosting. Knowing how to do things yourself has its own value. This reminds me a bit of developers who can't write anything useful without using a framework. There is life outside of frameworks and life outside of the cloud.
These are the technological foundations of being able to do anything we are doing. Granted, it can be useful to outsource these parts, but that doesn't come without it's very own short and long term costs, it strengthens some market asymmetries that you maybe don't wanna strengthen, it let's your organization forget to do things that are the basis of your business and so on.
I don't say everybody has to run their own infra, just that the work of doing so can pay off in different ways — and whether these trade offs are worth it depends entirely on the nature of your endevour.
In any case ”don’t use serverless functions” is a very far goalpost from “move back to the data center”.