I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.
Hum... I'd say it's much more reasonable to look at the ROI. Making investments to supply peak demand or to hedge against rare risks is perfectly ok.
Once your past a few racks of equipment, you have generator tests and service appointments, redundant AC, Redundant UPS, dual power to each rack, etc. Dual Internet connection, and a link to your other server room that you use for DR, etc. The costs and complexity quickly escalate after a server or two.
Absolutely, nevermind managing OS upgrades and needing to use configuration management to mitigate against drift. It took a lot of effort to make sure that each server was not a special snowflake that could not be reliably reproduced.
Also, dealing with vendor warranties, and being on hold with HP (or whoever), then assuring them you're running the latest firmware.. please for the love of god just replace the failed memory/disk/cpu.
I found the sheer physicality of computing infrastructure to be a source of exhaustion and burnout. It's a big part of the reason why I'm a software developer now :)
This is not always the case. For computationally-focused workloads like the OP describes, without direct customer interaction, it may be reasonable to accept risk of downtime in the event of failure. If you are doing computations that take weeks to complete, and you checkpoint regularly, does it really matter if your computation finishes on Saturday morning or Monday morning? If not, you can probably accept 6 hours of downtime once every couple of years and eliminate all of the redundancy overhead described above.
In HPC, the general rule of thumb is buy your hardware if you can be sure you'll run compute on it more than ~40% of the time.
Mind you this was only about 4U worth.
No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.
A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t.
The cloud actually requires more investment for large organisations. Previously you might of only had a handful of sysadmins but now you have a large dedicated platform team doing devops type work building your own abstractions/paas on top of your cloud.
The advantage the cloud has is flexibility. You don’t need to go through a lengthy process to acquire hardware. Likewise cloud services are disposable, no longer need something? Hit the delete button.
I think the more PaaS like services such as Heroku, Lambda, Fargate, Google Cloud Run do better realise the less maintenance story but not cloud generally.
Completely agree here. Running an EC2 Spot instance + ebs volumes for 730 hours a month is a total waste of money, but running RDS behind fargate and ECS with an ALB is likely to save you time and money by month 2.
Owning a BM server different toil - server parts fail, the network it attaches to needs control and it fails too, firmware needs updating more regularly than ever, DC space needs managing over time etc. For a small number of machines maybe this is NBD. For thousands of machines this is just grunt work which while automated, still needs change management and control - rebooting the whole fleet in the middle of the day definitely opens doors in your career.
Doing everything you did in the DC in the Cloud is absolutely the worst way to adopt Cloud. Owning an OS is a non-goal, you’ve gotta climb to a higher abstraction - workloads, and quit caring about machines. This is where most companies fail.
Ani. If the hardware you’re running on is dying. In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed.
Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal you got to shut down and replace.
The cost of managing hardware is gone with ec2.
Software on the other hand yes that is the same amount of effort.
If you don't have sysadmins and network admins with the right experience, you can easily find yourself in a bad spot with single points of failure, servers that can't be easily replaced, oversubscribed PDUs, misconfigured switches/routers... and any number of other problems that aren't occurring to me right now.
The crossover point where cloud is more expensive is really low even if you have zero in-house experience. Exactly where it is depends on your amount of egress, as that is where AWS in particular really takes advantage of you.
We went dedicated early and while it didn't make sense at the time we now run at way lower cost than the competition could dream of.
You mean, you reimage? That is the slow step, you reimage, and plug the new server. Wait a bit, and your service has one more server.
> With bare metal you got to shut down and replace.
You take the disk out and plug a new one. You don't turn things off because of a disk.
No doubt, those are costly. They are also rare (disk failure is less rare, but still rare).
No. When you /stop/ an EC2 instance, and /start/ it again, it moves. You do not need to reimage. This is even requested from AWS when they are having hardware failures and need to move customers off so they can decomission the hardware. They request you stop / start the instance, if you do not do it by the due date they do it for you.
> You take the disk out and plug a new one. You don't turn things off because of a disk.
If you have a storage array sure. But if you're getting bare metal hosting from a provider, you're not always getting hot swappable storage arrays.
> No doubt, those are costly. They are also rare (disk failure is less rare, but still rare).
It was 1 example, obviously there's many different hardware issues that can go wrong.
If you have any server-level hardware bought in the last 20 years or so, it will have the drives in hot-swappable bays. If you then choose to not set it up in RAID, it's just incompetence.
> If you have a storage array sure. But if you're getting bare metal hosting from a provider, you're not always getting hot swappable storage arrays.
If you're getting bare metal hosting from anywhere including your own colo, you have failover and the ability to order replacements while your system is still running. This is only an issue if you're architecture is fundamentally flawed, in which case you're likely to mess things up whether you're on bare metal or in a cloud.
> we tied Hetzner servers into our private cloud layer, and migrated containers and shut down servers as it fit
Building your own services on top of AWS is always going to come out more expensive. EC2 + EBS volumes alone are going to be more expensive than going with hetzner (particularly if you're not looking at reserved instances, and not utilising spot for burst). You mentioned that you are building your own private cloud layer and migrated containers; the cost of building that out in the first place is likely enormous compared to building and running on top of fargate.
At the time we didn't have a choice, as nothing like Fargate existed, but today it's also easier to do setups like the one we did. It mostly involved rsyncing base images over, rsync and a super simple storage service for backups, a LDAP based directory service, and and a thin layer over vzctl (first) and docker when that became an option, coupled with a VPN setup to tie our locations together, and a reverse proxy setup that did dynamic lookups in our private DNS fed from LDAP.
It is hard to do as a multitenant public service, it's trivially easy to do as an internal tool that needs to support only exactly what you need.
I've built out setups like this for a number of clients since, and it's typically 1-3 months of work to automate pretty much everything depending on complexity, and so it pays for itself quickly from a very low scale.
The first company I did this at wouldn't have been profitable at all if we'd relied on AWS
Not really, this is what RAID is for
Or you can fork out the extra cost to have hot swappable.
If you're saying: "I dont want to pay $300 / month for EC2 instances at AWS when I can get the same hardware specs for $150 at X"
Chances are you're getting a shitty old desktop or barebones rack that lacks features like hot swap and raids.
Do you believe you're getting redundancy when you go with hetzner and a cheap desktop grade processor, ddr4 non ecc memory, and a consumer grade SSD?
> Do you believe you're getting redundancy when you go with hetzner and a cheap desktop grade processor, ddr4 non ecc memory, and a consumer grade SSD?
Irrespective of how much I'd skimp on the hardware, I always have a HA setup, including on EC2, so it doesn't matter in any case.
Definitely not true. With a dedicated server, you need to handle backup and security yourself.
Any reasonable dedicated setup will involve imaging your server, and so the OS image is not something you need to back up - if it fails you reimage. If you even store the OS image on the server at all rather than network boot.
For you to lose your OS image with a dedicated server only takes your HDD to die.
For you to lose your OS image on EC2 (where you made a snapshot of your volume in one-click) would take a lot of shitstorm to happen at AWS -- as I presume that they backup across sites.
Only if you don't have a backup.
Why in the world do you think anyone would store their only copy of an OS image on a single server?
For systems I set up, to start with, the OS is mostly immutable, booted and updated transparently to match a master image. If it gets destroyed, we just image a new server. The applications all run in containers, based on images stored on replicated file servers. If they get destroyed, we just re-deploy on a different server (in fact, automatically redeploying is trivial).
Only the application data is unique to running servers, and that needs to be backed up just as much whether those containers run in a cloud environment or locally, and again it's trivial to have automation in place for the backup and re-deployment of that. Been there, done that many times.
> For you to lose your OS image on EC2 (where you made a snapshot of your volume in one-click) would take a lot of shitstorm to happen at AWS -- as I presume that they backup across sites.
For me to lose my data on any bare metal system I've run, multiple servers in at least two different data centres operated by at different companies would need to fail at the same time. This is not hard to set up, and it's a one of setup. You then need to test your backups, just as you need to with AWS - an untested backup is not a backup.
But your assumptions of failure scenarios is also flawed. You need to protect against e.g. disgruntled employees, hackers, bugs as well. If you rely on the same security to protect your backups as your main setup, you don't have a backup.
EC2 is great when you can justify the cost, but it does not remove the need for a proper backup policy and processes to test them.
I used to manage multiple racks worth of servers on top of managing the 1k containers running on them, maintaining the (pre-kubernetes) orchestration software I had written to deploy containers to our servers, and still had time left over to spend the majority of my time on the architecture and project management of new projects for clients.
It also should not take anywhere near a day per server per month - if it does, then in a disaster scenario it means you're unable to recover at a reasonable pace.
It is additional time because the cloud handles the whole class of "your hardware died" problems.
I've set up more than one hybrid setup where you didn't need to know if your workload was running in AWS or Hetzner or somewhere else, so we could use AWS for elasticity and Hetzner to keep cost down.
This is not a hard problem. And if people don't have the right skills in house, it's easy to outsource this (I for one used to make my living of automating setups like this and operating them on a retainer basis).
I don't think anyone is claiming it's a particularly hard problem. It's the opposite. We're talking about 1 hour a month. Even that little effort still ~$200 a month and a $200 a month managed DB instance is pretty beefy.
Been there, done that, many times, and I know exactly what it costs people, because I billed by the hour for it, and your cost assumptions are just way off.
If this was the case, my billable hours when I was doing contracting would be about 1/10th of what they were. I lived very comfortably of troubleshooting for teams who had gotten themselves into a thorough mess with this attitude. In fact, I earned more from the teams who insisted on cloud setups because they rarely understood the operational issues with it, whereas teams who chose dedicated servers generally thought about operational concerns more.
> Moving from 1 server to 2 is an architecture problem.
Moving from 1 container to 2 is an architecture problem.
Moving from 1 server to 2 running those containers is an architecture problem with well established known solutions.
E.g. for starters you're assuming no containers. But putting the application in containers and leaving the host OS only for basic infrastructure setup is basic practice today if you're running your own servers.
Once you've set up a network boot source (tftp etc.) to network boot of an installer with a suitable setup script that you can trigger via IPMI, and a directory service and a basic orchestrator (be it Kubernetes or something else) on your network, it doesn't matter much if you have 1 server or a 100 - they come up and you put containers on them, and they look no different than a cloud service to the devs.
You're right that it's easier to take shortcuts if you have just a few servers, but it's just as easy to take shortcuts with just a few cloud instances - the number of pet containers I've seen over the years is terrifying.
> The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run
Which is why you don't provide ssh access to the host servers to anyone without an understanding of ops concerns, log everything, and do all updates via an automated setup of your preference, and why you regularly recycle the containers whether you run a cloud environment or dedicated servers.
The reality is that cloud systems do not at all make you immune to this - I've done year long projects to regularise AWS setups that were full of undocumented manual changes to bring everything into a terraform config for example. Often they're worse, because there are a whole lot of unobvious places to look for extra bits and pieces.
> and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway.
Nobody here suggested a monolith. Nobody is suggesting you forgo redundancy. The. The point of this is that often you can put clear upper limits on the computation you will need to do for either your system as a whole or for a given subsystem, and you can guarantee that you will never need more than one server for a given part of the system.
E.g. a real example: I've worked on a system that did some processing of data about companies. We know this will always fit on a single system because the population growth of humanity is slower than the performance growth of a server and the total number of companies worldwide fits on a single system with several magnitudes to spare today, and the type of companies we were interested in is just a subset. At this point, if you architect a system like that on the basis of assuming you will need to resize, you will risk making choices that makes you far more likely to have to resize. E.g. all of the data I'm talking about can easily fit in RAM on a relatively moderate server now and forever, but the moment you start planning for partitioning the data you have added orders of magnitude of performance overhead for communication.
Properly assessing which parts of a system needs to be able to scale is at the core of architecture, and architects (or devs; it's terrifying how many places lack anyone with architecture experience) who are just planning for infinite scalability for everything is huge red flag to me. It usually means they don't understand their system. For a startup, especially, this is an existential question - preparing for unnecessary scaling has killed many startups.
Definitely not as clear-cut as you seem to imply.
Thinking thru the factors, many seem obvious, but are often forgotten/ignored when comparing rental or IaaS costs.:
Space is a fixed cost driven by market rates and maximum Server Capital Costs i.e. floor space. Failing to fill the room increases your cost/server efficiency. Pretty common to run out of thermal/power before you run out space, as equipment efficiency increases through the lifespan of the facility.
Server and Power costs scale together, carry a minimum cost for keeping machines on, and vary based on utilization. Again if the servers aren’t doing work, your efficiency ratio will drop. Larger space typically have pre negotiated power commitments too - failing to consume that carries fiscal penalties. Servers unit costs are fairly cheap, storage not so much. Full utilization throughout capital/lease lifespan is the goal - anything less increases relative cost/server.
Network costs scale with Server Costs, and vary again by utilization - minimum invest rules apply, all servers need at least one network port, as well as upstream Core/TOR/Miniswitch gear. The network gear lifespan is typically longer than servers, but shorter than facilities. It usually incurs annual support/maintenance charges too. Bandwidth charges are variable as expected.
People costs scale with a step-function and numbers driven by minimum coverage requirements, task complexities, and level of human toil. Performing any task on a device by-hand is expensive in most markets - touches on tickets, change management, task time etc. Fully burdened S+R in Western cultures is typically ~2x the salary - a $80K employee probably costs around $150K by the time all the workplace costs, taxes, and benefits are paid. Network folk are typically premium resources compared to DC Ops. Sustainable 24x7 coverage looks like a staff of 3-4 people.
Even in the pure rental or managed BMaaS, the human cost can quickly dominate the economic model. Owning machines and OS’s is expensive at anything more than a couple of racks of machines. Eliminating people and human change/release from touching things in the datacenter is probably the first priority. Otherwise it is hard to consistently drive that human number down and meet service quality expectations for 24x7.
What is truly expensive is buying a bundled service with massive margins.
Sometimes you have the luxury of not caring, e.g. when building very high margin, low scale tools where human costs dominate due to dev or other parts of the business, but for anything that starts requiring hosting costs beyond even as low as a couple of k a month, you're leaving money on the table.
There’s more than a kWh rate for that though. There’s battery backup power, diesel backup, and failover hot switching. It’s not like just plugging in your phone to charge.