Many people use AWS because everyone uses AWS. Many of my clients have no need for AWS but still use it, at least until the VC money runs out. Usually then I have to go first, then servers are moving to somewhere cheaper when a new CFO comes in.
I spend half my time debugging Hyperscale Bullshit™ that the client never needed and that failed the moment the devops guy left.
If you want this stuff, you need a long term tech team, and a devops member on hand. Every time you hand your project around this stuff costs you hours. You also need to tell your devs to use the devops guy like tech support. They built this rickety tower of obtuse config files, make the devops guy fix it and don't let your devs spin their wheels for 4 hours before their assets compile on the Super Cool Distributed Docker VM microservice stack of teetering jenga blocks.
Their site was so ridiculously low traffic that I did some napkin math and figured out they were paying around $0.10 per web request.
It's only $7/month to deploy a web application. More if you want some of the paid features and a database instance. It all works out of the box with instant deployment out of the box, it's fantastic.
Then it suddenly goes $50 more per gigabyte of RAM, which makes it massively expensive for any serious workload. It's crazy hot much they can try to charge, makes AWS looks like a bunch of clowns in comparison.
Although whether it is worth it depends on your view.
( I keep thinking SalesForces is just not a good fit for Heroku, they should have sold it to MS or Google )
Which is what I think the OP misses discussing in as much detail as they could with AWS -- are there ways AWS is saving some customers developer/ops staff time over the cheaper alternatives? Cause that is often more expensive than the resources themselves. Could just be "because that's what we're familiar with" (and that's not illegitimate as cost savings), but could be actually better dashboard UIs, APIs, integrations, whatever.
[I am currently investigating heroku for our low-traffic tiny-team app, after we had our sysadmin/devops position eliminated. The performance characteristics are surprising me negatively (I can't figure out how anyone gets by with a `standard` instead of `performance` dyno, even for a small low-traffic app; currently writing up my findings for public consumption), but the value of the management they are doing so we don't have to is meeting and exceeding my expectations. (We currently manage our own AWS resources directly, so I know what it is we can't do sustainably anymore with the eliminated position, and the value we're getting from not having to do it).]
As such a lot of devops consultants certainly have all the wrong incentives to recommend it, and these days most of them also lack experience of how to price out alternatives.
E.g. a typical beginners mistake is to think you'd price out a one-to-one match of servers between AWS and an alternative, but one of the benefits of picking other options is that you're able to look at your app and design a setup that fits your needs better. Network latency matters. Ability to add enough RAM to fit your database working set in RAM if at all possible matters. And so on. With AWS this is often possible but often at the cost of scaling out other things too that you don't need.
And most developers won't do it if you don't give them budget responsibility and make them justify the cost and then cut their budget.
Development teams used to AWS tends to spin up instance after instance instead of actually measuring and figuring out why they're running into limits to keep costs down. Giving dev teams control over infra without having someone with extensive operations experience is an absolute disaster if you want to manage costs.
I work for a VC now. When I evaluate the tech teams of people who apply to us, it's perfectly fine if they use AWS, but it's a massive red flag to me if they don't understand the costs and the costs of alternatives. Usually the ones who do know they're paying for speed and convenience, and have thoughts on how to cut costs by moving parts or all of their services off AWS as they scale, or they have a rationale for why their hosting is never going to be a big part of their overall cost base.
The only case where AWS is cost effective is if you get big enough to negotiate really hefty discounts. It's possible - I've heard examples.
But if you're paying AWS list prices, chances are sooner or later you'll come across a competitor that isn't.
I'm having trouble following your theory of why having clients stay on AWS ends up leading to more consultant billable hours, I think because I don't understand what alternatives you are comparing it to. I am pretty sure it is not heroku, as in the earlier part of the thread?
Or are you talking about compared to simpler "vps" hosts like, say, linode? Doesn't that require a lot more ops skillset and time to set up and run compared to aws, or you don't think it does?
When moving them off AWS it'd usually be to managed hosting on monthly contracts.
Heroku turns expensive real fast. You're paying for AWS + their margins on top of AWS.
Managed hosting ranges from API-based provisioning not much different than AWS to ordering server by server.
In practice the amount of devops time spent dealing with the server itself for me at least is generally at most matter of downloading a bootstrap script that will provision CoreOS/Flatcar and tie it into a VPN and record the details. The rest of the job can be done by simple orchestration elsewhere. I have servers I haven't needed to touch in 5 years other than recently to switch from CoreOS to Flatcar (other than that the OS auto-updates, and everything runs in containers). Once you've done that, it's irrelevant what the server is or where it is.
For modern server hardware, if you run your own colo setup, that's a matter of having PXE and tftp set up once in a colo, and you can then use an IPMI connection to do the OS installation and config remotely, so even with colocated servers, I'd typically visit the data center once or twice a year to manage several racks of servers. The occasional dead disk would be swapped by data centre staff. Everything else would typically be handled via IPMI.
E.g. one of my setups involved 1k containers across New Zealand, Germany and several colo facilities in the UK. Hetzner (Germany) was the first managed hosting provider we found that could compete on total cost ownership with leasing servers and putting them in racks in the UK. Had we been located in Germany (cheaper colo facilities than near London), they'd not been able to compete, but putting stuff in a colo facility somewhere we didn't have people nearby would be too much of a hassle and the cost difference was relatively minor.
Small parts of the bootstrap scripts we had were the only thing different between deploying into KVM VMs (New Zealand), managed servers not in the same racks (Hetzner), and colocated bare metal booting via PXE on their own physical networks (UK). Once they were tied into the VPN and the container runtime and firewall was in place, our orchestration scripts (couple of weeks of work, long before Kubernetes etc. was a thing - we were originally deploying openvz containers and so the same tool could deploy to openvz, KVM and docker over the years) would deploy VMs/containers to them, run backups and failover setups, and dynamically tie them into our frontend load balancers.
We did toy with the idea of tieing in AWS instances to that setup too, but over many years of regularly reviewing the cost we could never get AWS cheap enough to justify it. We kept trying because there was a constant stream of people in the business who believed - with no data - that it'd be cheaper, but the closest we got to with experiments with AWS was ca twice the cost.
For the record, in my current job we do use AWS entirely. I could cut the cost of what we're using it for by ~80%-90% by moving it to Hetzner. But the cost is low enough that it's not worth investing the time in doing the move at this point, and it's not likely to grow much (it's used mostly for internal services for a small team). That's the kind of scenario where AWS is great - offloading developer time on setups that are cheap to run even at AWS markups.
I tend to recommend to people that it's fine to start with AWS to deploy fast and let their dev team cobble something together. But they need to keep an eye on the bill, and have some sort of plan for how to manage the costs as their system gets more complex. That means also thinking long and hard before adding complicated dependencies on AWS. E.g. try to hide AWS dependencies behind APIs they can replace.
When I moved from an ecommerce company with 1000 logins/sec to a SaaS company I could not believe the low traffic :-)
I'm forced to think about stuff like Kubernetes because the devs are popping "micro-services" for almost anything and the answer to trying to keep things manageable is, hopefully, K8s.
Of course they then send funny memes and argue that K8s is overkill. Yet they have no idea how much a devops guy has to do to actually ramp up one of their multi-gigabyte container.
The whole micro-services mentality is proving to be backwards in my environment. It's seen as the answer to everything. "Oh we have a behemoth, let's just not use this part of the application and reimplement it somewhere else." In essence that's great. However, the grunt of the work is making tests to capture old behaviour/semantics; and who are the (indirect) clients of this piece of code. Ignoring a piece of code to death and reimplementing it somewhere else behind a socket is only part of the solution.
The hyperscalars are benefiting greatly of this mindset imo.
I tried my best not to rag on devops as I am always incredibly impressed by these systems and what can be achieved and automated. It absolutely has it's place.
It's just that all the technology that big companies use become trendy and then end up used at small companies too. Small companies don't realise that the tools aren't solving the problems they have, and they don't understand the vendor lock ins and dependencies that they've introduced into their platform for no real gain.
This happens in all kinds of ways, not just devops. Your example of developers using the microservice pattern is exactly that. When I see a developer recommend microservices my mind flashes forward to all of the extra complexity that entails and how we've just turned a small monolith on one server into 6 pieces of software across distributed servers. Great for big companies, too much complexity for small companies.
But I do recommend companies have a means of deploying stuff to AWS and doing basic security there for clients that really require AWS. Having a VPC ready to go costs nothing operational and is good for at least having some semblance of collective skill with a major public cloud provider.
Invariably though when I see undisciplined developers take on colo style hosting systems become brittle and unable to accept changes and with any success it becomes a Herculean task to deploy new features without putting something important at risk. This results in more time spent on systems archaeology for new engineers eventually than creating new features or improving maintainability.
In 2020 I’d recommend developing as much as possible on PaaS type platforms like Heroku, a managed K8S, or even App Engine to avoid more bike shedding that really doesn’t buy a company anything materially advantageous early on like deciding which kind of EC2 instance to standardize on. Until engineers know what to optimize and concentrate upon in terms of process and there’s perhaps tens of thousands in monthly infrastructure costs most skilled (read: expensive) ops engineers won’t really be of much value except offloading work that developers shouldn’t have been thinking about in the first place.
The points that always seem to come up:
* AWS is a known quantity and it's easier to evaluate our business with it.
* AWS provides "outage damage control" because AWS outages make the news and customers are more understanding. When our ISP has issues it just looks bad on us.
* Our company doesn't look as innovative because we're not cloud. Bleh.
Our app is compute, storage, and data transfer heavy but switching to AWS being a, literally, 10x cost for us is apparently not enough a good enough answer.
Anecdotal, but AWS had more global problems than the triple connected data centers I've used in twenty years.
I suffered through many rough times, data center connection or power problems were not (very seldom) one of those.
Most of the problems came from apps that we didn't build well to scale or that had bugs (most frequent cause of problems and sitedowns).
"Confident"? "Most" failures? Are you merely hopeful that the probability of a bad failure is low, or are you able to test the AWS resiliency techniques you mention and to ensure that they stay working? At what cost?
Not sure if you do not know anything about typical ESXi and vSphere setups, but if a server fails, all virtual machines are automatically migrated to a healthy server. And of course, your HPE G10s are compute only, all storage is on the fiber channel connected SAN.
You’re still paying a cloud provider, but in this case it’s VMware
Multi-DC failover is handled by announcing our public IP block at both locations with different weights. It’s technically active/active because traffic can come in at the secondary DC but we have a internal site-to-site VPN that is used to direct traffic to the primary. If the primary DC goes down the secondary starts handling the traffic instead of passing it along. All the database masters flip to the secondary and things keep humming along.
If we lose the site-to-site then the secondary stops advertising altogether and all traffic is forced to the primary.
So we can lose the site-to-site (which is dedicated) or one of the DCs at any time.
However the startup was all built on oracle and sun boxes because "this is what investors want to see" and we'll get .80 on the dollar if we have to liquidate. We had some nimrod spend 8 months trying to get oracle to run on bare drives for our 100 tps (max) website.
They refused to let us use mysql or linux even tho the owner was very familliar with them from the ISP.
I think we spent 3x headcount on the hardware and software, eg we could have run for another 2 years had we been more scrappy. Not that the business idea was all that good.
We also got .3 on the dollar iirc.
Sure, if it's an established company then using their own hardware (and people to set up and manage it) might make sense; iirc Dropbox is a fairly recent big player that made that move. But otherwise it's a big upfront investment to make, and you can't know if it'll pay itself back.
So sure, AWS can be 2x or more as expensive as renting servers at a company like OVH or building your own datacenter, but it's paid by the minute if need be, not all in one go. If your startup or its VC money runs out in six months at least you can quickly scale down and pull out.
Or something like the Olympics where 95% of your demand comes in a predictable spike across a few days.
Why would you build exactly the same setup in AWS as for on-prem, unless your objective is to (dishonestly) show that on-prem is cheaper?
Lift-and-shift-to-the-cloud is known to be more expensive, because you aren't taking advantage of the features available to you which would allow you to reduce your costs.
It was far better to invest a little up front, and maintain at $600 my operations than the same for $96K a month, that's why.
I never "lifted and shifted", I built and deployed, with physical servers, a 3-way duplicated environment that flew like a hot rod. At a fraction of cloud's expense.
I personally think there's room for both, and I think hybrids between on-prem and cloud are the ideal for long running apps: you size your on-prem infrastructure to handle 99% of the load, and scale to the cloud for that one-off peak.
That's still pretty complicated due to different types of vendor lock in (or lock out in some cases). Google has invested in k8s to get people some value for moving away from AWS.
Comparing managed services like databases is maybe more meaningful than just EC2, but also so much more difficult.
Last time I checked, GitHub was also mostly on bare metal and cloud-free.
This is where your ALBs and ASGs come in. If your app doesn't use local writing and you can shift your caching to a shared cache, the cost savings are good.
Counterpoint: I've seen a great many false economies with people trying to go on-prem and do alternative hosting because they don't think the AWS premium for e.g. GPU instances is worth it. I don't think that has generally worked out well.
I could spend a million EUR a year on AWS without the need for most of the services of AWS?
But it YMMV and 1M EUR/y is not meaningful, perhaps we differ there.
It's not an impossible scenario... But imagine the sort of company that trains their own model instead of using a huggingface refinement or an off-the-shelf redistillation. (These can be done reasonably on an average gaming PC, no need for a cluster.) Such a company has expensive human resources. They bothered to get a data scientist and at least a research engineer, if not a full researcher. Were they hired on six-month contract as well? This is a huge expense, so it must be an important differentiator to have built a custom model -- and it's a one-and-done? I don't see it. I think it's going to be an ongoing project, or it shouldn't have been approved in the first place.