Lock-In and Multi-Cloud
tbray.org
tbray.org
I have no interest in developing skills that are essentially just shallow knowledge of a single proprietary vendor's product brochure. I want to develop skills that are highly transferable, fundamental skills that allow me to continuously build over a career. I also think there is danger here for the profession if engineers allow their skills to be pigeon-holed and segmented that way. It's already the bane of our industry ("oh you only know React, we are looking for a Vue developer ...") that recruiters look at shallow skill segments instead of fundamentals.
For that reason if nothing else, I'd always want to work on Plan C - cloud providers as suppliers of standard commodity infrastructure. Possibly in a managed way but still standard things that, after I spend years acquiring knowledge about them, will have some value when I show up at a different cloud provider or need to design for internal infrastructure.
In the 90s I remember what it was like to be in an industry beholden to MS and other proprietary software. I worked for an MS partner and did the cert thing etc but it felt uncomfortable to me. There were lots of people happy to soak in everything MS and ignore everything else. They were basically tiny pawns in a global Microsoft sales strategy. Most of their knowledge was product details, and how to glue multiple MS products together.
I switched to the FOSS world to get away from that product focus and lockin. I enjoyed the freedom of evaluating and choosing tools and building things from interchangable layers. I wasn't beholden to any vendor and was part of a community instead. As Linux later grew into a mainstream OS it worked out great for my career too.
Now with everything in my circles moving to AWS and moving into higher level managed services with massive lockin, the creeping notion of everything (incl the certified engineers) turning back into proprietary products is reminding me of how I felt in the late 90s about MS.
It's all in the name of efficiency, simplicity and value for money. But I'm not seeing it play out like that. Our Infra/Devops/SRE/etc teams seem to be getting ever larger and ever more swamped in ever complexifying CI/CD pipelines and endless freaking IAM policy tweaking and hand holding of self service development teams. Meanwhile the annual AWS bills dwarf our previous on prem colo and hardware replacement budgets (eg 10x), even though we haven't saved anything on staff costs as that has gone way up too.
We're back to just gluing vendor products together again, with general purpose tech knowledge being replaced by proprietary product knowledge. And the new generation of engineers and managers that don't remember the 90s see all this as progress and not a problem.
I am now an "old man despairs at cloud" meme....
You can definitely beat it with certain workloads (e.g. high-volume egress) but I’ve seen more cases where someone thought they had beaten it but was really comparing unlike things (e.g. comparing S3 to unreliable storage in a single location) or leaving out operator time.
The example of S3 is apt as often ephemeral files are kept in S3 in which case they don't need the reliability storage guarantees that you'll get with S3.
I've also found that just because you're paying over the odds for AWS's SLA doesn't mean jack to your systems actual uptime. All systems I've dealt with in AWS / Azure have had unplanned outages. I think almost all of these have been problems in the software itself nothing that AWS will help you with.
… or which didn't seem necessary until something breaks. For example, the cost of using S3 for temporary files is often negative because in most cases your usage charges are going to be lower than keeping a sufficiently-large storage volume provisioned 24x7 for it, and if you know that you don't need geographic redundancy you can use the one zone storage for that. In all cases, you'll benefit from things like bitrot protection which that scratch storage space doesn't have built-in.
> I think almost all of these have been problems in the software itself nothing that AWS will help you with.
Definitely. What you get from a cloud environment is better tools for dealing with this — for example, shifting load is easy and making it trivial to get new resources makes it easy to do things like high-fidelity testing, blue-green deployments, rolling reboots, etc. which most organizations struggle to do efficiently in a traditional data center environment.
More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”
It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures.
Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack.
That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.
I believe the way forward is not going back to self managed infrastructure. It looks like there will be sufficient competition to limit the most egregious price gauging. The way to avoid lock-in is to build on APIs that are at least somewhat portable.
I know the portability devil is very much in the non functional details. But I think we'll see more of what happened with S3. Everybody will be adopting the more popular APIs. Over time we will converge on some new POSIX-like industry standard for the cloud.
Were you either sending a huge amount of email or was this something related to the internal traffic needed to generate each message? I've definitely seen this in other contexts but am impressed to see email driving that (and, of course, preventing use of SES).
Note that I said “more typical” — I would define “cutting corners” as something like having a production service running with only one instance but once I go greater than one I'd be investing in automation anyway for reliability, and that would have included things like health checks & automatic rebuilds. It sounds like your business was pretty aggressive in region usage and I'd be curious what was driving that many regions.
> That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.
Don't take anything I'm saying as defense of their egress pricing. My point was just that cloud pricing is very visible and that can lead many places to focus on that without accurately assessing what they're currently paying for their own infrastructure.
I'm not saying it's worth it, but we should evaluate it on those merits. I, too, have a suspicion it's not, but maybe the larger ops team on AWS can run circles around the smaller ops team on on-prem? There has to be a reason, after all.
> I am now an "old man despairs at cloud" meme....
"At cloud" indeed :P
With open source tools (hell, even they are often turning into products now) there was always a feeling you could dig in and understand them, then join the community and contribute, and even shape their future. Now tools are just things to consume on the vendors terms.
Maybe the open pendulum will swing back again in 10yrs - probably at the hands of something like the Cloud Native Foundation, rather than the last peak 10yrs ago driven by Linux distros and some key apps/languages as well as dissatisfaction with MS etc.
I did the same thing 8 years later to learn the AWS ecosystem. Where I now work 2 years later in ProServe (just disclaiming my biases). There is a chance I could retire from AWS or I could move on to the next hype cycle. Managing a career in technology is about staying relevant.
It’s not like my vast knowledge of 65C02 assembly that I learned in the 80s in middle school comes in handy in 2021.
However “my general tech knowledge” has served me well as well as my ability to know what level of the stack provides a business an “unfair advantage”. Managing infrastructure is seldom it unless you’re a DropBox.
Every company is dependent on dozens of vendors for their operations.
The first was data-change notifications, i.e. getting notified when a record changes. I toyed around with this idea with MySQL some time ago, masquerading my server as a replica server, but generating derived data form it. IIRC PostgreSQL has something similar with WAL streaming. With Firebase, I learned how this can actually work and convinced me that the idea is sound. It definitely encourages me to actually do this for real when I work with PostgreSQL again.
The second is what PostgreSQL calls row-level security, i.e. using the data in a row to determine required permissions. In Firestore, this is the only kind of permission control (except for all-in admin accounts), and it showed me how this easily prevents a whole range of security issues that can occur through bugs in a back-end server.
The specific coding around Firestore is shallow as you said, and non-transferable, but understanding what is possible is something that will definitely help me in the future, with other databases.
On the other hand, I've rejected all certs that relate to "_______ certified architect/developer/whatever", most famously pushed in our company as the "AWS certified in-house advertiser".
In any case, those certs are extremely lucrative business.
I’ve been developing professionally for over 25 years and 10 years before that as a hobby. My goal has always been to solve business problems. Infrastructure is just “undifferentiated heavy lifting” to me. At my second job over 20 years ago, we sold hosted storage because we had to and we had to maintain our own SAN, infrastructure, databases, 20 or so application servers, DB servers, etc. That wasn’t part of our core business. Today, a company would never do that themselves.
I went from belatedly opening the AWS console in 2018 for the first time, to rearchitecting my previous 60 person company’s entire architecture to be “cloud native” and “serverless” so we could effortlessly scale as we moved into selling access to our microservices to large health care systems. Being able to scale came in handy in 2020 when there was a little worldwide pandemic. They were acquired 6 months after I left for 10x revenue. They weren’t concerned about “cloud lock-in”. My CTO who championed the charge to go all in on AWS - and is still very up to date technically - is in his 50s.
Two years later, I got a job at AWS. There is nothing magical about the concepts of learning AWS. Most of the services are just hosted versions of well known open source tools and the rest are conceptually the same as I have done for over 25 years. I didn’t get my job because of the vast knowledge I had about AWS from 2 years working at a 60 person startup (please note sarcasm). I got it because I knew how to lead development and deployments and do “application modernization”.
I spent half my loops discussing how I had used Hashicorps Nomad, Consul and Mongo to create a master data model at a company that didn’t use AWS at all.
There are plenty of people who are hired in my department who have no prior AWS knowledge. But are subject matter experts in their domains.
I’m 100% sure that if I applied for a similar role as mine with MS/Azure or Google/GCP they wouldn’t be deterred from hiring me even though I know nothing about either.
The cloud providers are just one abstraction above of OS's. AWS is Linux. Azure is Windows. GCP is Mac. No, not really. Just as an analogy of approximate customer market share, and again illustrating that there are plenty of people who could build careers on specializing in only one. Most developers, even.
Then there's something like Kubernetes. Which, extending the analogy above, makes me think of HTML5/JavaScript development with Electron - write your logic once, run on any OS (or Cloud Provider).
Now Electron is great, and web development is clearly very popular. But would you recommend that from an engineer's point of view, focusing on Web Development and Electron is the only way to build skills that are "highly transferable, fundamental skills"?
Now if by transferable skills, you meant actually the underlying OS/networking/infrastructure/data center skills, well then that's A) not really cloud skills, and B) soon will be as niche as C/Assembly programmers - still vital, essential, and widespread - but also a tiny minority.
Heh, the vice-versa reminded me of: A profitable, growing, useful, legal, well-loved... failure (2012), https://apenwarr.ca/log/20120326
Kubernetes is already multicloud, and you don't need Terraform, Nomad, or Consul if you're using k8s. I would argue Vault is the exception.
Heck, AWS in 2022 has changed quite a bit since AWS 2018, so for better or worse we all need to keep our skills fresh even if we are locked in!
tbray did address this point with a different question:
So if I'm in technology leadership, one of my key priorities... is something along the lines of "How do I succeed in the face of the talent crisis?"
I'll tell you one way: Go all in on a public-cloud platform and use the highest-level serverless tools as much as possible.
You'll never get rid of all the operational workload. But every time you reduce the labor around instance counts and pod sizes and table space and file descriptors and patch levels, you've just increased the proportion of your hard-won recruiting wins that go into delivery of business-critical customer-visible features.
This is an argument for how bad lock-in is though!
AWS was fucking everyone on egress price and nobody could do anything about it. It was not dropped until a serious alternative to the moat came in.
That’s not an example of some innovation or anything like that. It’s exactly like the Microsoft and oracle examples where the cloud is just extracting as much as the market will bear.
And they of course still are, as they didn't change anything related to the prices, but just increased the amount of traffic in their free tiers. That removed the pain for toy projects and workloads not requiring much egress traffic, but doesn't really make a difference for traffic heavy applications.
For example transferring 10TB/month from EC2 in us-east-1 to the internet did cost $899.91 before and costs $891 now.
For reference here is the announcement of these changes: https://aws.amazon.com/blogs/aws/aws-free-tier-data-transfer...
Sure it's highway robbery but they neatly segmented the market into the people who are and the people who don't and charged the former less.
To me this sounds a bit ideological-driven decision making.
The fact is they never raised their prices. People decided to use knowing the price. They weren't screwed afterwards. If the service is saving precious time and money, buying opportunity for the business, why would I ditch it?
Because AWS is extracting as much as they can? Great, I wanna use services from successful people. I just want them to come forward in advance about how much they cost, not later after I decided.
The point sometimes people miss is that the value of a service varies to each person, business or team.
It might be that, for you, in the contexts you were in, they were capturing too much with their price to the point its not advantageous for you to use the service. That's fine, use something else. But it doesn't mean everyone lives the exact same experience as you.
The problem I see with vendor lock-in is when, after you're locked, the service provider exploit this to change their terms (price and/or else) in a disproportionate and unjustifiable way against your initial interests in the service.
This never happened with AWS, to my knowledge (please correct if I'm wrong). Everyone knew the price when they subscribed. Little changed afterwards. AWS was able to meet a demand with quality enough products in an acceptable price to whoever decided to buy.
This is a commercial transaction. They're tremendously successful at marketing, sales and delivering, and that's OK. They don't have to be ashamed of that, there's nothing immoral or "rent-seeking" here. They're providing a service and selling it honestly, with transparent and predictable prices.
Lock-in exploitation happened with GCP at least once, that I know, with App Engine. And their service was doomed after that, as developers reacted against their practice and they weren't able to grow past a tiny fraction of the market.
> This never happened with AWS, to my knowledge (please correct if I'm wrong). Everyone knew the price when they subscribed. Little changed afterwards. AWS was able to meet a demand with quality enough products in an acceptable price to whoever decided to buy.
Vendor lock-in does not require any changes to terms, price, or any part of the service or product provided. In most classically studied cases of vendor lock-in (e.g., Microsoft Windows and Office binary formats, VHS vs. Beta), everyone knew the lack of interoperability when they bought the service/good, and nothing changed afterwards. That didn't stop those cases from becoming "vendor lock-in".
> AWS was fucking everyone on egress price and nobody could do anything about it
I disagree because AWS has been transparent with their prices since day one. Everyone who subscribed was aware of how much it cost. AWS didn't screw anyone by increasing prices after they were locked-in.
It's rent seeking when you don't produce new value, you just leverage existing systems to extract value.
But can we make it more tangible with an example?
Would you argue that AWS, for instance, is a company that produces no new value?
One might argue they add value, but their price is too high for the value they add. We'd go back to what I've said before: value perceived by a service is subjective and highly dependent on the context.
The prices didn't get higher, they just didn't get lower by as much as they could have. That really describes profit though. While I agree that all profit is rent seeking in some way and thus unfair, it's not very useful for splitting up some practices as bad rent seeking and others as less bad
It is more like the position of power AWS has; you can go elsewhere, but you'd have to choose not to make use of the facilities AWS provides. So you either pay for egress at AWS, or pay for reinventing the wheel elsewhere so you won't have to pay the same egress rates.
Correction: "Network costs obscene amounts of money" ~= "making it hard to move your business data and processes to a different offering"
"High cost" doesn't mean "not hard". "Hard" isn't exclusively owned by engineers; business decisions can also be hard.
> Hard would be ...
Stop right there. You're awfully close to gatekeeping what "hard" means, but you have artificially limited your vision to "hard engineering problems".
> Expensive isn't hard, it's just a calculation.
Everything is a calculation. The cost of reverse engineering (or using an RE'd method) is a calculation. The cost of switching is a calculation. Deciding where to stop when making a pithy comment on a forum is a calculation.
Here's a simple method to determine (in the context of lock-in) what's "hard": is it significantly preventing you from doing what you want?
Maybe you have a lot of money to burn in getting your own data out from a predatory silo, so it's not "hard" for you, but consider that not everyone might be in the same boat as you.
Known runtime cost is part of normal OPEX. Having to reverse engineer something because your MRI vendor won't give you the specs to write drivers for the 64-bit PCI-X acquisition card is vendor lock-in. If you need more examples, just ask around in any organisation of reasonable size. And no, we are not talking about individual people here (just in case you were making that assumption - it's hard for me to detect that in your writings).
If you want to change the meaning of what vendor lock-in means (it already has a definition, which I have written down a few times for your ease of not having to look it up yourself), good luck trying that out in your own context.
Just for the sake of completeness, here is a random paper that explains it in extra detail: https://personal.utdallas.edu/~liebowit/paths.html you could have found that relatively easily, even Wikipedia refers to it making it highly discoverable. If you don't like papers that are published by the author, feel free to request a copy at the Journal of Law, Economics, and Organization. If an explanation, reference or paper isn't good enough for you, why bother posting at all?
Perhaps the simplest distinction might be: you keep talking about OPEX while vendor lock-in is vendor-controlled CAPEX to 'get out'.
The world progresses. Language evolves (yes, even legal language, often understood to move glacially). But I'm not redefining anything here. "Vendor lock-in" is well understood at exec, management, and engineering levels and doesn't need to be artificially restricted to your limited reading of it.
> I am writing about the ability to use a different vendor.
The word "ability" also has meaning. It does not mean "absolute ability". E.g., Nearly every human has the "ability" to fly, through the modern technology of airplanes and the commonly provided services of airlines; but those (say, in countries like mine) who cannot afford a flight ticket are said to not have the ability to fly.
> Not about runtime cost or integration.
> Known runtime cost is part of normal OPEX.
Migrating off of one vendor is not "runtime cost" or "normal OPEX". By making it more expensive to migrate away, a vendor raises OPEX costs of such an "abnormal" move. Such weaponisation of OPEX costs is made to price-out the option, and is absolutely a denial of ability for business who cannot afford or justify such high abnormal OPEX costs.
----
> Having to reverse engineer something because your MRI vendor won't give you the specs to write drivers for the 64-bit PCI-X acquisition card is vendor lock-in. If you need more examples, just ask around in any organisation of reasonable size.
Look, no one is taking away your use-cases of the term "vendor lock-in". No one is disagreeing with you that those examples you cite absolutely are "vendor lock-in". But those are not the only cases or methods of it.
While intentional incompatibility is a far too common method of creating vendor lock-in, it is not the only method. For example, vaporware by dominant vendors used to deny market access to competitors is an accepted form of vendor lock-in.
> If you want to change the meaning of what vendor lock-in means (it already has a definition, which I have written down a few times for your ease of not having to look it up yourself)
I'm not trying to do any such thing. I have seen your "definition" and explained to you how the AWS egress tax is a means for promoting vendor lock-in, _within your definition_ of it.
> Just for the sake of completeness, here is a random paper that explains it in extra detail
I have read your "random paper"; it is a document explaining some history of economic literature. It does not define "vendor lock-in" (itself being careful to often use the phrase "lock-in" in scare quotes), but restricts itself to describing "lock-in by historical events".
What it does define, though, is _path dependence_. And what does it define it as? "... a minor or fleeting advantage or a seemingly inconsequential lead for some technology, product or standard can have important and irreversible influences on the ultimate market allocation of resources ..."; or, simply put, "sensitive dependence on initial conditions."
----
> If an explanation, reference or paper isn't good enough for you, why bother posting at all?
If an explanation, reference, or paper isn't even pertinent to the topic, and doesn't illustrate your point, why bother posting it at all?
I see that you don't like others contributing to your salads. My bad.
> network egress cost (which is OPEX) ... (migrations are CAPEX). Make up your mind before you try again.
You're the one pushing the agenda here that migration cost off of AWS is a "known" network cost, and thus OPEX, and thus missing the point. I don't care how you pencil down the money; the point is that migration costs are often NOT a "known ongoing network cost" at the beginning of service, and have been artificially inflated.
I should've seen the signals before venturing this far down this thread, trying to engage with you, but this drives the nail in the coffin. I no longer believe you're engaging in honest discourse; or worse, that you don't actually understand the business terms you're talking about.
----
From your other comments, you appear like you're an AWS apologist; or at the very least, that your business depends on people using AWS. I have a strong feeling I should have listened to Upton Sinclair's advice about getting someone to understand something, but it's not too late for me yet.
Good day to you.
Example: for small migrations we order an IBM Cloud Mass Data Migration device and an Azure Data Box, transfer between boxes, wipe the IBM box, send both back to their vendors, then do a delta transfer of the remaining updates.
Other Example: For acquisitions where we can't retain IAM between AWS accounts, order a snowball or snowmobile between them, all your RDS and bucket data goes from A to B in a matter of days. Not weeks or months over some slow 10Gbit link.
Maybe you could come up with some real-world examples where your point was a reality, because I'm not seeing it, the terminology doesn't match, and none of the scenarios from small to large have seen egress OPEX as a migration CAPEX issue.
We've been running successful migrations with no vendor lock-in woes between clouds for over a decade, egress and transfer costs has been a very small fraction of the whole operation, and never has been an ongoing pain point during post-migration usage. And none of this is AWS-specific, except the egress cost element the article linked here in HN. But hey, you do you.
At this point, I don't know what you're talking about. My point has been simple — and correct — from the beginning: AWS promotes vendor lock-in by making network egress obscenely costly.
> This scenario where we migrated clouds ...
That's your scenario, not mine. I did not make this thread about migration, you did. I did not follow you down your strawman path, but you seem to be believe I have.
> ... we don't migrate using network egress anyway, we migrate using physical bulk exports/imports using trucks
Good for you, I guess. Doesn't serve my purpose. Also, still on the wrong track about one-time, total migration.
> The article is also not about migration egress but about operational egress.
You almost came back on to the right track here, but quickly jumped back to you strawman path.
> Maybe you could come up with some real-world examples where your point was a reality, because I'm not seeing it
You're ignoring use-cases that aren't similar to ones you've had before, largely because you keep trying to see everyone else's needs through your "terminology" and comparing them to your "scenarios".
Here's a real-world use-case #1:
1. I started building an industrial-IoT analytics product for a client when it wasn't clear which cloud (private or public) they were going to run it on. Think lots of data, and lots of ad-hoc compute for the analysis.
2. I built it in a cloud-agnostic way, with all cloud-provider interaction safely nestled away behind a common API. It also had first-class live data replication across deployments. (If you're wondering, this kept most of the "migration CAPEX" down, in case the client wanted to move workloads after deployment.)
3. The client initially deployed it on AWS.
4. Later, the client's physical DC (closer to HQ) was upgraded and became capable of running our stack with nearly no changes.
5. Eventually, the client wanted to run some workloads in their physical DC. Some, mind you, not all.
6. Even including the cost of running their DC, it'd have been cheaper to run some compute workloads in it than on EC2. It seemed like it made perfect business sense to go through with it. Until network egress costs of the data were calculated.
7. Snowmobile was more expensive than biting the bullet and paying the egress tax. Not that it'd have mattered anyway, because ...
8. Snowball didn't exist, but even if it did, repeated usage to keep moving deltas of updated data was too operationally onerous.
9. They owned the data. They were willing to pay the price of a normal data transfer. Had they deployed to their DC first, the situation would have been so much better. They'd have to pay egress costs to their DC's ISP, and they'd still be saving money on splitting workloads across AWS and DC.
----
Real-world use-case #2:
1. While building the above, I gave every developer on my team a dedicated environment to work on, with a beefy shared server in our office; all this for low-latency dev work.
2. To mimic production scale and behaviours as closely as possible, I set-up a live sync of (anonymized) production data (back when it was small) to our dev environments.
3. Going from beta (~ 100-1000 devices) to GA (>70k devices) necessitated cutting off the umbilical cord, because our calculations showed the cost would be ridiculous.
----
Real-world use-case #3:
Same client as above. Wanted to keep their devices' data around for seven years, for auditing purposes. Didn't know if they'd still be on AWS in seven years. It appeared ending the contract would've been costlier than just paying for another year, because they weren't large enough to justify the expense of Snowmobile and Snowball didn't exist back then.
----
You see, AWS knows that if you can get your data out, you can get your compute out too. Of the two things keeping compute on any platform — data and code (including platform features like IAM) — only one is sticky. Other vendors can catch up on the code front, but if a customer even thinks about a partial migration, they'd see that the costs can get high enough that they might as well go for a complete migration, which is a significantly higher expense and much harder to justify.
A massive drop in prices like that means people were locked in and the switching cost in the market just dropped. So Amazon had to adjust to keep the customers.
I feel like scales have dropped from your eyes. :( Sorry buddy.
Unfortunately now finding engineers who can work outside of the big 3 cloud providers are rare. The lock-in is actually industry wide, not just from a resourcing point but also if you want to be taken seriously from an investor perspective. And of course with VC money you can just throw a chunk of it on AWS, but if you are bootstrapped it’s a different story
Having said that, eventually there will be a need to move to cloud once you are about to hit a certain scale or have more compliance and security requirements
Don't think VCs are much enthused about AWS either: https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap...
Fortunately that are still some that do care about training, without going through such actions.
But they are the minority either way, when training is considered a perk on job offers.
That said it's not just the budget for hardware to take into consideration but the often ignored costs like man hours, support contracts etc.
It also ignores security. Many security departments would much prefer the risk of having a dedicated cloud provider take that on, rather than a bunch of local sysadmins who will come and go.
With hybrid you can use any cloud you want, but when it starts doing things you don't agree on, you can revert back to your own systems.
Use the force, but have alternatives ready.
The problem is not deployment or orchestration (though that is one of them), the problem is building a team with the necessary expertise to support what could be a gigantic variety of tools and technologies at an expert level.
I'm at a place that uses GCP now and very little of that AWS knowledge I've accumulated is applicable here.
Cloud platforms/services have a lot going for them, but I think they've become complex monsters that don't really save people as much time as they think. If you're just pushing buttons on the EC2 console, then it's faster than provisioning an Ubuntu server with Ansible for sure. But, those toy deployments aren't indicative of the actual effort to manage a production system. At least in my (mostly start-up) experience with AWS going back to 2009.
It's true to some extent but I would typically characterize it as having DIY add additional layers you have to be concerned with. For example, I've seen more problems with SANs in some weeks than I've had total using EBS since the late 2000s (repeat for dodgy network cabling, server firmware causing issues, etc.).
The other thing to consider is what the alternatives are: for example, the reason why some of your RDBMS knowledge isn't relevant to DynamoDB is because it's a NoSQL database – if you wanted a SQL database, RDS is going to be at least an order of magnitude less time than running your own, especially if you are concerned with performance or reliability. If you've determined that NoSQL is a good fit for your application, that means that the relevant comparison is how much it costs to run DynamoDB versus, say, MongoDB.
> I'm at a place that uses GCP now and very little of that AWS knowledge I've accumulated is applicable here.
My experience has been the opposite. There are some differences but an awful lot of the principles transfer well to the other major cloud platforms and a solid foundation will make it easier for you to transition between the major cloud providers more easily than on-premise.
> If you're just pushing buttons on the EC2 console, then it's faster than provisioning an Ubuntu server with Ansible for sure. But, those toy deployments aren't indicative of the actual effort to manage a production system.
It's definitely not trivial but I certainly have no desire to go back to running a VMware cluster, either. I still have memories of being on the phone with their support team explaining how the flaws in their the design of their cluster health-check system lead to a big outage.
Some commenter mentioned the dark days of Oracle/Microsoft/SAP ruling over the server market. But at least these companies had the most basic decency to let you house/access your own hardware and diagnose things yourself if you had the skills. Now in the "cloud" you can just go to hell and suffer all the problems/inconsistencies, or rather your users can suffer since you as a sysadmin have zero control/monitoring over what's happening. Oh and bonus point: if users report some problems, they will be reproducible 0% of the time since there are great chances your users are connected to a different availability zone: yeah it's easy to reach more than 5 9's when "ping works from a specific location" counts as uptime for a global service.
So in the end, is it better to have a silly answer to "Why was the database upgrade delayed this time?"? Or is it better for the answer to be "i don't know, but upgrading the database cost us 37k€ in compute, 31k€ for storage and 125k€ for egress for backup of the previous db" ? I much prefer silly answers, but maybe that's because i don't have dozens of thousands of euros to be shorted of even if i wanted to :-)
It's easy to detect: ntpdate -q will tell you the drift, your logging would tell you when ntpd gave up because skew was too large.
Correction would depend on why it was happening: you might be able to tell ntpd/chrony to adjust more frequently or to accept larger deltas, but at that point I'd also say that the best path would be pulling that instance from service and replacing it so it's not critical.
> You'd have to send dozens of email to tech support who of course would say there is no problem because their machines have 999999999999999999% uptime and nothing could be wrong on their side, but hey they can sell you advanced support/engineering if you've got way too much money to help you find the problem in your code.
Has that been your experience? Based on mine it would be more like “we confirmed the problem you showed and have contacted the service team” followed by an apologetic call from your TAM. I've opened plenty of issues over the years but have never had someone insist that something is not a problem because their systems are perfect — they'd basically have to be saying you faked the evidence in their report.
Sure, that's if you got root on the machine. I was not talking about VPS hosting which we've been doing for a long time, but rather so-called "serverless" services.
> Has that been your experience?
That's been my experience with most managed hosting i've had over the years, though i've had some good experiences too. I've never dealt with Amazon but i'm assuming since you can't run diagnostics (since you have no shell) and like any other business it's likely their first level of customer is inexperienced and reads from a script, i'm guessing you're gonna have a bad time if you encounter weird/silent errors from your cloud services.
Some hardware errors are hard-enough to diagnose with root and a helpful customer support, i can't imagine without those.
I've never seen a hardware problem which the platform didn't detect a fault first in those environments (e.g. you see some requests have latency spike for a minute prior to a new instance coming online — it doesn't say “underlying hardware failure” but clearly something changed on the host or perhaps a neighbor got very noisy) and that includes things like the various Intel exploits where you could see everything rotate a week before the public advisory went out. I will say that I've had a few poor run-arounds with first-level support but I've never had them refuse to escalate or switch to a different tech if you say that first response wasn't satisfactory.
1. e.g. on AWS “Fargate” refers to the managed Firecracker VMs, as opposed to the traditional ECS/EKS versions using servers which you can access, but you can use https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... to run a command inside the container environment.
That is just as valid for cloud solutions, though.
The place for expensive cloud is when you're growing fast and need to spin up infra in a hurry. When you can't be bothered to configure software outside of your core competency.
Second, the reason people go cloud is not because they can't be bothered to configure things. It's because they have realized that any time spent doing undifferentiated work like that is a complete and utter waste of time and capital.
Third, the clouds power (and ultimately why it's less expensive) is not its ability to spin up infra in a hurry, the clouds power is elasticity. Properly architected cloud applications spin resources down as freely as they spin up, without intervention. You don't pay for idle and you certainly don't have to pay an army of people to maintain things that are now completely automated and realized in code or at the very least managed (without you having to think about it) for you.
Going multi-cloud, and forcing yourself to use the lowest common denominator services that are common across cloud providers is a really great way to miss out on all of those things.
Disney literally just started and are rapidly growing.
I don't know anything about Intuit or Capital One. Do they make you wear a suit and tie while working on Oracle databases? (I kid.)
[1] https://www.theverge.com/22787426/netflix-cdn-open-connect
I find it hard to find a balance between the OVH model of 'purchase a server for a year with discounts for a 12/24 year commitment' and AWS where you can purchase a single hour of bare metal compute for cheap.
I would say DigitalOcean fills this, but it's not like they've committed to bare metal, and I imagine they'll continue to look into expanding their suite of managed offerings like Apps[0] (which I think is heroku-esque auto scaling for containers) since the margins on these have to be way higher than their VPS offerings.
In some firms they still struggle with basics, like staff with 200k salary get a terrible slow pc/ crap monitors, etc.
Second, if you've got more than $10M per year of cloud expense you most likely should be thinking about switching to a hybrid model. You have the resources to manage your own computers as well as analyze your expenses and should be using the cloud companies when it saves money. The biggest problem at this level is that your employees probably use "cloud" as a way to bypass your IT department because they're not responsive enough because everybody with an MBA thinks of IT as a "cost center" and shortchanges it.
Third, I think this gives WAY too much credit to the competence of modern large enterprises. The fact that Oracle isn't bankrupt demonstrates that competence and technical execution really don't dominate the enterprise business cycle. And Microsoft really didn't have all that much control until Office365--I've never heard anybody talk about losing control of their desktop spend as all they had to do was slow down the refresh cycle on their desktops to control that cost. It's only been since Office365 that things have changed--and the mid-tier businesses love Office365 because now they don't have to deal with managing Outlook anymore.
In between $100K and $10M cloud spend, however, is a tough spot. I suspect that answer is still "just keep going" and that delivering features requested by marketing and sales is more important than reducing cloud spend.
So... It's marketing, licenses and lawsuits? (I'm serious, since I don't know Oracle from... Bad reputation and Java.)
A recent one was about how clouds should be dumb pipes, but the blog forgot about the 30 FTE you'd be hiring just to maintain your 'own' IAM, API and on-demand CRUD of generic resources (like compute, storage, networking and higher level resources like databases, object stores, queues, buffers, pubsub). And then they also forgot about the benefit of scale they would never be able to compare with as a single customer vs. a multinational specialised company with practically every possible service scenario under the sun...
Once you are beyond about 200K with enough in-house dedication you can start thinking about higher level abstractions that make you less bound to a single cloud or less bound to a single implementation (which can still be on a single cloud vendor's offering). You'd end up going that way anyway because at scale some facilities start to need adjustments to keep up with demand, like the creation and lifecycle management of object stores; you really can't be manually or bare-API managing those anymore when you have a ton of them. At that point you make processes, policies and automation a thing and if you implement them with some abstraction in mind it suddenly can be made to work everywhere... this is also where the bulk cost and ROI comes in; say you have your internal adaption in place for S3, adding B2 or R2 is really not that hard. Bam, object egress solved (For non-IAM scenarios; signed URLS would still work tho). But maintaining and owning this when you aren't big enough is just a drag your customers (internal and external) aren't going to profit from, and you really should be spending your time and resources elsewhere. Because you should probably focus on your business first, on what your customers want (so you can sell more in a commercial setting), and then on optimising your processes at which point you might be getting to the size where your increase in vertical integration starts making sense.
Several open source vendors get tired of Amazon taking their innovation, repackaging, and reselling it. They band together to create the strictest open source license ever. Stricter than AGPL3: "If you run this as PAAS, your entire PAAS down to the billing layer must be open. Every ounce of your code. Even the internal employee management bits. Maybe you're allowed a budget of 30kloc secret blob sauce, but that's it."
It should also say: "All billing reports must be instantaneous, easy to understand, and attributable to the products and the teams that use them." And: "Egress should never cost more than ingress, and high quality export tools must be provided for every service offered."
Use this as the foundation to build a fully open source cloud platform with the tools anyone could use to stand up their own data centers.
It would be a dream to see cloud reduced to commodity. Then open source vendors could make good money on support contracts, focusing on the software they love building rather than being eaten out of house and home by the giants.
So far, it has not been used to build fully open source cloud platform. Instead, it has been used (with limited success) to ensure that people who want to run ElasticSearch in the cloud have only one choice, going to Elastic NV.
It's not like it is new or something.. I remember reading the same discussions back in 2000's, when people discovered that Windows used parts of BSD's network stack.
I am not sure why people suddenly get outraged about cloud. Most webhosting providers in the last 20 years had business criticaly dependent on Linux kernel and Apache. Almost none of them contibuted anything to those projects. And people were not complaining.
Because now we are in the part of the business cycle where companies merge into a few huge sellers.
I couldn't name most of the webhost providers from 20 years ago, but Amazon is literally a household name.
So now it's not "A few Geocities knock-offs are using PHP for free and making a few bucks off a few people" it's "One of the biggest companies ever is making (seemingly) tons of profit off of almost everyone in the software industry."
Also the fact that in the last 20 years FOSS really succeeded in its mission - Proprietary software on the desktop is a joke now. The frontline has been pushed back to network apps where the backend is proprietary and the frontend is throw-away code that's useless without API keys.
So "stuff on the web" has gone from a low-profit novelty to 90+% of all software revenue.
I won't cite anything because we're talking about feelings. This is probably what the average 30-ish programmer feels about the situation, and if I got the facts wrong, I doubt anyone else is looking at these facts before deciding how to feel about AWS' marketshare.
These days the conflict seems to be about big cloud vendors productizing OSS code made by SaaS vendors, thus directly competing with them. It seems more like an economic argument (we want to get paid for the OSS we create) dressed up as an ideological one (OSS is about community, or whatever).
If you expect contributions, _get it in writing_. Copyleft was always the way to assert what you expected without this passive-aggressive "The user MAY contribute code to us or MAY just reap huge profit and contribute nothing."
This is completely inaccurate. The vast majority of changes to non-Free open source code, let alone the wider software that has subsumed them, does not get contributed back.
Such licenses are a bit like letting go of a balloon. Most times, the balloon will float away and pop on some tree with no benefit to anyone. In a few rare occasions, your obnoxious neighborhood kid will grab that balloon and play with it all afternoon.
You might not like that the kid is enjoying your balloon but no right minded person would expect that the kid’s parents recompense you for the balloon you let go willingly.
TBF, not long ago, someone posted on HN that AWS simply used their open source ware without even a thank you. It created quite a bit of noise on HN, so your opinion isn’t too far outside the norm.
Quite a lot of the time the user may not be able or want to use the rights granted them, but that doesn't make those rights useless. Indeed, they serve an important function, see the various offshoots of that early Linksys WRT code!
It saddens me greatly that so many have lost sight of this.
[1] https://www.fsf.org/blogs/community/201cthe-printer-story201...
Now imagine Amazon comes by and lists the books in the reading library on it's website for sale. You offered them for free after all. Further they are knocking on your door obnoxiously asking you when you are going to get a particular book in stock, since they have a customer who wants it.
What matters is the 'stuff' you get for your money. AWS knows this (as does Azure and GCP), as do the B2B customers. It's just a simple sum. Either the customer does more of the hard but generic work, or the provider does more of the hard (generic) work. You make two calculations (including potential benefits of vertical integration) and you pick what works best.
"Inventing" generic compute with an API, web console and service-wide IAM is possible, but adds no value at all to say, e-commerce. So you might as well just pay some vendor to already do that, and if possible to do it at a scale and quality you will never be able to reach as a single business anyway. So you go to a cloud since that already has it and you can't replicate it anyway. If it turns out to be expensive but still cheaper and better, that's a win, OSS/SSPL be damned.
1. Controlled by the community of users, not a big provider. 2. Is optimized to be run multi-cloud and multi-region. 3. Assumes that you will use SaaS services for common tasks (Replicated, Nobl9, Tackle, EdgeDelta, etc...). 4. Has a fully hosted solution for CI/CD (with integrated security offerings) that is, again, not controlled by a big provider. 5. Makes enough assumptions about "sane defaults" that services are well tested and service and operator boiler plate is can guide developers to best practices.
My view is that K8s was designed to attack AWS hegemony by providing a cloud abstraction layer that assumes that a company will have a single cluster in which they will run multiple relatively trivial applications (see Helm), but for some of us, the reality is that we have many clusters which each run the same mission critical application.
Let’s say you can define all your containers, orchestration, database and middleware in a compact way (Dockerfiles, Terraform, etc) with cloud vendor independent open source tools, using managed OSS.
How much of that code would be dependent on your cloud provider, really? Maybe a thousand lines of hopefully modularized platform dependent configuration code?
Migration from one system to another would still be an effort, but not that big one?
"Several regulators, mainly in the EU and focused on financial services, now mandate an exit strategy. Those organizations that do look at exit strategies often focus on contracts — aspects such as terms and conditions, and service-level agreements." [1].
[0] https://www.bestexecution.net/esma-warns-financial-service-f...
[1] https://www.itconvergence.com/blog/importance-of-cloud-exit-...
1. In the beginning (new startup), you don't know what usage looks like. It may not even be used at all at night / during the day. If you're not full-in on managed services, you're burning money on stuff you're not using.
2. Eventually you start to see consistent use. Run back-of-the-envelope numbers on using something less managed, i.e. Lambda -> Fargate -> EC2. If it makes financial sense to switch, do it. If not, stay where you are.
3. Continue to optimize over time in the direction of bare-metal as your cloud spend hits seven figures and higher.
For data:
1. If you have a small amount of data, then managed services are a trap, as services make implicit dependencies on the managed service. Stick to open core. If the amount of data is small, then it won't be too expensive to export it off the environment where it is to a new environment where it will be cheaper.
2. If you have a large amount of data, then your data is probably locked into whichever environment / provider is hosting it anyway, due to egress issues (either via network or via logistics shipping drives). You might as well use a managed service and reduce talent dependencies, as described in the article.
I have a small book I'm going to be releasing soon. It's going to be self published as it's an entirely online mixed media book (text, videos, quizzes, etc.) and I'm going to be packing it all into a Go binary (minus the videos, those will be on YouTube) to make deployments insanely easy.
(The binary will be less than 50MB in size - smaller than most Docker images. It can also be made to run on any OS I care about without a third party tool. Just sayin'.)
I have a 100/100 fiber line here at the office with an SLA. I have plenty of hardware too. I could self host it here, but instead I'm going hybrid.
It's simply too easy to spin up an EC2 instance fronted by an ALB, TLS being handled by ACM and DNS by Route53, all "as code" (Terraform is my weapon of choice.) I can buy that instance as a reserved instance and pay up front for the year too, reducing its cost by 45-50%.
Then I can attach that instance to an AutoScaling Group with a minimum of '1'. If I get traffic influx it will scale out to meet the demand at per-hour costs. It's more expensive but it shouldn't be for long.
If the (static) instance goes down the ASG brings one back up at "full fat" per-hour fees... but at least I'm up and running.
The landing page is static and delivered via CloudFront, and that's what most people will be hitting. The application won't even see that traffic at all.
As for on premise: I'll be self-hosting analytics here and also logs from the application and servers. Why put them on expensive compute in Cloud when I can just do it here locally on a cheap quad core with 16GB of RAM and TBs of NVMe storage? It's a no brainer really.
So you have to pick your weapon correctly. For this project, here are the weapons I'm selecting: AWS EC2, ACM, ALB, Route53, CloudFront, S3; YouTube; on-premise hardware for analytics and logs.
The Go binary is a custom web server for handling access to the book. This will be similar to how a few other book authors have handled access: a link you click in an email (like a one-time sign in token) that gets you access. The Go binary is also responsible for receiving and resolving Stripe Links web hook calls.
The book itself is made up of hundreds of pages, images and more. It's all built using MkDocs + Material.
The custom workflows meant I needed a small, custom app.
You can do a site search for hacker news with just one of the cloud names in it to find a ton of blogs about it.
Most of them say "lock-in" but aren't actually about lock-in because you're not locked in, just in a bad position (power vs. utility).
Not everything is priced using bottom up pricing, where you calculate the cost to produce something and then add a profit margin. There is also top down pricing where you price it based off of the value it provides or based off of what your competitors are selling at.