Hacker News Reader

You touch on a point that has been bothering me for a while.

In the 90s I remember what it was like to be in an industry beholden to MS and other proprietary software. I worked for an MS partner and did the cert thing etc but it felt uncomfortable to me. There were lots of people happy to soak in everything MS and ignore everything else. They were basically tiny pawns in a global Microsoft sales strategy. Most of their knowledge was product details, and how to glue multiple MS products together.

I switched to the FOSS world to get away from that product focus and lockin. I enjoyed the freedom of evaluating and choosing tools and building things from interchangable layers. I wasn't beholden to any vendor and was part of a community instead. As Linux later grew into a mainstream OS it worked out great for my career too.

Now with everything in my circles moving to AWS and moving into higher level managed services with massive lockin, the creeping notion of everything (incl the certified engineers) turning back into proprietary products is reminding me of how I felt in the late 90s about MS.

It's all in the name of efficiency, simplicity and value for money. But I'm not seeing it play out like that. Our Infra/Devops/SRE/etc teams seem to be getting ever larger and ever more swamped in ever complexifying CI/CD pipelines and endless freaking IAM policy tweaking and hand holding of self service development teams. Meanwhile the annual AWS bills dwarf our previous on prem colo and hardware replacement budgets (eg 10x), even though we haven't saved anything on staff costs as that has gone way up too.

We're back to just gluing vendor products together again, with general purpose tech knowledge being replaced by proprietary product knowledge. And the new generation of engineers and managers that don't remember the 90s see all this as progress and not a problem.

I am now an "old man despairs at cloud" meme....

Yeah it's history repeating. I've said it before on here and I'll say it again I don't think running on AWS will ever be cheaper than DIY. It's kind of a fallacy and a sales tactic by AWS to tell you how the elasticity of their offering allows you to "save money during the quite hours". I don't think I've ever seen a system that actually does that.

The savings usually come from the corners being cut on DIY solutions: not accounting for the time spent on infrastructure services, skipping various security and monitoring features, sacrificing reliability, etc.

You can definitely beat it with certain workloads (e.g. high-volume egress) but I’ve seen more cases where someone thought they had beaten it but was really comparing unlike things (e.g. comparing S3 to unreliable storage in a single location) or leaving out operator time.

The counter to this is that companies rarely cut ops staff when moving to the cloud. The problem is that you still have security and monitoring knobs to fiddle with in a cloud. And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems.

> And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems.

More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”

I'm talking about things like exorbitant egress fees changing how you have to architect your systems. Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters.

It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures.

Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack.

That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.

I think companies like Cloudflare and fly.io are trying to tackle some of those multi-datacenter complexity issues you're describing. Those ridiculous egress fees appear to have peaked as well since Cloudflare announced their zero egress S3 competitor.

I believe the way forward is not going back to self managed infrastructure. It looks like there will be sufficient competition to limit the most egregious price gauging. The way to avoid lock-in is to build on APIs that are at least somewhat portable.

I know the portability devil is very much in the non functional details. But I think we'll see more of what happened with S3. Everybody will be adopting the more popular APIs. Over time we will converge on some new POSIX-like industry standard for the cloud.

> Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters.

Were you either sending a huge amount of email or was this something related to the internal traffic needed to generate each message? I've definitely seen this in other contexts but am impressed to see email driving that (and, of course, preventing use of SES).

Note that I said “more typical” — I would define “cutting corners” as something like having a production service running with only one instance but once I go greater than one I'd be investing in automation anyway for reliability, and that would have included things like health checks & automatic rebuilds. It sounds like your business was pretty aggressive in region usage and I'd be curious what was driving that many regions.

> That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.

Don't take anything I'm saying as defense of their egress pricing. My point was just that cloud pricing is very visible and that can lead many places to focus on that without accurately assessing what they're currently paying for their own infrastructure.

Cutting corners assumes not doing things that are necessary. What I've found is that systems in the cloud are often unnecessarily resilient in areas which they do not need to be.

The example of S3 is apt as often ephemeral files are kept in S3 in which case they don't need the reliability storage guarantees that you'll get with S3.

I've also found that just because you're paying over the odds for AWS's SLA doesn't mean jack to your systems actual uptime. All systems I've dealt with in AWS / Azure have had unplanned outages. I think almost all of these have been problems in the software itself nothing that AWS will help you with.

> Cutting corners assumes not doing things that are necessary. … The example of S3 is apt as often ephemeral files are kept in S3 in which case they don't need the reliability storage guarantees that you'll get with S3.

… or which didn't seem necessary until something breaks. For example, the cost of using S3 for temporary files is often negative because in most cases your usage charges are going to be lower than keeping a sufficiently-large storage volume provisioned 24x7 for it, and if you know that you don't need geographic redundancy you can use the one zone storage for that. In all cases, you'll benefit from things like bitrot protection which that scratch storage space doesn't have built-in.

> I think almost all of these have been problems in the software itself nothing that AWS will help you with.

Definitely. What you get from a cloud environment is better tools for dealing with this — for example, shifting load is easy and making it trivial to get new resources makes it easy to do things like high-fidelity testing, blue-green deployments, rolling reboots, etc. which most organizations struggle to do efficiently in a traditional data center environment.

Optimistically, those unplanned outages are software errors, instead of data center management errors. There is an impact to your uptime by using AWS, but it's relative to having the same thing runing on not-AWS rather than an absolute value of uptime. The scope of what you can screw up is smaller

I share your frustration, but you have to factor in what we get for all the complexity. IAM roles are much more secure than "pwn one, pwn them all", containers are much easier to develop and deploy with, etc.

I'm not saying it's worth it, but we should evaluate it on those merits. I, too, have a suspicion it's not, but maybe the larger ops team on AWS can run circles around the smaller ops team on on-prem? There has to be a reason, after all.

> I am now an "old man despairs at cloud" meme....

"At cloud" indeed :P

It's not the cost or complexity that really bothers me (that was just a side rant hehe), more the vendor "productisation" of all the tools. After all, in many ways open source stuff is often lower level with the extra complexity that involves.

With open source tools (hell, even they are often turning into products now) there was always a feeling you could dig in and understand them, then join the community and contribute, and even shape their future. Now tools are just things to consume on the vendors terms.

Maybe the open pendulum will swing back again in 10yrs - probably at the hands of something like the Cloud Native Foundation, rather than the last peak 10yrs ago driven by Linux distros and some key apps/languages as well as dissatisfaction with MS etc.

Ah yeah, that's frustrating to me as well. The network is now the OS, and it's proprietary.

I also did the 6 developer MS cert thing in 2010. Not because I thought certifications were valuable. Just as a guided learning path. I never told anyone or put them on my resume. I was transitioning from a decade of C bit twiddling.

I did the same thing 8 years later to learn the AWS ecosystem. Where I now work 2 years later in ProServe (just disclaiming my biases). There is a chance I could retire from AWS or I could move on to the next hype cycle. Managing a career in technology is about staying relevant.

It’s not like my vast knowledge of 65C02 assembly that I learned in the 80s in middle school comes in handy in 2021.

However “my general tech knowledge” has served me well as well as my ability to know what level of the stack provides a business an “unfair advantage”. Managing infrastructure is seldom it unless you’re a DropBox.

Every company is dependent on dozens of vendors for their operations.

I found that even a proprietary product can improve your skills in a general way, and it has the advantage of being pre-selected for "what works". Anecdata: Google Firebase, specifically Firestore. On a superficial level, it's just another NoSQL database and you'd probably (rightfully) realize all the things that MongoDB can do that Firebase cannot. OTOH, I learned two things from it that I didn't from previous work with MySQL and PostgreSQL. (Note that I did not previously work with MongoDB beyond some tinkering, so maybe that would work as well).

The first was data-change notifications, i.e. getting notified when a record changes. I toyed around with this idea with MySQL some time ago, masquerading my server as a replica server, but generating derived data form it. IIRC PostgreSQL has something similar with WAL streaming. With Firebase, I learned how this can actually work and convinced me that the idea is sound. It definitely encourages me to actually do this for real when I work with PostgreSQL again.

The second is what PostgreSQL calls row-level security, i.e. using the data in a row to determine required permissions. In Firestore, this is the only kind of permission control (except for all-in admin accounts), and it showed me how this easily prevents a whole range of security issues that can occur through bugs in a back-end server.

The specific coding around Firestore is shallow as you said, and non-transferable, but understanding what is possible is something that will definitely help me in the future, with other databases.

This might be true in some cases. For example, I'd argue that doing a Cisco CCNA/CCNP certification is going to teach people not entirely familiar with the matter a ton of useful stuff about network technology in general even if they go to work on a Juniper site.

On the other hand, I've rejected all certs that relate to "_______ certified architect/developer/whatever", most famously pushed in our company as the "AWS certified in-house advertiser".

In any case, those certs are extremely lucrative business.

Disclaimer: I work at AWS in Professional Services take my bias as you will…

I’ve been developing professionally for over 25 years and 10 years before that as a hobby. My goal has always been to solve business problems. Infrastructure is just “undifferentiated heavy lifting” to me. At my second job over 20 years ago, we sold hosted storage because we had to and we had to maintain our own SAN, infrastructure, databases, 20 or so application servers, DB servers, etc. That wasn’t part of our core business. Today, a company would never do that themselves.

I went from belatedly opening the AWS console in 2018 for the first time, to rearchitecting my previous 60 person company’s entire architecture to be “cloud native” and “serverless” so we could effortlessly scale as we moved into selling access to our microservices to large health care systems. Being able to scale came in handy in 2020 when there was a little worldwide pandemic. They were acquired 6 months after I left for 10x revenue. They weren’t concerned about “cloud lock-in”. My CTO who championed the charge to go all in on AWS - and is still very up to date technically - is in his 50s.

Two years later, I got a job at AWS. There is nothing magical about the concepts of learning AWS. Most of the services are just hosted versions of well known open source tools and the rest are conceptually the same as I have done for over 25 years. I didn’t get my job because of the vast knowledge I had about AWS from 2 years working at a 60 person startup (please note sarcasm). I got it because I knew how to lead development and deployments and do “application modernization”.

I spent half my loops discussing how I had used Hashicorps Nomad, Consul and Mongo to create a master data model at a company that didn’t use AWS at all.

There are plenty of people who are hired in my department who have no prior AWS knowledge. But are subject matter experts in their domains.

I’m 100% sure that if I applied for a similar role as mine with MS/Azure or Google/GCP they wouldn’t be deterred from hiring me even though I know nothing about either.

There are plenty of PHP developers that focus on PHP careers, C# developers that focus on C# careers. Windows developers that don't develop for Linux, and vice-versa.

The cloud providers are just one abstraction above of OS's. AWS is Linux. Azure is Windows. GCP is Mac. No, not really. Just as an analogy of approximate customer market share, and again illustrating that there are plenty of people who could build careers on specializing in only one. Most developers, even.

Then there's something like Kubernetes. Which, extending the analogy above, makes me think of HTML5/JavaScript development with Electron - write your logic once, run on any OS (or Cloud Provider).

Now Electron is great, and web development is clearly very popular. But would you recommend that from an engineer's point of view, focusing on Web Development and Electron is the only way to build skills that are "highly transferable, fundamental skills"?

Now if by transferable skills, you meant actually the underlying OS/networking/infrastructure/data center skills, well then that's A) not really cloud skills, and B) soon will be as niche as C/Assembly programmers - still vital, essential, and widespread - but also a tiny minority.

> Windows developers that don't develop for Linux, and vice-versa.

Heh, the vice-versa reminded me of: A profitable, growing, useful, legal, well-loved... failure (2012), https://apenwarr.ca/log/20120326

That seems like pretty much the entire point of containers and/or kubernetes.

...and of HashiCorp, more aptly.

I think HashiCorp is made fairly redundant by Kubernetes, but yes if you don't want to adopt Kubernetes, you're correct. And that’s totally valid.

Kubernetes is already multicloud, and you don't need Terraform, Nomad, or Consul if you're using k8s. I would argue Vault is the exception.

I understand your point about career vendor lock-in, but I am not sure if it is as big of a problem as you may think. If I were a hiring manager for a team using Google Cloud or Azure and I saw a candidate with a strong background using AWS specific services such as Lambda or DynamoDB I certainly would think they can handle whatever GCP/Azure's versions are too.

Heck, AWS in 2022 has changed quite a bit since AWS 2018, so for better or worse we all need to keep our skills fresh even if we are locked in!

>> I think there's a missing angle from the engineer's point of view... I'd always want to work on Plan C - cloud providers as suppliers of standard commodity infrastructure. Possibly in a managed way...

tbray did address this point with a different question:

So if I'm in technology leadership, one of my key priorities... is something along the lines of "How do I succeed in the face of the talent crisis?"

I'll tell you one way: Go all in on a public-cloud platform and use the highest-level serverless tools as much as possible.

You'll never get rid of all the operational workload. But every time you reduce the labor around instance counts and pod sizes and table space and file descriptors and patch levels, you've just increased the proportion of your hard-won recruiting wins that go into delivery of business-critical customer-visible features.