Hacker News Reader

Comment by angarg12 | Hacker News Reader

The question: "Why it’s so hard to find roles in cloud technology, while jobs go unfilled."

The answer: "The key thing here is that businesses are not hiring fresh-out-of-certification AWS experts to take over the infrastructure of their established operations teams. They want people with solid experience, a risk-free bet that the new hire can execute the tech flawlessly. This often means insane job requirements [...] and an existing track-record of success."

So jobs openings stay empty because companies have unrealistic expectations. Is that something new and surprising? This is the pattern in tech recruiting as far back as I can remember.

View on HN

Wait. These aren't really unrealistic expectations. If you want transformation against a risk appetite of hardly any risk acceptable, you jack up requirements or no go. That sounds business wise to me. In my firm we've had many cloud related problems. The result was higher and higher demands in the direction of our suppliers. Obviously this limits the number of possible suppliers, and makes some suppliers unhappy, but it is proper business in line with the way we want to reach our goals.

It becomes unrealistic if those people are in short supply but the companies also try to keep compensation low.

If you need a special expert fast, but you don’t expect to pay a lot, then you are being unreasonable.

Judging by the claimed urgency and how wisespread the need is becoming, not every company will be able to get an experienced person for this role.

> If you need a special expert fast...

A job posting doesn't necessarily mean that they need someone fast, just that they'd like someone, someday, for a certain price. Companies have many important but not urgent goals that can wait for the right people.

Sure, but that seems not to be the case with cloud migration and engineering at the moment.

They really are, when every employer insists on 3 years' experience with a 3-year-old product and only 10% of candidates started on day 1. That's how you get 90% of experienced candidates unemployed while 90% of roles remain unfilled.

Especially unless they want to pay for it. Most experienced people already have all the jobs they need. No one else is going to train your employees for you. You don't want to pay the premium? Hire on potential and award those that do well. I don't know how many of my friends in these kinds of roles had their pay stagnate while the company were hiring e.g. consultant. Eventually they leave and things gets even worse for the company when the new project fails, because now they don't have either old or new infrastructure working properly.

Wait you mean companies should pay for training? That's unpossible. I thought training on high level systems just happened by unemployed people taking free Youtube courses or paying for their own training. Why are we asking so much from companies. It's bad enough we only give them massive tax breaks but now we want them to train people? What's next paying them well for experience? /s

Yes, AND one more factor no one is mentioning: resume inflation. If people read that they need several years of experience in a product or tech that's only several years old, like Kubernetes for example, then they will start to lie. And since they start to lie, others feel they should lie, too. The moral of the story is: tech management frequently doesn't know what it's doing.

Correct.

But what they should do is have hiring managers who understand that people with experience and wisdom, and ability to learn, can be applied to new technologies more effectively and securely (with past experience of seeing what goes wrong in infrastructure) than people with 5 years of the latest buzzword technology. Have a look at DevOps jobs descriptions lately and notice:

. 5 years experience in AWS/Cloud Deployments

. CI/CD pipeline

. Containers a must (Kubernetes / Docker)

. Basic bash scripting

. One of Python/Ruby/Node/etc.

. BS/Engineering Degree or equivalent experience

That's it.

Now you understand where horror stories in infrastructure come from, and why companies hire expensive consultants or "DevOps Shops" to repair shoddy setups.

This, however, is one step above the following:

"We're seeking a full-stack developer to develop our front and back-end systems, including designing and enhancing AWS infrastructure utilizing Docker / ECS and CI/CD pipeline via Jenkins or equivalent CI system." That's the Startup 10X Engineer Special that pops up a lot.

Yup, good luck!

I have a certification, and I hire people after asking for AWS certifications on the job description. My best hires have been people with IT experience, with or without certs.

Honestly, AWS solutions architect cert is to easy to memorize and pass, apparently. I had someone with 2 certs not know which AWS service would work as a good "serverless replacement to running scripts".. (Lambda) and didn't know how you could schedule lambdas to run every few minutes or on a Cron expression.

Basically, people think this cert means a lot, but it doesn't on it's own. It's like getting A+ certified without any evidence you ever touched a computer component. People are putting the motherboards in the right place, but don't know how to orient it.

Don't get me wrong, the cert can have value... But I like for people who have the cert and use AWS in some way... Even a GitHub webhook to Lambda.. or an s3 static site with CloudFront... Some evidence they actually touched and used an AWS service.

So yes, I don't like freshly certified individuals, because the AWS _associate's_ certs are NOT an expert certification. Sure they are slightly hard, but they are certainly high risk if you expect a solutions associate to migrate your data center or databases. Someone with 3 years IT experience has a better shot

I’ve been going down the certification road for a little over a year and should have 7 (now 4) by the end of the year. The company is paying for them and paying me above the local market average (I’m not talking about SV money here) - especially considering that I negotiated not to be made lead.

I started down this road as a dev lead for another company that wanted to “move to the cloud”, but at the time I didn’t know anything about AWS. Studying for the associate cert was mostly to get an overview of AWS so at least I could talk intelligently about from the infrastructure side and know what I didn’t know from the development side. I knew I would have to dig into the SDKs to actually do anything.

They hired a bunch of consultants who were old school net ops guys that didn’t know anything about AWS besides how to mirror on prem infrastructure. They were “lift and shift” guys. Of course it ended up costing more. I didn’t know then that AWS always costs more unless you are willing to change your processes, let a few of your infrastructure people go and let AWS do the “undifferentiated heavy lifting”.

I saw the situation was hopeless and all the company wanted to do was host a bunch of EC2 instances and the netops guys and consultants were so entrenched that there was no hope of me actually getting any real world experience with AWS.

I changed jobs and found a small software company that was already on AWS but the newly minted CTO wanted to actually take advantage of AWS. The MSP they had chosen before he came aboard were also a bunch of lift and shifters.

I was hired with no real AWS experience to lead the effort. After doing a lot of green field small proof of concept projects and fumbling through, I feel badly for anyone who tries to implement anything on AWS with just the certifications, without the industry background, and not working for a company with a business support agreement plan with AWS.

Good info.

You are right, AWS can cost more, but I disagree with "always". I run my personal website for $15/year, and most of that is the domain.

I would say: if you're trying to list and shift to AWS, you're doing it wrong, and it will always cost more. You also can't really fire many people aside from pure network guys. Server admins will still be needed for the virtual servers.

If you take the opportunity to redesign, you can realize cost savings. I still advocate avoiding layoffs, because you can retrain competent ops people to support the new infrastructure. The grizzled old guys that have trouble adapting are still usefull because chances are high you are not fully serverless within 3-5 years after a lift and shift.

As you become more familiar, you realize the tweaks that can be made to save money. We used to spend $600/month on 3-4 concurrent CICD workers. We swapped to docker-machine Spot-based CICD and spend $100-140/month for 12+ concurrent workers (we started at 6 and keep raising this number because the costs simply are so small)

Our front end is all static files (JS driven API calls). The $100/month server is now under $5 month for S3+CloudFront.

So I agree, AWS always costs more if you are trying to replicate a physical data center and ignore all the possibilities for improvement... But with slight tweaks (even starting with auto scaling.. or even more simply, timed lambdas to turn on/off servers overnight), you can reduce costs. Migrating to containers is also a huge cost savings, since you can pack more work into fewer servers without worrying about as much (I lived the nightmare of virtualenv's and managing 12 apps on 1 server.. I prefer containers now as an ops guy)

My goal and marching orders are to use AWS managed services for anything possible. We (the CxOs and myself) agreed that out of all of the business risks we had, our dependence on AWS is the least of them.

For builds we are completely serviceless. We use CodeBuild with mostly the prebuilt Docker images and one custom image to build some .Net Framework code. We use CodePipeline to deploy lambdas with CloudFormation and Code Deploy agents for EC2 instances.

For any new work, we have to justify a need for servers instead of using lambda for both APIs and back end ETL work loads - with step functions. There are a few back end workloads that we will need to transistion to Docker eventually but even then we are looking to use Fargate.

I have liked Corey Quinn's advice about saving money in the cloud. Here's a tweetstorm about the topic:

https://mobile.twitter.com/QuinnyPig/status/1091041507342086...

I could not tell you the names of many of the AWS services, but I have lambda functions running in AWS (written in Go). They monitor my DB loads in RDS. I get email and text alerts when things are not as expected via SNS, but I can't recall the name of the AWS service that I used to schedule that alert... CloudWatch maybe. For sure it was CloudSomething ;)

I'm not AWS certified, but I've been a technologist for many years and I majored in CS, so it all just sort of makes sense. Some names such as ELB, RDS and Lambda are intuitive. But some are overloaded, for example Alexa. They have Alexa for business and Alexa Top Sites (domain name research). BTW, has anyone else used Alexa Top Sites in AWS, it seems way overpriced to me.

I'd be leery of someone who knew all these names, but didn't have much actual experience as a technologist, but in an interview, they'd probably sound better than I would.

Your response is exactly the right kind for an interview.

"I don't use those AWS services, but let me tell you what I have used". I agree, most people won't know one of: Sumarian or Sagemaker, but chances are you can easily know most features that matter for your job interests. Most new AWS services over the last few years are niche markets. A good company to work for would not expect you to know this stuff without experience. I personally don't expect someone without certifications to answer AWS questions, so it's always a plus when they can. But if they have certifications and can't answer any questions? That's a big red X. I can't speak for everyone, buy I usually have 2-3 different questions on different areas, and if you're certified and have used AWS, you should be able to get 2 of them.

Ex: what is the difference between EC2 server types M, R, and C.. my previous lambda question, and usually a question on ECS or CloudFront

I have used Alexa Top Sites but have never paid for it. There is already a free csv you can download daily that contains the top million sites. For more granular lists, Builtwith often satisfies my needs.

Lots of questions about this. Some say it is old, stale data. I don't believe it is current. They charge 25 cents per 1,000 queries in AWS. Why give that away for free?

See thread here: https://gist.github.com/chilts/7229605

It is good enough for my needs even if its stale. I just want the top 100k sites and dont care about trends, and precise rank. I can see how someone else would want to pay for it though.

I maintain they bought that service just for the name (and toolbar installs).

Let me also state: If you have a certification and are having troubles landing a job, do a personal project to highlight your capability. It doesn't have to be hard.. or expensive! It barely needs to even look good -- if you're a back end guy, getting into an interview room and taking about the great CICD or script setup you did matters more than if it's a white page with black text... And if you're certified, you should know there's plenty of free it under $1/month options for AWS service. I hired one guy without a cert because he showed me the game server he set up in AWS to play with friends

I'll never understand why so many businesses in this industry seem to be so short-sighted when it comes to talent development. What these companies need is an experienced leader and a small team of more entry level people, some of whom will stick with the career and become tomorrow's experienced leaders. This is how it has worked for time immemorial. Why do so many companies seem to be looking for only one or the other level of experience? You need both!

With average stay 2 years as is statistic right now, the person will leave before or around the time they will become experienced.

Those people may stick with career, but not with company. Moreover average startup does not last two years either, so the company won't exist at that point.

All the company has to do is pay the person their new market value. I came to a company with lots of development experience with the sole purpose of getting AWS experience and leaving in two years.

After about one year, I had a discussion with my manager, I said I did everything I said I would do to help him meet his technical goals, I have both the theoretical knowledge and real world experience and my market value is $x. They gave me $x and now I have no plans on leaving soon.

You're right, it's a collective action problem. It would be better for everyone if everyone developed entry level people, but looking at just your own company, rather than the whole trade, it may make more sense to avoid investing in entry level employees. So I guess I lied when I said I'll never understand it. I do understand it. I just think it's misguided.

That, and also that in emerging fields like this, everything moves so fast that most people 'at the coalface' have learned substantial parts of their skillset on the job. So when one of these people leaves, you're not looking for "a PHP dev" or "a DBA", you're looking for "someone who's good at PHP 5.6, in-depth knowledge of postscript interpreters, and also adept at kernel hacking and assembling servers" or whatever odd combination of roles the outgoing person had taken on and learned during their tenure.

The idea of “you can find a job immediately after getting a degree or certification” ended a few decades ago. Today, you need some experience, especially in cloud computing.

Thankfully, in this field there are few barriers of entry, save for having some natural curiosity. All clouds have free tiers, excellent documentation, sample projects, etc. Kubernetes is free and open source. Technologies are interesting. Play and learn, and you’ll get a job.

Completely agree.. to many applicants I see think having a cert alone makes them gold, but they can't answer a thing about the topic that's not in the certification exam.

APPLY the certification in some way that YOU own and you're better than 50% of applicants I see. Many have certs, some claim some knowledge.. but few seem to actually have done anything themselves... They claim cloud CICD knowledge, but it was set up by someone else, and they don't know what a security group (firewall) is.. or they deploy to CloudFront, but don't know where the files are behind that... It's frustrating as a hiring manager spending an hour on these interviews with people who expect that paper to mean something without even 1 week of their own experience to back out up

There's also free training available for Kubernetes on EdX: https://www.cncf.io/certification/training/

Only personal tinkering doesn't count as "experience" in those companies books...

I tend to hire a lot more people that personally tinker without certifications, than those with certifications that don't have a pet project.

Certification proves memorization. Tinkering proves interest. I don't want a candidate that is not interested personally.

It does count on the interviews. A lot.

Create an LLC. Now everything you do is "selling cloud enterprise-solutions" (as all businesses are enterprises and you aren't going to sell to your grandma).

I think personal projects count a lot at CV filtering stage.

Knowing Kubernetes will not help you at all with most cloud infrastructure. But they all allow you to set up a free account.

>They want people with solid experience, a risk-free bet that the new hire can execute the tech flawlessly. This often means insane job requirements [...] and an existing track-record of success.

So much this. I've seen job listings that say "DevOps" and they really mean "SRE" or "Software Developer to own the whole SDLC w/On-Call Rotations".

Of course, the industry is changig but I think the industry hasn't let the "lead-in" occur, which existed in times before, and carry-over into the new momentums. For example, requesting 5+ years' experience in enterprise-level deployments in Azure and/or AWS isn't precisely synonymous with 5+ years in OOP.

I think that the reason for adding to the insane job requirements is that they're trying to condense the whole SDLC down to three people - at most - (e.g.: a PM and two devs) and still produce Fortune 500 level code products for the world to consume; thus, making the actual intial cost overhead to produce the initial product much lower.

This is, of course, pure conjecture and you're welcome to tell me that I'm talking out of my ass but, as far as my interviewing experience goes - so far, they want someone who's owned a small product (from an SRE perspective) for a few years' time, which doesn't really "translate" across the industry, yet.

SREs might come from Google, FB, Twitter, MSFT, Avanade, etc. but doesn't seem to - as far as I can tell - translate to an industry standard, across the board.

So, you'll have a developer with 7+ years' experience with OOP and maybe does the full-stack but, because he's never been exposed to things like being responsible for the deployment of the product (because it's responsibility of another team at his company, to make sure it goes through a security review cycle, has been tested in tested pre-prod environments, etc.), he'll automatically be "disqualified" from being considered for the given position - because the company doesn't want someone who can learn it, they want someone who can do it - literally, starting the next day.

(Sorry for the long tirade but the insane requirements are only insane because we don't understand the rationale behind them; understanding the reasons "why" helps us determine how the whole sector is shifting to new "paradigms". <insert Office Space reference here>.)

Cloud tech also changed the economics of operations and support. 10 years ago you might need 3 trained ops folks to run a production service on a given tech stack. Often the only real requirements for these jobs was a competent claim that the person could keep the lights on when the server went down at 4 am.

Modern cloud shops don't need dedicated staff to manually reboot servers, manually install software, or manually handle unavoidable hardware problems. Often what they're looking for is core software knowledge to add automation, fault tolerance, and scalability to their stacks under the banner of platform/devtools/backend software roles.

Reply on news.ycombinator.com