The cloud skills shortage and the unemployed army of the certified
itnext.io
itnext.io
Amazon, Google, Microsoft and others make it simple to get going but the devil is in the details when it comes to building it right, safe and ready for production. Just a few examples:
• Putting RDS SQL Servers on a public IP with no protection
• No templating of servers so if it disappears you have no record of how to get a new one running again
• Servers with SSH password authentication turned on with passwords of “Password”, because SSH key auth was “too hard for our devs”
• No backups because “it’s in the cloud, isn’t the cloud backing it up for me, like iCloud?”
I have an AWS Solution Architect Professional certification and agree you could get a cert and still not be qualified to run or design much on AWS but it does put you ahead of much of what I’ve seen out there.
I'm normally pretty certification averse (I don't even really value my undergraduate degree much), but I'm thinking of getting the AWS Solution Architect cert, because it seems like the material out there for learning that stuff is genuinely useful (and once you've learnt the material, you may as well get the certificate).
Associate's cert is mostly memorization and high level, but doesn't qualify you for being a trusted individual to do the job. It's good to make you aware of the right way to build things in AWS.
For $2/month you can make a static website, posted to S3, with a CloudFront front end and web hook to lambda to trigger content refreshes on specific Git actions.
I'd hire someone who did this with no professional experience with or without certification for an entry level position. That personal interest and drive is worth 1-2 years experience in my book... And I don't look for more than 6 months experience in entry level (I offer internships/ hourly for that group)
The tricky part seems like it would be dealing with tens-to-hundreds-to-thousands of machines which should automatically deploy and scale. And things like databases, which might also need to scale up/out at some point, and ideally should do without downtime.
That's much harder to try out on a personal project (due to the cost, but also due to the lack of genuine need shaping the implementation)...
There is a huge amount of IT work that never gets to that scale, and of the IT projects that get there, lots don’t actually need to be.
I have a client that does $50M a year on a SAAS product using only two EC2 isntances, but 1 of those instances is a developer/beta/test server. Only 1 is used to manage sales across the entire world. They rely heavily on Lambda and SNS & SQS as well.
Another client operates eCommerce at around $80M a year revenue using 3 EC2 instances. One of those is a test/dev server and the other 2 are used to distribute load and for redundancy with a load balancer sitting on top.
I have 2 other companies that are smaller that generates about $2M and $5M a year respectively that both run a static site hosted on S3, using Lambda for deploys and cloudfront for distribution like mentioned above.
Point being, you would be surprised how low key many company's AWS deployments are.
---
If I could pin one skill that will stand you head and shoulders above the rest among cloud architects, it is to learn Serverless functions. In this AWS world this is "Lambda". Microsoft calls it "Azure Functions" and Google Cloud Platform calls it "Cloud Functions". If you can write serverless functions AND are AWS certified, that seems to be the hottest thing that I have seen.
It especially confuses me that people want people to "learn" serverless functions. Serverless functions are just like normal functions, except with a library to use for communicating with the outside world!
If I was hiring, then I'd be asking people about VPCs, IAM roles and Security Groups. Those seem to be the complicated bits that you need to understand to do pretty much anything. The service-specific APIs seem relatively simple...
You must be dealing with exceptional companies who have application which need auto scaling. And need alone is not a sufficient condition, application should support it too.
Most application are simply lifted from legacy with no code changes and serves at-most a few hundred users at max and will crash and burn if autoscaled.
I wouldn’t go that far though. I would probably use something like Consul to do a local health check first and restart the app if it dies and then do the target group/health check as a second level check.
I have two or three ASGs up with a minimum and maximum of two and an HTTP health check that will kill an instance if the health check doesn’t pass and bring up another one.
I also have applications that autoscale based on queue sizes.
Scenario I am referring to involves application team closely guarding their servers like babies which means they are married to server names and IP addresses. These are cloned from on-premises and load balancing is just sticky sessions going to one node with other one sitting idle until first one fails.
First group is rare.
So you're tested on your understanding of the platform but not on your actual competences on it.
I've seen this in other areas however: people getting LPIC certifications and failing to run a proper grep.
In my experience, Red Hat is one of the few tech companies that get the certification program right by using an hands-on approach: either you actually are capable of performing tasks at a certain level, or you get not certification at all.
There are many professionals certified on AWS that aren't really able to perform properly on that platform, for example. Add the fact that plain old system administration I'd dying (less and less needed with the cloud and things like ansible)... And here we are.
From what I can tell, AWS recently offloaded their certification program onto a third party, so I doubt it will improve anytime soon.
https://www.cncf.io/certification/training/
(Disclosure: I helped develop the exam.)
(Also heard your interview on Software Engineering Daily the other day, very informative)
And, thank you. I've been a listener of Jeff's for a while and enjoyed our chance to chat. https://softwareengineeringdaily.com/2019/01/14/kubernetes-i...
The AWS Associate certs are similar to what you describe, particularly the "Solutions Architect" which is IMO common sense for a systems admin. The "Developer" one, though, requires memorizing some trivia around APIs and limits (eg., SQS queue sizes and stuff like that).
However, the AWS Professional certs are said to be MUCH harder, and while multiple choice, require reading a lot of scenario information quickly, and then selecting more than one answer (sometimes not being told how many answers). I haven't taken it but I expect to take the DevOps Professional exam at some point. I could be wrong, but everything I've seen so far, including sample questions, suggests it requires a fair bit of hands-on experience.
However, where everything I said becomes false is around the culture of answer dumps. I'm pretty sure there are dumps sold out there for cheaters to purchase. It's a pretty silly thing to do, especially because you'll be fired if you demonstrate incompetence in the field, but it does undermine the multiple choice test mechanism for certifications.
I would never try to get a pure AWS job with just theoretical knowledge.
Walk in and show a list of externally visible problems (in a way that clearly does not suggest "hacking"), and propose a project to get them correct.
I have an acquaintance that is aiming at that market:
Most applications are simply cloned from existing On-Premises Infrastructure because every project have a 2 weeks deadline and every single stakeholder just wants to stick it in. Patterns like Auto-Scaling etc is just a far fetched dream.
Supporting such Infrastructure where every server is a pet trying to survive in a Slaughter House is a deeply painful.
It's a terrible idea, but cloud providers love it because they end up getting paid a lot of money for idle capacity.
People of this kind are extremely conservative, because they are not in a position to fix major issues if something goes wrong--esp new architecture. So, they want to lift as is to aws.
A decade ago, I was involved in physically moving three solaris servers that were running un-supported BEA weblogic instances. When I powered on these three servers on a different rack, weblogic cluster failed. No IP change, just moved machines to free up racks in one row. And the people who developed the app, and the dev architect behind it were so mad at us. In the end, they gave up on that app.
That's how it goes.
For example stateless web component is a primary requirement to allow servers to fail as well as auto scale. Most enterprise applications I have seen falls flat on face.
I have seen enough cases in which customer endlessly debates why his snowflake of a web server went down and expects a RCA with god knows what details.
Get them on the cloud, then they can have the 'option' to start utilizing pure cloud type services, and migrate over.
Just reproducing their current setup on the cloud in many ways makes perfect sense.
Remember that for most companies, IT is just IT - it's an enabler, but not a huge differentiator. What 'works works', and tech change is expensive and risky.
Going to the cloud and shifting paradigms is two big steps, they can be taken one at a time.
But yes - any HN reader would be kind of 'shocked' to walk into almost any company in the world and see how 'high tech' really works. It's a valuable lesson in perspective, though.
Legacy apps asummes a certain degree of reliability from the underlaying infrastructure. This assumption falls flat on its face in the cloud. Netflix wouldn't have needed to develop Chaos Monkey if this wasn't the case.
Secondly leap to cloud native never happens because no one wants to take the pain once it works even barely. Any issues can be pushed onto Infrastructure team. It's pure and simple risk avoidance.
I said in another post that I was a dev lead for a company that decided to move to AWS. After dealing with “consultants”, realizing how shallow their knowledge was after I started learning about AWS and seeing how much we were paying them, I changed my whole career focus.
I got another job at a small company that was running completely on AWS who needed some “adult supervision”, self demoted to an individual contributor and I’m getting real world AWS experience so I can become an overpaid consultant who actually can consult with developers, devops, and infrastructure departments.
I am "lucky", I think, as my employer has already been through some very painful experiences (and learned nothing, apparently) - and they seem content to let me do whatever the fuck I want, as long as I sufficiently song-and-dance them. And yes, I've made some mistakes along the way. But I've also delivered results I didn't even think were going to be possible when I started. Advice I'd give to people walking this path:
1) You begin in an emergency. Everything is always an emergency. Because production is down. And you will always be wondering why your customers just don't drop you. But do take a moment to establish baseline metrics. Some way to measure how much you're getting, performance-wise, in terms of uptime, records processed, users, customer calls per day, cloud costs. Whatever you can pull out of your ass for minimal effort. Get SOMETHING as a baseline. As you go along, if you miss a deadline, nobody will question you if you can show progress in your metrics against that baseline.
2) The theoretical ideal model web apps may perform radically differently when you start monkeying with the architecture. If the original developers are gone, if the code is more than 5 years old, it may not be feasible to debug the squirrelly behavior you see when you change from a traditional proxy to say, an AWS load balancer. You can burn a shit ton of time trying to troubleshoot things like this. Slapping a band aid on top of 10 years old band aids is not always the right thing to do. But sometimes it's the best approach to delivering results that keep people convinced to fund your effort.
3) even if nobody else in your company has a fucking clue what you're doing, or understands it; document everything.
I’ve been using C# and MySQL and SQL Server for years. I’ve only done APIs in Node and Python with lambda. The on,y web server I know well is IIS.
It tells me whether they have bothered to learn how to do stuff in the cloud or are just applying their old patterns to the cloud (at that point, they should just have colo/managed servers - they would save money, but usually the driver for the cloud came from the C-Suite in those situations - either way some training and a cost cutting project usually pays for itself in those situations).
* what do you mean who installs patches? I think our cloud provider does.
* (same, but for "firewall" or "policy")
* "I think they turn them off automatically if they're idle"
and so on. The "devops" hype has put people who think nothing of operations at all in charge of something they don't understand.
This always kills me; the worst part is that keys are easier to use after you take 5 minutes to set them up once!
Still, it's interesting because some of it requires that you actively work against AWS to achieve (e.g. publicly accessible RDS). Could be that they gave up on, say, security groups and just found that to be the easiest way to "get it to work" (a frequent overriding directive).
In any case, I'd put some of it on Amazon too. Their documentation always struck me as piecemeal and task-oriented; lacking a sense of overall architecture or design.
But, I suppose that's what certifications are for.
The answer: "The key thing here is that businesses are not hiring fresh-out-of-certification AWS experts to take over the infrastructure of their established operations teams. They want people with solid experience, a risk-free bet that the new hire can execute the tech flawlessly. This often means insane job requirements [...] and an existing track-record of success."
So jobs openings stay empty because companies have unrealistic expectations. Is that something new and surprising? This is the pattern in tech recruiting as far back as I can remember.
If you need a special expert fast, but you don’t expect to pay a lot, then you are being unreasonable.
Judging by the claimed urgency and how wisespread the need is becoming, not every company will be able to get an experienced person for this role.
A job posting doesn't necessarily mean that they need someone fast, just that they'd like someone, someday, for a certain price. Companies have many important but not urgent goals that can wait for the right people.
Thankfully, in this field there are few barriers of entry, save for having some natural curiosity. All clouds have free tiers, excellent documentation, sample projects, etc. Kubernetes is free and open source. Technologies are interesting. Play and learn, and you’ll get a job.
Certification proves memorization. Tinkering proves interest. I don't want a candidate that is not interested personally.
APPLY the certification in some way that YOU own and you're better than 50% of applicants I see. Many have certs, some claim some knowledge.. but few seem to actually have done anything themselves... They claim cloud CICD knowledge, but it was set up by someone else, and they don't know what a security group (firewall) is.. or they deploy to CloudFront, but don't know where the files are behind that... It's frustrating as a hiring manager spending an hour on these interviews with people who expect that paper to mean something without even 1 week of their own experience to back out up
Modern cloud shops don't need dedicated staff to manually reboot servers, manually install software, or manually handle unavoidable hardware problems. Often what they're looking for is core software knowledge to add automation, fault tolerance, and scalability to their stacks under the banner of platform/devtools/backend software roles.
Honestly, AWS solutions architect cert is to easy to memorize and pass, apparently. I had someone with 2 certs not know which AWS service would work as a good "serverless replacement to running scripts".. (Lambda) and didn't know how you could schedule lambdas to run every few minutes or on a Cron expression.
Basically, people think this cert means a lot, but it doesn't on it's own. It's like getting A+ certified without any evidence you ever touched a computer component. People are putting the motherboards in the right place, but don't know how to orient it.
Don't get me wrong, the cert can have value... But I like for people who have the cert and use AWS in some way... Even a GitHub webhook to Lambda.. or an s3 static site with CloudFront... Some evidence they actually touched and used an AWS service.
So yes, I don't like freshly certified individuals, because the AWS _associate's_ certs are NOT an expert certification. Sure they are slightly hard, but they are certainly high risk if you expect a solutions associate to migrate your data center or databases. Someone with 3 years IT experience has a better shot
I'm not AWS certified, but I've been a technologist for many years and I majored in CS, so it all just sort of makes sense. Some names such as ELB, RDS and Lambda are intuitive. But some are overloaded, for example Alexa. They have Alexa for business and Alexa Top Sites (domain name research). BTW, has anyone else used Alexa Top Sites in AWS, it seems way overpriced to me.
I'd be leery of someone who knew all these names, but didn't have much actual experience as a technologist, but in an interview, they'd probably sound better than I would.
See thread here: https://gist.github.com/chilts/7229605
"I don't use those AWS services, but let me tell you what I have used". I agree, most people won't know one of: Sumarian or Sagemaker, but chances are you can easily know most features that matter for your job interests. Most new AWS services over the last few years are niche markets. A good company to work for would not expect you to know this stuff without experience. I personally don't expect someone without certifications to answer AWS questions, so it's always a plus when they can. But if they have certifications and can't answer any questions? That's a big red X. I can't speak for everyone, buy I usually have 2-3 different questions on different areas, and if you're certified and have used AWS, you should be able to get 2 of them.
Ex: what is the difference between EC2 server types M, R, and C.. my previous lambda question, and usually a question on ECS or CloudFront
I started down this road as a dev lead for another company that wanted to “move to the cloud”, but at the time I didn’t know anything about AWS. Studying for the associate cert was mostly to get an overview of AWS so at least I could talk intelligently about from the infrastructure side and know what I didn’t know from the development side. I knew I would have to dig into the SDKs to actually do anything.
They hired a bunch of consultants who were old school net ops guys that didn’t know anything about AWS besides how to mirror on prem infrastructure. They were “lift and shift” guys. Of course it ended up costing more. I didn’t know then that AWS always costs more unless you are willing to change your processes, let a few of your infrastructure people go and let AWS do the “undifferentiated heavy lifting”.
I saw the situation was hopeless and all the company wanted to do was host a bunch of EC2 instances and the netops guys and consultants were so entrenched that there was no hope of me actually getting any real world experience with AWS.
I changed jobs and found a small software company that was already on AWS but the newly minted CTO wanted to actually take advantage of AWS. The MSP they had chosen before he came aboard were also a bunch of lift and shifters.
I was hired with no real AWS experience to lead the effort. After doing a lot of green field small proof of concept projects and fumbling through, I feel badly for anyone who tries to implement anything on AWS with just the certifications, without the industry background, and not working for a company with a business support agreement plan with AWS.
You are right, AWS can cost more, but I disagree with "always". I run my personal website for $15/year, and most of that is the domain.
I would say: if you're trying to list and shift to AWS, you're doing it wrong, and it will always cost more. You also can't really fire many people aside from pure network guys. Server admins will still be needed for the virtual servers.
If you take the opportunity to redesign, you can realize cost savings. I still advocate avoiding layoffs, because you can retrain competent ops people to support the new infrastructure. The grizzled old guys that have trouble adapting are still usefull because chances are high you are not fully serverless within 3-5 years after a lift and shift.
As you become more familiar, you realize the tweaks that can be made to save money. We used to spend $600/month on 3-4 concurrent CICD workers. We swapped to docker-machine Spot-based CICD and spend $100-140/month for 12+ concurrent workers (we started at 6 and keep raising this number because the costs simply are so small)
Our front end is all static files (JS driven API calls). The $100/month server is now under $5 month for S3+CloudFront.
So I agree, AWS always costs more if you are trying to replicate a physical data center and ignore all the possibilities for improvement... But with slight tweaks (even starting with auto scaling.. or even more simply, timed lambdas to turn on/off servers overnight), you can reduce costs. Migrating to containers is also a huge cost savings, since you can pack more work into fewer servers without worrying about as much (I lived the nightmare of virtualenv's and managing 12 apps on 1 server.. I prefer containers now as an ops guy)
For builds we are completely serviceless. We use CodeBuild with mostly the prebuilt Docker images and one custom image to build some .Net Framework code. We use CodePipeline to deploy lambdas with CloudFormation and Code Deploy agents for EC2 instances.
For any new work, we have to justify a need for servers instead of using lambda for both APIs and back end ETL work loads - with step functions. There are a few back end workloads that we will need to transistion to Docker eventually but even then we are looking to use Fargate.
https://mobile.twitter.com/QuinnyPig/status/1091041507342086...
So much this. I've seen job listings that say "DevOps" and they really mean "SRE" or "Software Developer to own the whole SDLC w/On-Call Rotations".
Of course, the industry is changig but I think the industry hasn't let the "lead-in" occur, which existed in times before, and carry-over into the new momentums. For example, requesting 5+ years' experience in enterprise-level deployments in Azure and/or AWS isn't precisely synonymous with 5+ years in OOP.
I think that the reason for adding to the insane job requirements is that they're trying to condense the whole SDLC down to three people - at most - (e.g.: a PM and two devs) and still produce Fortune 500 level code products for the world to consume; thus, making the actual intial cost overhead to produce the initial product much lower.
This is, of course, pure conjecture and you're welcome to tell me that I'm talking out of my ass but, as far as my interviewing experience goes - so far, they want someone who's owned a small product (from an SRE perspective) for a few years' time, which doesn't really "translate" across the industry, yet.
SREs might come from Google, FB, Twitter, MSFT, Avanade, etc. but doesn't seem to - as far as I can tell - translate to an industry standard, across the board.
So, you'll have a developer with 7+ years' experience with OOP and maybe does the full-stack but, because he's never been exposed to things like being responsible for the deployment of the product (because it's responsibility of another team at his company, to make sure it goes through a security review cycle, has been tested in tested pre-prod environments, etc.), he'll automatically be "disqualified" from being considered for the given position - because the company doesn't want someone who can learn it, they want someone who can do it - literally, starting the next day.
(Sorry for the long tirade but the insane requirements are only insane because we don't understand the rationale behind them; understanding the reasons "why" helps us determine how the whole sector is shifting to new "paradigms". <insert Office Space reference here>.)
Those people may stick with career, but not with company. Moreover average startup does not last two years either, so the company won't exist at that point.
After about one year, I had a discussion with my manager, I said I did everything I said I would do to help him meet his technical goals, I have both the theoretical knowledge and real world experience and my market value is $x. They gave me $x and now I have no plans on leaving soon.
But what they should do is have hiring managers who understand that people with experience and wisdom, and ability to learn, can be applied to new technologies more effectively and securely (with past experience of seeing what goes wrong in infrastructure) than people with 5 years of the latest buzzword technology. Have a look at DevOps jobs descriptions lately and notice:
. 5 years experience in AWS/Cloud Deployments
. CI/CD pipeline
. Containers a must (Kubernetes / Docker)
. Basic bash scripting
. One of Python/Ruby/Node/etc.
. BS/Engineering Degree or equivalent experience
That's it.
Now you understand where horror stories in infrastructure come from, and why companies hire expensive consultants or "DevOps Shops" to repair shoddy setups.
This, however, is one step above the following:
"We're seeking a full-stack developer to develop our front and back-end systems, including designing and enhancing AWS infrastructure utilizing Docker / ECS and CI/CD pipeline via Jenkins or equivalent CI system." That's the Startup 10X Engineer Special that pops up a lot.
Yup, good luck!
I've been casually looking at job postings and sometimes wonder what drugs the job poster is on when writing job requirements.
Certifications are moot. I don't care if you are certified, or if you have used a cloud (or kubernetes) following a tutorial. I do care if you have seen enough issues throughout your career, so you can debug every time (not when, nor if) mud hits the fan. I don't care if you can type a port in a yaml file or in your browser. I do care if you know how many ports are there, what's the ephemeral port range and so on. Systems are 75% experience, 25% basic knowledge, you can't get away just with certifications which matter only after you get experience and knowledge.
If you are experienced in ops, I know you can learn and work out any cloud. If you claim you are experienced with a specific cloud though, then better have to share a ton of war stories or it will just be a huge red flag on your CV.
Let's call it for what it is, clouds, like data-science are hot fields right now. It's only natural that many people will go after a good-paying job claiming expertise in the field.
YMMV for anything you can’t see or touch regardless of the size or experience or reputation so far.
There are definitely some workloads that lend themselves to serverless and some that don't -- it's not the silver bullet a lot of evangelists say it is -- but when it works it just works.
Companies used to own and run their own machines on-premises. But doing it properly (HVAC, power, raised floors, standardized racks, management, redundancy, preparedness...) is expensive and not actually most company's core competency. So they moved to colocated datacenters, where a competent company would take care of the infrastructure for a fee which would hopefully reflect a discount based on savings from scale.
But managing generations of servers at colo datacenters takes manpower for hardware replacement, upgrades, cabling, and generally doing things right; the customer companies mostly don't have that as a business feature but see it as a cost. So managed infrastructure services came up, where the datacenter company leases you the hardware (standardized so they can have spares) and racks it for you and puts in network switches and remote services and gets it to the point where all you have to do is log in to a console server and install your OS and start deployment.
But sysadmins who can keep track of security updates and package dependencies and keep an OS properly organized are relatively expensive, and what used to be
"well it works on my laptop"
becomes
"well it works in my container"
and the container itself gets shipped out. It's cheaper not to do security work, you know.
So VM and container accepting services come up, where the datacenter company now runs a hypervisor and managed storage systems for you, in exchange for the salary of those sysadmins. The start-up costs can be much lower, to the point where it really doesn't make fiscal sense for any tiny company to do anything themselves except ship their containers/VMs out and puzzle over IP allocation schemes and load-balancing services.
It's a trap, but a delicious one.
So now your company is hooked on the ease and speed and cheapness of just spinning up another container or VM any time someone expresses a desire, or even automatically, when you get the bill at the end of the month and you managed to spend how much? That's ridiculous. Why are we getting billed for containers that we don't even need to run all the time?
Along comes "serverless", which is a logical successor of inetd. Yes, inetd, the very old "internet super-server" which would read a table of ports and programs, open all the ports, and when a connection came in would run your program and connect stdin/out to the socket, isn't that great?
Serverless is just a management system that spins up your very restricted single-function program on demand -- now it answers an HTTPS API instead of a raw socket -- and does the accounting work to make it profitable.
Complexity is simultaneously the enemy of correctness and the source of profit.
I've also seen serverless used to describe services like Google App Engine and the Azure App Service. They're a bit less granular than cloud functions, since you deploy entire applications instead of individual functions. Again, you don't really have to think about servers or infrastructure. You deploy your code, and it just runs and scales up as necessary.
Docker and k8s and containers are great in general, but require a lot more hands-on active management than the services I usually see described as serverless. Though even here, services like AWS Fargate and Azure Kubernetes Service and (I think) Google Container Engine take a lot of the manual work out of container orchestration.
It's Kubernetes and microservices! Even though Kubernetes is what, four years old, where I am, numerous employers are chomping at the bit and demanding that you be an expert at these. Now, I'm a crusty old Linux sysadmin/programmer who puts "SRE" and "DevOps" on his resume, and I'm noticeably getting spurned because I don't have "5 years experience" with "containers".
At first I was worried that my skills were obsoleted, having worked in large enterprise environments doing infrastructure engineering for over 15 years. I worried that I was too "systemsy" and not enough "cloud-y". Oh, and even though I spent years using TeamCity and Jenkins, I was also spurned because I didn't use the right lingo within "CI/CD pipelines" and demonstrate a Docker deployment system exactly like they used.
Then I realized something.
The job descriptions are asking for five years experience total, number one, and the second red flag: the salaries and contract rates are lower than the rates for large enterprise-y type jobs (minus management).
Conclusion: hipster tech coincides with youth and coincides with companies following trends and buzzwords over wisdom and experience.
Back to the point. Someone with a strong basis in the core disciplines of systems design, infrastructure automation, perimeter security and so forth are going to be able to adapt to AWS knowledge, as I have (for the record I have two AWS Associate certs and didn't find them hard). And as the author points out, there are these 5 year cycles where all we look for is experience in certain tech. Is it any wonder that there are horror stories about fragile and insecure cloud environments?
It goes back to the incompetence of the hiring process which we see again and again discussed on HN.
So what we have here is a failure in the hiring process. The people writing the jobs don't actually know the space, but are looking for candidates with literally impossible amounts of experience.
Most Federal agencies are on or moving to Cloud services right now. They are squarely toward the top right of the S.
I don’t think you can say only “large trailblazers” are there. We can argue about how many services have moved and how it’s being used but the whole “let someone else manage it” argument is long done and normalized.
I remember panic-talk when outsourcing began. When offshoring arose. Etc. None of these extinguished the need for IT workers.
I view the cloud as the next generation of the mainframe. By that, I mean your experience and understanding of the environment (nobody will master all of it) will be a career differentiator. Knowledge of Kubernetes, Istio, etc. will help you to navigate the next career paths. System admin skills (and associated extras, like networking) will likewise move you up the ladder.
Bill Gates once said there is an infinite number of problems yet to be solved, an infinite amount of work to be done. I believe him.
The senior sysadmin at the DMDC around mid California coast: only one certification, VMware. A French offensive exploits broker, solid researcher and reeigne: no certs. I only have a few certs because work paid tens of thousands for offsite training. A few certs are okay because corporate HR wants staff to demonstrate continual learning and achievement, lots of low-quality ones (A+) are a negative signal. It's more important to keep up on news, tools and techniques on the front lines.
My opinion is managing \aaS needs solid rack&stack ops experience at scale. For IT folks: either start huge if you can or start small and move towards huge. The biggest challenges are customer communication, managing expectations, anticipating and planning where possible and showing empathy... it's a 5% technical skill and 95% getting along productively game. Communicate the business impact, don't overwhelm customers with unnecessary* detail.
Some of these firms realize they need to renovate their IT
but they attempt to do this through hiring “Digital
Transformation” roles which do little but frustrate the
revolving door of hires because they have no executive buy-
in and budget.
It's impossible to find enough experienced in the new field because it's so new and/or growing too fast. But the HR department or technically-inexperienced managers don't understand this and complain to politicians et. al. of a "shortage". (Those in know have seen the pattern multiple times.)
The answer is simple: find a similar field and accept a learning curve for the employee to transition. If you need something quick, hire a consultant to both move the project along and to mentor the new employee.
I don’t care if you know what an auto scaling group is. I care that you can write code that automates builds and deployments. The AWS part is easy and you can learn what you need to know on the job. In my experience, coding is something you either can do or you can’t, and if you come to me with a cert and no programming experience it’s very hard for me to know what you’re capable of.
i spend like 5% of an interview asking about AWS stuff and like 50% talking about code and automation.
Maybe I've had bad luck, but I've encountered very few prospective employers who were OK with this. The ones I encountered expected to hire someone who would fit in like a jigsaw puzzle piece and they wouldn't settle for anything less. My confidence in my ability to pick up almost any tech quickly did not impress them, and they latched on to my lack of that particular bullet point and kept hammering on things like "oh, you don't know what a [platform-specific term for auto-scaling groups] is?..."
I answered all the questions, have a proven track record, am self-taught for 15 years with increasingly high salary / rates, and yet for this one guy I was incapable of learning quickly. This was determined in a 30 minute whiteboard interview I passed.
Some of the interviewers have extremely low sociability on the psychological scale, even if they're tech geniuses (which isn't even evident). Everyone talks about the skillsets, S-curves and so on, but no one talks about the mediocre evaluation process by those who aren't good at it.
Wow. Just wow.
A lot of talented people are going to find them self in a problematic situation because the area in which their talents lie is only going to be handled by low wage jobs at google, Azure and Amazon. Even big companies won’t bother setting up their own hardware because the people are costly even if they have slight savings on private hardware.
I have no idea what makes you say that. As far as I'm familiar with these roles, none of them are low paying. These companies tend to pay very well because of the amazing scalability involved in these roles. Any engineering work that scales linearly with the number of users is automated almost immediately and the focus is generally on very high-level work.
Now don’t get me wrong there will always be avenues for the most talented players, but the crisis will come when the 45 out of the 50 strong infrastructure team at every larger corporation are no longer needed, and the last 5 will end up doing work that’s completely unrelated to their prior expertise.
I've seen similar and they tried the following:-
- We'll provision VMs for you. Raise a ticket.
- We're doing "Hub & Spoke". You're not allowed to route any internet traffic except through our inspection proxies.
- We've disabled the API. You can only use the Console.
Basically, a couple of old school guys will do anything they can to disable automation, as otherwise they'll be accepting they can't really contribute anymore.
- A mail server which was an open relay, promptly shutdown for abuse
- Every single internal server on an external IP address with an allow any/any ACL
- Brand new environments built with PHP 5.0 in 2018 to run new development projects (EOL over ten years ago)
- Managers patting themselves on the back about the power of Devops
Guys not all of you are the mythical 10x engineers working on the ground breaking stuff. Deal with it.
We have had most of these technologies way before they became commoditized. What we have done is that we have made them cheaper and more accessible to your average joe.
Containers ? Give me a break. We have had Solaris Zones provisioning mechanisms at large telcos before any of you even knew what a container is. People have been provisioning jails/zones with a click of a button for ages.
It's funny to me because just 10y ago people like you were yelling and screaming about losing jobs to offshore developers in India and Eastern Europe. There's no apocalypse anytime soon.
Things are getting revamped, they are better, faster and what's more important they are accessible. Just because you know how to use docker does not mean you are able to manage a production ready infrastructure. AWS or any of the big providers are not a silver bullet. Never will be. They are at the end of the day very costly services not suitable for all business.
I have seen it many times with big data, chatops (we wont ever need to login to the system! we will do everything via hipchat/slack/whatever), openstack (who even needs aws?!).
Yadda yadda.
We built immensely complex systems that allow us to scale. The thing is we have increased the complexity so much that we no longer can justify hiring incompetent.
Simplicity is beautiful and we will have a renaissance of simpler systems. Something along the lines of what Go brought to the game.
Where the biggest adopter, Kubernetes, is ironically a way too complex system of automation routines that still regularly fail in production environments. And oh dear, if they fail, it's gonna be hard to figure out what exactly went wrong. We uncovered a bug in the job controller just yesterday that was probably not biting anyone else due to how a race condition usually plays out, but this kind of bug is way too common.
Yes, but there are better tools suited for the job. Ironically, or perhaps not, Java or .NET would have faired better due to maturity of debugging and introspection tools.
regularly fail? Do you have any data to back that up? No, a google search for one kubernetes failure doesn't count as proof of "regular failure"
I should note that by regularly I mean edge cases and quirkiness, not that your run of the mill cluster mostly using deployments is going to randomly explode. But our teams still find ways to regularly break it in unexpected ways.
So I guess you could call that anecdotal if you want.
Go is a (IMO too) simplistic language.
It does in no way follow that it will be used to build simple systems.
Go isn't simplistic. It might be seen as simple but it isn't so simple that it loses utility. There's plenty of nuance and elegance in there to solve real problems.
A simplistic version of go would have a bunch of wizards and a giant code generator that runs after you click Next enough times.
You mean like Go and how people compensate for lack of generics? Generating code?
That kind of simplistic?
Not to mention how mocks for unit tests are generated. Golang is a step back from Java and .NET