You can't git clone a team
virtualize.sh
virtualize.sh
We’re talking about skills that span kernel-level programming, hardware quirks, low-level debugging, distributed systems, security, orchestration logic, even the capability to work with the UI/UX team... and the ability to explain all that without scaring interns. You can’t just hire for that. You have to grow it. Nurture it. Beg for it. Or in some cases, resurrect it.
If you are that person, what is the best way to market yourself? I am the person described. I've got experience from poking registers in firmware, to wireline transport protocol implementation, to infosec, to writing microservice framework middleware, to pipeline orchestration at the OS level, and on and on. In the last week I've debugged Linux UDS issues and TLS cipher suite problems, and wrote code to provision WiFi-connected devices over BLE.
But it's incredibly hard to demonstrate that in an interview, if I can even find a role that warrants it. You're not going to find me on a university campus or in a research lab because I'm at a FAANG trying to pay my mortgage.
Hate to break it to the author, but there are plenty of people who can write a driver, bootloader, distributed systems, PAXOS, etc. but there's NO JOBS DOING THAT so they all work for <generic SaaS company> making <generic NodeJS app>.
The author mentions their grand strategy is fishing for talent out of universities, which is probably smart. Pulling people out of generic web dev world to go write "container orchestration logic" (or some other niche) is going to be a hard sell - most of those people were burned before by straying too far from the lucrative web dev jobs. Nothing like going into job interviews and telling the hiring manager that your last job was some ultra-obscure niche.
My first job out of university was working on an embedded Java runtime, with a group of peers from said university. Which sounds similar to what Virtualise.sh is doing.
Now, 20 years later? Tech lead on a Typescript/Flutter/AWS internal system.
In some ways I'd be excited to write container orchestration logic, or hack on a hypervisor. There's precious little of that work available, especially so in a small country like mine (NZ). My CV is crazy enough: being burnt isn't the issue, just finding the work is.
The cynical part of me wonders if the other reason to hire people straight out of university is that they're cheap...
(throwaway account, my CV is bad enough already)
Or the dumb leetcode stuff because "we are going to be cool like Google over here".
So many gravitate to companies with more sensible hiring practices, and that is how a company loses candidates, if it is one of those unicorns that the whole planet wants to work on.
So many times this. It doesn't even have to be five years to disqualify so many applicants these days. If you aren't working on it currently, there's already so much bias against you in the hiring process at large (e.g. the "I don't consider anyone that isn't currently employed" mantra that pops back up now and then).
Now I have a generic web dev job working on a generic nodejs app. There are many more jobs available and in general it's much easier to have a remote web dev job.
I've also done a couple sessions of peek at a problem in production and fix it / tell people how to fix it based on reputation, which is networking.
Oh, and one more thing: never ever mention any experience with mail handling, or you'll get roped into doing it again. People remember, even if you only said it once. :P
Make sure you are demonstrating it to the people on your team so that when they leave and go somewhere else, they can hit you up. This takes some time.
And, sorry, you have to get out and hit some gatherings in person (hackerspaces, meetups, professional meetings, etc.)--online-only isn't going to cut it anymore. With the AI garbage clog of inline interactions, your professional network is back to who you know in-person.
Demonstrating that you are a stable hire who will fit well with the team is almost all that matters. You wouldn't be having an interview to begin with if you didn't technically qualify-ish. Even when speaking to technical interviewers, you will intimidate them. Actually selling yourself comes after your first 90 days of acclimation.
Even past that step, it's a bit random. I fully admit I ignore the resume when doing interviews, as it's a bias for the interview role I get, which is typically working through some coding problems.
People like that are in demand. Our company just filled few position like this, and hiring was hard. Some people were rejected because of not enough low-level knowledge, some people were rejected for not having enough high-level knowledge.
Agree with the TFA's author though, people with all the skills are super rare.. We were lucky to end up with few hires like that, but most of the people don't have all skills at once, and they have to learn on the job.
Let's rephrase that:
I'm at a company with top of the line talent in many different domains and I prefer the compensation of this company over networking to find someone who knows the hiring manager of a team like this.
You have the easy way into this. For people without this networking, the best advice is to make them come to you. You gotta share that knowledge out there and get regards in various communities online if you can't stand out in a batch of resumes (and no one these days can, for reasons entirely the fault of HR and ATS). Do that and your newfound network will connect you.
Tldr raise your price and they'll belive easier.
If that has worked for you, that's amazing! But it seems really counter-intuitive to me.
It's misguided of course, but that's what they think they want and if you say "I've done all sorts of things and I'm good at all of them" they'll hear "I don't have much experience with anything" and discount you.
So it's better to pretend to be a specialist.
- resumes which claim generalist tend to be SEO’ed than real, or AI generated and they will say experience dozens of technologies most of which would be untrue, throw everything and see what sticks .
In most cases it just tells me what they have heard of rather than what they know especially if they are not very senior .
- It takes really hard effort to be beyond a superficial generalist, even when they know some of those skills and remember them during an interview it is not with a lot of depth they pursued only in passing interest not in a professional capacity (that is fine, but it is a lot work to make the distinction in evaluation on all relevant skills in a timed interview)
- It is harder or simply not viable for a specialist to interview a generalist , so you need to have few to hire more . If you don’t have any or they are not doing interviews you are not going get more .
- Being a generalist with depth of understanding means you are the type of person who needs to understand things properly before doing them . .You are lot of time learning things which are not required for get the task completed.
That means either you need are prepared to spend a lot of personal time and be perpetually stressed or be slow in completing something .
It is hard thing to master to let it go . I don’t think I have learned it yet
- Complexity and depth of technologies change a lot in short duration when they are corporate backed .
I learned Linux architecture or vim or git 20 years ago they haven’t changed much. You can be productive in any of these to stacks very quickly even if you didn’t use them for years .
Last I worked on android or swift is more than few years ago . I doubt I could even build a serious app without spending major time . To be productive to learning curve is steep and the prior knowledge is limited in usefulness.
It's just how my career swung me. I would have loved to have developed as a subject matter expert, but everytime I get into the swing of things, layoffs came around or the studio shut down. Now I'm freelancing and that by nature requires a generalist approach.
no one's really investing in specialists, so I have no idea how the millentials/Gen Z of the world will ever get to properly specialize. Specialization requires time to master something, and that time implies stability to do that thing.
>To be productive to learning curve is steep and the prior knowledge is limited in usefulness.
business wise, sure. It's a shame all business sees as "productive" is based on how many widgets you churn in that time. It's no surprise such companies want to force AI into it without quality considerations.
Ha, I feel seen. Though also I wouldn't frame this quite so negatively. I've seen a lot of tasks "completed" by people who just got the job done without fully understanding it and they frequently get it done badly.
Changes also happen, but they are kind of glacial with areas where we are hunting the new shinny every here.
Usually I tend to be a laggard on the adoption graph, the large majority of stuff hardly makes the curve, and I have better things to do with my time.
Plus that vintage stuff that isnt' cool to write blog posts about, usually pays good enough.
Stuff like Android or iDevices suffer from yearly fashion, platform owners feel compeled to reboot the development experience every year, forcing app developers to keep up, and also as means to sell new devices.
That's something hiring managers do find compelling.
Most HR hiring managers, and headhunters alike, have a real hard problem trying to see the point of generalists.
They want the easy way out, what to focus on.
As I usual say I tend to do whatever, so is the life of agencies, however when applying for positions sell the skills relevant for the position.
Not e.g. "I can do .NET and Java", rather "I can do .NET, proud of X, Y and Z projects in .NET, and by the way, I may do Java as well if needed, there was this project...", something like this.
It got worse in last two years. Now Senior Engineers must have exact combo of the weird tech stack and tools with N years of experience exactly as the existing employees, else gtfo, you don't even get screening calls. You don't have something from nice to have list? Lol, why are you wasting our recruiter's time even? She needs to use GenAI to write her next rejection email.
Also, your 15yoe does not matter, unless you are coming from direct competitor, in which case your 1.5yoe with internship is also excellent.
The minimum level of sophistication required to be effective in systems software has increased dramatically since the 1990s, when I first started doing systems software. The kinds of systems software we were putting in production in back then would be considered a trivial toy today. This shift has placed an enormous amount of upward pressure on the minimum level of experience and skill that would allow someone to become productive within a useful amount of time.
It is no longer feasible in many cases to have companies effectively subsidize the development of highly skilled systems software people. The training time has become so long that companies will never recoup the investment. It is easy to see how the incentives have created the current situation and it is not clear how to address the shortage. Even before the current situation, it was widely noted that most systems software developers were self-taught over many years rather than trained up in a structured environment.
if the company would rather burn out and age out all their talent than invest in the future, I don't know what to say. For niche domains, you can't treat talent a a pure business expense. That's how you lose out all that talent to China and suddenly you can't compete at all. If anything, training should be treated like insurance.
>It is no longer feasible in many cases to have companies effectively subsidize the development of highly skilled systems software people.
"subsidize" implies that somehow engineers are saving money by choosing to take these more niche paths. It doesn't make sense. You have a product and you need talent to develop and maintain it to compete. There's no "subsidies" in a business relationship like this.
your choices are few and simple: train in-house, lobby schools to train for you, or publicize your tech and hope people train themselves for you again. It seems like these days companies aren't looking out long term though and just want to jump the ship that has plenty of time to steer past the iceberg everyone sees ahead.
I would love to work on low-level, systems stuff (anything as close to the hardware as possible), that's even my education and area of expertise. BUT SaaS companies in reality:
- pay better
- have lower costs
- are way easier
- don't have that many (or any at all) geographical restrictions (e.g. importing hardware prototypes).
People follows the money.
okay. Promote them with proper pay then. This HR debacle of putting more budget into hiring over retaining is entierly self-inflicted.
>People follows the money.
Well that love sounds more like a low priority whimsy in that case. I don't think tech workers of all people are ones who ever complain about low compensation. Unless you work in games, I suppose.
Does the high skill standard for surgeons mean the market for surgeons is saturated?
Sure, it is possible for the market to desaturate at some point as key people leave. But at that point, assuming suitable, per the earlier comment, replacements don't fill the void, the market will either come to accept lesser skilled people or it will come to accept fewer surgeries, returning to equilibrium. Not exactly magic.
Maybe there's a good argument against training, but it could also just be irrational and stubborn in this case.
If you get the ratio wrong, with too many juniors and weak technical leadership then you will end up in a very bad place in your code base.
In terms of value, even if juniors are half the cost, it is much wiser to hire one senior instead of 2 juniors for the same money.
Maybe in terms of pure productivity, but if you can match hiring to roadmaps you can give them more approachable/further from revenue work for which seniors would be an overkill. Etc. I'm just saying anyone with a simple explanation is only telling you part of the story.
Besides, hiring is expensive. If you say live near a university, you have an edge in finding talent.
My experience suggests there is no such thing. There is work that seniors might not want to do, but if you hired well then they will be professional enough to do it, if it really needs to be done. And it takes some experience to determine which “non-revenue generating” work (e.g. tech debt) actually needs to be done, to advocate doing it to the stakeholders and to actually do it well.
Juniors need a lot of supervision and that is not free. Which is not a reason not hire them in the first place, just that that it should be done mindfully.
This mentality is part of the problem. You can't fundamentally treat juniors as a profit center. Especially in more niche fields. They need to be trained and schools don't cover your specific pipeline.
maybe it doesn't "need" to be done, but some refactoring work will pay off to turn your juniors today into seniors tomorrow. If you can only think in productivity charts, then you don't hire juniors.
That said, in my previous job in a startup we hired very junior engineers and gave them plenty of opportunity and support for growth and several people did stunningly well. Pity the company didn’t do very well (IMHO due to focusing more or this sort of thing rather than making money).
The truth is bound to be somewhere in the middle.
Mentoring is a key part of technical leadership, way easier to help the talent grow into your requirements (even if they are under-qualified, for now)
once you get a few years under the belt, sure. You're probably fine doing everything online. I can't think of many remote only studios hiring juniors anyways.
There's few exception when we don't have the choice, but really, I cannot imagine myself to be full remote without having seen a company "in the flesh" before.
That said, I do agree with the premise of the article that it's hard to learn "the stack", especially with the advent of generative AI.
"Back in the day", when google spat out a link to something resembling your problem, you still had to deconstruct the answer given to apply it to your particular case. That required some sort of understanding, or a mental model of the domain. With ChatGPT I find myself copying/pasting error messages between two browser windows, or a terminal, not really understanding what is going on.
No learning is happening. Does not bode well for new folks coming in.
It's not a promotional piece of something, it's my personal experience as a CEO and co-founder of a company using Xen as the core of our stack. I like to share my views in a transparent fashion on how it's hard to do very technical stuff, not just for technical reasons, but due to the lack of people trained for.
But hypervisors did not disappear, they just got replaced. When we run virtual machines, they are usually backed by KVM (low-level) and qemu (higher layer). Sometimes there is libvirt on top of it too, but running qemu directly is not that hard.
And there is plenty of exciting research about this stack, for example KVM can be driven by things like firecracker and crosvm, and there are some rust projects targeting it too. There is also BSD's bhyve which
My impression is that it's not that people find hypervisors in general are boring, but just Xen specifically (or maybe all classic Type-1 ones? hard to tell with Xen being the only open-source representative).
I respectfully disagree with much of your comment.
First, this wasn't intended as a promotional piece. It's a personal blog post where I share some of the challenges involved in building a full virtualization stack — a stack that happens to be fully open source. It's unfortunate that sharing real-world experience is sometimes immediately perceived as promotional.
Second, I think there's some confusion between using a hypervisor and mastering one — or building and maintaining an entire stack around it. KVM/QEMU is widely used, but it has significant issues, especially regarding security, performance, and latency consistency. Very few groups in the world are actively trying to tackle these challenges holistically (even major players like VMware have made some questionable shortcuts).
When it comes to low-latency, real-time use cases with a strong security model, Xen remains unique among open-source hypervisors. It's definitely not boring — in fact, it's one of the few that enable certain classes of critical applications at all.
We also work closely with academic research labs, and I can tell you: there’s still a lot of exciting work happening around Xen — even if it's less visible than buzz around newer projects like Firecracker or crosvm.
Sadly, the program I was supporting just had all of its funding yanked, I expect to get laid off tomorrow.
Sometimes, probably most of the time, it is better to work through it and understand an issue than to blindly copy pasta.
Recently got into a completely new language/framework and used Copilot to understand what was going on. I still made all the big decisions and wrote most of the code myself.
You can definitely use AI to avoid learning though (up to a point).
It goes on to say that it's hard to find and develop expertise for low level software like hypervisors.
What's the connection between the topics? It feels like two different rants.
If it's difficult to find kernel developers then wouldn't it help to not require them to also know web UX?
That means hiring two people, and in $current_year, companies expect one person to know everything. Sysadmin, backend programmer, frontend programmer, designer and a DBA used to be different people not that long ago, now they expect one person to do all that... + it seems they want kernel development experience now.
A single person can in fact write a program for a computer.
People who came after you would write it vb6 people who came latter bootstrap.js or use material icons.
If you want to do your own networking, or run email, that's a whole 'nother specialism; but just running a website is easy enough for one person to do.
just think of your favorite video game character in 2000 and then one in the 2020's and consider how much tech is needed to render, animate, light, and conceptualize it. in 2000 this was all done by maybe one artist and one gamedev, probably making a character with some hundreds of polys at best. now that artist has a pipeline of riggers, material artists, animators, and concept artists, while that single dev became a graphics programmer, gameplay programmer, tech artist, and build engineer.
It's like moving from custom built cars to the assembly line where someone's job is putting in one screw. I understand it's cheaper/faster because you can hire anyone unskilled for cheap but cars were all suppose to be identical. Software should be unique (if not just copy the last thing built) but I guess when it comes to major games things are more of factory throwing millions of pixels of characters at existing game engines while copying gameplay of successful games. That's why games are shovelware these days like a netflix original.
Jack of all trades, master of none. This is why we need clusters, "stacks" and "clouds" on the server side and gigabytes of ram on the client side + many megabytes transfered, just to show one simple weather forecast website that gives the user the same amount of information as a WAP site did on a five line mobile phone back in GPRS times.
And if you find or train those low-level/system-oriented people, they also need to understand how a feature they build will be exposed functionally to a user (and why they need it in the first place). Because things are not make into thin-air but required to work in a bigger picture (ie: the product).
I don't see "pay like Google" listed.
If you are looking for the place to hire new engineers, or researchers to work on an advanced project, I would recommend it too.
The proposition by these companies for new hires is: "Put in a lot of work, longer hours and solve difficult problems, so that (after taxes) you can earn 2k more a year than someone with a far more relaxed job".
The only people you are going to find are the very few ones intrinsically motivated to forego their social life in favor of your job offer and new graduates who don't know any better and will burn out and resent you and possibly the entire industry for the rest of their lives.
I will not put in exceptional effort, unless you are offering me something exceptional in return. And no, the "opportunity" to work hard on some problem does not count.
Basically all of Europe has this problem, companies are looking for people who put in exceptional work for average salaries. The problem isn't that these people don't exist or that you couldn't find them. The problem is that everyone who fits your role knows what kind of effort it involves and knows that their time will never be adequately rewarded.
Seriously, book a plane ticket to ATC/OSDI, EuroSys, etc and talk to all the people there. The good ones are already hired by one of the big established players (FAANG, Red Hat, Intel, etc), which is why you need to offer competitive compensation to lure them away.
Onboarding is important Training is important Retaining is important
Maybe your system is 100% documented, conventional (looking at you Rails, Angular), debt free, tested, instrumented - but more than likely it's not.
But if you get down to staff who can't teach a a system, including product teams that don't respect feature overload, value, or internal feature training. If you prioritize new features over team development and training (aka a team that doesn't know the system), you're likely to get muddy with existing features both technically and use facing.
Sometimes it's even a catch-22 situation, where the technical/generic knowledge is already hard to find, but you absolutely need it to train more junior people. Luckily we found such experimented people, but then you also need to use their expertise to actually fix stuff and not just mentor juniors. A very very delicate balance to find, especially in a timed market.
I had to chuckle.
LLM are quite good at explaining systems and frameworks. I would never got into kernel programming without Deepseek guidance.
As for universities: too expensive, too much paperwork, too slow, too elitist.
However I agree: to learn a topic, LLMs are providing a great speedup. As a CEO/co-founder, I have no issue to hire people without a degree if they are good at what they do. However, our biggest chances are to scout directly in universities to find motivated students (motivation >>> everything else)
Map knowledge doesn’t make you able to drive. Driving happens in real time - steering, obeying road rules, avoiding accidents.
In engineering, the real time part is when you use mental models you’ve painstakingly developed - how the hardware works so you can debug it, what syntax is valid in your programming languages, what APIs exist in your stack. But, those depend on knowledge - so mental models can be self-taught with AI as tutor in your own time. The LLMs supply the knowledge, you integrate it via study - or a side project.
An organization cannot clone itself a team, it’s true. They need engineers now, not in 6-12 months. But - a motivated individual can use AI to make themselves employable as a full stack engineer faster now than at any point in history.
How cool is that!
The lack of training material for LLMs is of course a lack of training material for people too. Some areas of software have a long history of relying almost entirely on an oral tradition to pass down knowledge. This has some advantages but it doesn’t scale and it makes it basically “dark knowledge” for LLMs or people without access to those that know it. If you want to get into an area like this, you often need to find a way to spend a lot of time with people that already know it.
We call this "tribal knowledge" in games. I despise it. It's done for a few reasons:
- NDAs make public knowledge a landmine. And every game studio makes you sign NDAs. Even at the interview stage.
- Churn. No one gets time to really develop expertise as they work on a project for 2-3 years and then layoffs come. Only a relative few become experts, and probably not because of the studio itself
- lack of incentives. Games aren't very connected with acedemia to begin with, despite relying so much on cutting edge tech. So the best resources for sharing such techniques is shafted. This is slowly getting better as more tech conferences talk about games tech, but it's a pretty slow tricke unless you come from one of the largest studios and specifically come in for R&D.
>If you want to get into an area like this, you often need to find a way to spend a lot of time with people that already know it.
All too true. Open source development is one bastion for this, but that's overall why I keep trying to stay in this domain. You literally can't get the knowledge elsewhere. And it's knowledge that directly leads to better looking, more optimal, and less buggy games overall.
There are some elegant and sophisticated techniques related to database kernels that have been passed around for a decade or two over beers that still don’t have a single reference in literature that I can find. The original researchers probably stayed quiet because it was under strict NDA but also likely retired years ago. No one writes it down because it sort of feels wrong to claim second-hand knowledge of unknown origin as your own, or to even lead people to assume as much. You show people, and they think it is amazing, but when they ask you for references or sources you have no idea where it came from.
There is a real gap here and it is getting worse.
> No one writes it down because it sort of feels wrong to claim second-hand knowledge of unknown origin as your own, or to even lead people to assume as much.
Wouldn't it be possible to publish informally (say in a blog post) while fully disclaiming first-authorship or invention?
okay. So what to do in the meantie while an LLM can no in fact help you much with kernel programming? People love talking about the future but people also have current business needs.
> a motivated individual can use AI to make themselves employable as a full stack engineer faster now than at any point in history.
I'd love to hear one case of that happening. I'll even take someone ramping up from zero to freelance work as long as they have a good portfolio.
Seems they are plucking self-starters directly from universities. You could either use that funnel or build your own with the same idea.