From my perspective: They've got way too many people doing a lot of meh work and feeling really ... important about it. And it's on the whole asphyxiating the industry.
My understanding was that asphyxiating the industry was always the plan.
For those who worked in industry pre-2008, one of the major drivers that allowed new startups to be so disruptive was that startups could very often compete with the then major tech companies on compensation. If there was a difference in pay, it was usually minor and could easily be made up for by better quality of life and a nice promotion in title.
Once Google and Facebook started growing it's clear they realized that every talented engineer on the market is a potential threat. As time has moved on the gap in pay between startup and FAANG and now startup and any company offering RSUs is just too large for startups to compete for talent.
I generally prefer startups to large companies, but can't find a single startup offering comp even remotely close to what I have in my current role. Unless I started to absolutely hate my current position, and found a startup that had essentially my dream role, there's no way I would seriously consider a switch.
My suspicion is that as funding for startups starts to dry up as money becomes more expensive, we'll see large, high paying tech companies start to let go of more staff and start driving comp down. When the market cools Google and similar companies asphyxiation strategy will have been completed. Startups starved of talent and finally starved of dollars will be no threat, that means they can safely reduce labor costs while maintaining market dominance.
They totally didn't care about our product or the productivity of the employees they brought in. They let us sit and twiddle our thumbs and "rest and vest" for at least a year, after making a big show for the DOJ about how this aquisition wasn't an anti-competitive move ([1])
And I hung around there moving protobufs around and fixing the odd bug for 10 years but getting paid stupid money to not go and do anything for anybody else or keep on top of industry trends etc. (Not claiming I'm some shit hot engineer, just speaking for my personal situation)
I don't think most people who are 'organically' hired into Google see it this way because for them it's a thing they seek out, a life goal they pursued. For me it was always in the back of my mind.
Which is one of the reasons I eventually had to leave despite it putting my family in a more precarious financial situation. It wasn't a place I was ever going to find meaning in my work or feel like I fit in.
Re: startups and compensation one major thing that has changed since 20 years ago is that it is simply not possible to become wealthy (like "I can afford going to retire wealthy" or "I can now afford to seed my own startup wealthy") out of the equity package that a startup will offer unless you're the cofounders or a founding employee. The packages they'll offer will be a fraction of a % of equity, which if you're really really lucky will turn into a few hundred thousand in an exit after busting your ass for a few years. From a strictly financial POV there's no reason to choose working at a startup over a FAANG. Most will simply not offer anything to make it worth the risk. You should only choose to do it for lifestyle or enjoyment reasons, really. (That said, I'm working for a startup again right now.)
[1] https://www.washingtonpost.com/business/economy/source-justi...
I'm not saying we're smarter or innately better, but coding is a tradeskill and the masters of the trade are almost all here.
I'm interested to see how much this changes over the next decade, though, since a lot of people have scattered across the globe thanks to remote work finally catching on.
The experience in scaling products that big and that fast is concentrated and not evenly distributed.
More precisely, the number of people who were there when "scaling products that big and that fast" applied to Facebook (or other comparable companies) is a tiny fraction of all SV developers.
But that's far away from the claim from above, that any 250k+ total comp developer from SV can outprogram good people from outside the US who don't make 250k+ total comp in basically any benchmark.
Most SV employees never have to solve these problems.
I picked frontend intentionally because it's probably the area that's the furthest away from these problems, but still compensated at 250k+ levels, if you work in SV for the right employers.
But even among backend engineers, most have never touched hard to parallelize problems. I'd argue that most (definitely not all) FAANG backend tasks are easily scalable. Implementing them still takes work, whch is done by most FAANG engineers. FAANG level tooling definitely helps, and while developing that tooling is hard, the people who have built such tooling are only a tiny subset of FAANG employees (regardless whether they are more skilled than the average FAANG employee or not).
For one, developing frontends at the scale and complexity needed for quickly growing organizations is just as complex as backend scaling in a variety of ways.
If you're a frontend expert and can operate in the way needed to support a rapidly growing business/product - you can make just as much if not more than backend engineers (500k+ easily at senior levels). This instinct around large-scale abstractions is not developed by building small sites over and over gain. This is just simply experience not everyone has because it requires context.
Also programming ability alone is not the sole reason for the outsized comp packages. In my view, it is at most 1/3rd of the reason. There are a lot of other qualities that are important that can only be gained through a specific kind of experience.
What do you mean by large scale abstractions?
Generally, small frontends are different from large frontends. Already because you need to work on a team instead of being able to do it alone. But adding features is a quite orthogonal concern to scaling by the number of users I'd say. There can be extremely complex frontends that only have a dozen concurrent users (think some company internal admin console), and nothing prevents a small frontend from targetting billions. E.g. Google search used to have a quite simple frontend 17 years ago, and it still looks quite simple to the outside, but the results side has obviously been heavily enriched since.
From the frontend point of view, it is just calling an API, and processing its results, while there is backend magic happening to make it scalable. I'm disregarding SSR here for a moment. It shouldn't matter if the same frontend code is loaded by ten users or hundreds, or billions.
> There are a lot of other qualities that are important that can only be gained through a specific kind of experience.
Can you list some of those qualities? I'm curious.
Some problem that would hit one customer every few years and go away on refresh for a local mom & pop website will irritate thousands of people a day on Google.com, and those people will band together on Facebook into a support group, and that support group will get media coverage
You try to look into the issue and you can't reproduce it at all, you just have to figure it out from tiny wisps of clues
Software is a people problem and what big tech looks for is top notch technical skills and the mindset/ability to solve them in a chaotic environment.
I'm fairly certain you're confusing anecdotes and culture with "big tech vs. your local shop". Software being a people problem meaning big tech is more people focused is a false dichotomy.
Most of the big tech companies' engineers are doing the same type of work as outside big tech. However, their internal practices are scattered and faster than outside (IME), which means you have to be able to keep your head above water, drive your own career, and deliver impact. You need to understand the product space quite well, as you are expected to be a partner. Rather than delivering code, you are delivering value.
Maybe 1 in 20 engineers that I've worked with outside big tech had this trait. I think they set the tech bar high, but the real skill differentiation I have seen is ownership.
That, perhaps, you have never had to come in and save any team at my company, or thousands of other companies, because we are competent developers?
Being able to do the work of 10 people once, does not mean that you are able to take 10 people from any company outside of a specific geographic area and replace them with one of you.
If paying 250k+ for a developer is insane, then so is a company able to pay 250k+ without running huge losses. Those developers still make peanuts compared to what the stakeholders and executives do, despite these salaries. It's ignoring the elephant in the room which is fat profits being with a few companies, subsequently making it tougher to compete for anyone not born in wealth or able to get VC money to play with.
SV could both bankrupt dozens of companies outside SV, slash that 250k locally and up to ante globally if it had a mentality shift, in less than a few years. That's how insane the profits are.
In the States, those things can exceed the salary cost of the engineer.
Should be outsourcing at the programme or project level, not hiring employees to add to US teams, then.
I think the reason this isn't more popular is that our talent pool is small enough that unless you already have a Canadian presence it's not really worth opening up a business unit/office/whatever here. Obviously some big players have done it (MS, Amazon, EA) but if you're a medium sized company it may not be worth it.
I work for a US company but I'm a contractor for this reason. It works for me but if you don't have extended health through your partner or don't want to deal with the tax implications of being "self employed", it can be less attractive.
So I'm also working as a contractor for a US (well, international remote but US incorporation I guess) company right now. And, yeah, no benefits isn't great. But I have also found that paying out of pocket for dental, etc. is better than buying a plan for myself.
Also a bit of a bummer that CRA has tightened up incorporation stuff a bit since the last time I did this. Given I have a single client only right now, I'll just be doing sole proprietorship.
The idea is you scale your rate to include enough for a benefits package.
Or they use a POE/EOR service, that's possible in Canada I think.
Either way, incorporation still has some advantages, one being that you time shift your pay as needed. It can make it easier to subcontract people too. And it's easy, and only costs a few hundred dollars.
My employer is using Deel, paying me as sole proprietor, and that's fine. Deel can also act as an EOR and I could be "full time", and that could offer benefits. But it feels like a bit of a veneer overtop over what feels like contracting anyways, and it would mean less $$ in my pocket. It doesn't make a lot of sense.
Especially now that pharmaceuticals in Ontario are free for kids whose parents don't have a plan.
Also, it seems that telling my dentist and physiotherapist that I don't have a plan seems to make them behave more responsibly, on the whole.
If I end up at some point landing more clients and I end up being more consultant than single-client contractor, then I will incorporate.
FWIW an EOR/POE is more about the convenience for the employer, not you - they don't have to establish a local business presence, register a bunch of tax accounts, etc. You can end up technically employed by the EOR with access to a decent benefits plan etc., and paid directly by payroll. At least that's how it works in some jurisdictions, as you note. It's different all over, but can make a huge difference in somewhere with crappy medical coverage, or other benefits you want to keep consistent.
When the most practical option is to form a corporate entity with a single employee, and BTW you may have to pay GST + PST on labor depending on if other Canadians use the service, it begins to look really unappetizing.
I get that employees as contractor relationships are rife for abuse, but, like so is exempting tech employees from overtime protection (BC and Ontario).
It really just shouldn't be this hard to do work for a company that isn't in Canada.
So it ends up being sole proprietorship only, which has far less tax advantages.
I mean, as a tax payer and citizen who paid exorbitant personal income taxes for years as an employee of a FAANG, I think this is probably equitable. But as a contractor, now... damn I'd like to be able to take advantage of corporate tax rates like I could 13 years ago when I did this last.
It's extremely easy to be self employed in Canada with a foreign corporation. You just start getting your money wired through Wise or whatever service and declare it on your income taxes. As you work for a foreign corporation, no need to charge sales tax. That's it, no separate entity or incorporation needed. You can still expense your tools expenses, part of your housing, etc.
Sure if you want to incorporate, it becomes more complicated. But you wouldn't be incorporated either if you had a salaried job for a tech company locally.
But "extremely easy" is relative. It's still more hassle than being a salaried employee. If you make a non-trivial amount of money you'll need to pay your income taxes in installments and set it aside yourself (no deductions after all). You need to pay 100% of your CPP contributions. You're likely getting paid in non Canadian currency so need to get set up with a good forex provider or get ripped off by your bank (I use xe, but there's a few out there).
Also your point about not needing to charge sales tax depends on what country your employer is in. In my case it's true I don't need to charge because we have a tax treaty with the USA; for other nations this may not be the case. If no tax treaty is present, you need a GST registration #, charge your employer said GST, hold it yourself and then pay it out just like your income tax.
Also the extended health can't be overstated. If you have dental issues, bad eyesight, are on prescription drugs etc. it can make a real difference.
Generally I'd recommend contracting to people but I just want to be realistic.. it has it's downsides.
Having a team completely unavailable during the USA working day is a real pain for collaboration and it's hard to sue people in other countries that "borrow" your source code to make their own version. Also, knowing exactly how to pay (say) a German developer and how taxes work and what the labor laws are is always an issue.
Aside: The very best contractors I ever hired were Ukranian. So many absolutely fantastic talents at around $30 an hour. Then Russia invaded (the first time) and annexed Sevastopol and the USA made it illegal to pay them any more.
only ones who stayed in Crimea
Shameless plug- a buddy runs a dev shop out there, they have great devs and exceptional designers (and also are doing a fund for Ukrainians impacted by the war). If you're looking for help in that region, they're great people to chat with: https://beetroot.co/
Of course, people realized that and there's 0 unemployment in IT, however, there's still a long way to go before we get to the Bay Area salaries - 6 figure salaries are only for the top 1% devs and only working for the USA and usually short term contracts, but a decent developer makes north of 50.000 dollars a year which is a pretty good salary here.
2. english language barriers - half the people I have worked with in South America (Colombia, Brazil) and China were so bad with English that it seriously affected the team's productivity.
Canadians fit the bill on both counts - same timezones and language.
Something you have to keep in mind is that there are two parallel markets over there: SV caliber developers and the rest. The former won't have any issue getting a job in the US (takes maybe a week for a talented engineer to get one). Therefore, comp has to be priced appropriately. The later can't -and likely won't ever be able to- secure a US visa, mostly due to skills (there's a reason they immigrated to Canada, it's way easier and the quotas are close to 10x per capita compared to the US). Some companies leverage this and have floors of international devs they park in Canada for a fraction of their US counterpart through a subsidiary.
The "SV caliber" developers you identify [honestly, that's kind of offensive, but whatever] who can and do actively consider relocation to the US.
The same caliber of developers, but have no interest in ever living in the US, for personal or obligation reasons.
The permanent resident non-Canadian citizen category you allude to.
Google Waterloo for example is mostly full of the second category. Plenty of high caliber talent who are there and not in Mountain View because that's where they chose to be. Because any of them could relocate / transfer to MTV any time they liked, but prefer not to. In my last few years there, there were dozens of people who I met who had transferred (back) into Canada from SV/Bay Area because they simply couldn't stand living there anymore because of cost of living, politics, family, home sickness, etc.
There are other companies that set up shop in Canada merely to siphon the first. They offer an "ok" experience for people who choose to stay, but push hard to get talent to relocate. I get the impression there's lots of this happening in Vancouver in particular.
The prominence of the last (recent immigrant / PR) category is really something that has held back the Canadian tech scene in general TBH. It's a product of international recruitment by the Canadian immigration system. There are shops full of people essentially fresh off the boat from the PRC or Eastern Europe. Many of these are highly qualified -- but in the context of our industry as a whole -- underpaid/mistreated. Others are not as competent. And it's all mixed up, and it has messed up the local tech market, compensation ranges, and quality of work produced.
Of course all of this is in flux and changing because of the rise of remote work.
There are a lot of very talented people that have values that don't align that way for a variety of extremely valid reasons.
As someone who has the right to work in both Canada and the US, and who has been attempted to recruit by SV companies, I prefer the benefits of living where I do in Canada over the lifestyle costs of living in SV with 3x the money.
In fact Google pays Canadian similar to their non-premiun US locations like Raleigh.
I'm surprised cross boarder hiring hasn't become even more popular with the pandemic and forced work-from-homeness. Canadians mostly speak English (even most Quebecois do!) and are in a compatible timezone with the majority of the developers either in St. Lawrence valley or Vancouver and thus being either EST or PST.
Also, while the holidays differ I think the total number of stats is pretty close to the same.
It is going to be an interesting decade, the rise of working from home and ai creeping into every profession will make for some radical changes. Engineering managers can only hold their accountants overlords back for so long. I do hope SFO stops being the centre of the tech world as it is very inefficient, but that's probably 30-40 years away at the earliest unless all out war is triggered.
Source: recent offers and friends who recently got offers
It's not "easy" to achieve this, but it might be possible. One doesn't just wake up and decide to go work at a FAANG and get a >90% percentile salary from them for the level.
e.g. Amazon was giving out SDE II (google L4 equivlant) @ 400k (first year no RSU but cash bonus paid out monthly) about 2 months ago.
re L3 level, if you have a master's or 1+ yr exp, while you're still at the entry level, you are considered "industrial hire" and 300 is very attainable
in my current company for new hires (again justed to google's leveling) we have L5 550+ and L6 650+ (200~300 cash + RSU , not pre-IPO bs)
In my earlier years I used to doubt those "ridiculous" numbers until I simply asked for more. There are plenty of companies out there, many medium-big sized non-FAANG companies pay just as well.
Source: Nearly everyone I came up with as a programmer is in the 250+ range, and not just at FAANG. Check out https://levels.fyi
In my last job search which wasn't too long ago, I got several offers and not a single one was below $400k.
I started my career in small companies and startups and it took me a long time to understand how TC works. Both in tech and finance base salaries are typically around 200 or less, but total compensations factoring in both bonuses and RSUs brings the amount of money you bring in each year much higher.
It's gotten much better in recent years with levels.fyi and the like helping more people to see this, but I found that a few years back if you didn't go to an elite school you likely didn't have someone explain to you how comp works.
If you want to make > 250k you absolutely can, and it doesn't mean selling your soul. Don't convince yourself that these TC levels are myths.
Though I do suspect we'll see dotcom level changes in comp soon.
While I think the premise is wrong - not anyone can learn to code - there are also levels of "learn to code" so it's simpler to spin up a create-react-app and modify it a bit for a code school credit compared to advanced Scala - it's also true that learning to code is nowhere near the hardest part of software engineering productivity.
So in reality we almost always end up with a non zero price where supply equals demand.
My go to example is oxygen. There is more than enough of it for everyone, so there is no possibility/need to charge money for it.
Beyond that, I have no doubt that there are some people who would willingly charge everyone for oxygen, if there was only a way. This would be driven by the idea that even though there is plenty in the atmosphere, a person only needs to be deprived of it for a little while before the situation becomes dire.
My appetite for food is not infinite, I can only eat so much. Same, but less so, for cars, clothes, etc. in some abstract sense, maybe, there are people that want 100 cars, but not really.
Even with housing, demand is not really infinite. I don’t want to redo my kitchen every year.
But software, every company has a backlog far longer than they will ever be able to do, and the ROI is there for the most part.
No one actually believes "anyone can code" when you're at faang level of talent.
Statistically that's likely, but it's not always the case.
If you luck into one of the companies that IPOs and grows big (and doesn't need to be FB/GOOG-level big, just in the handful of billions in market cap), even the tiny sliver of options of a regular employee can become retirement-level money.
There are many public companies in SV that have market cap in the dozens of billions who made their regular employees who joined pre-IPO very rich even though the companies aren't FAANG and don't have such brand recognition.
It's unlikely, yes. But it happens.
I guess it depends how you define a "founding employee". I know several engineers who made enough to retire who joined unicorns around the pre-series A time or at the time of the raise itself but were not founding engineers.
More recently, last time I was on the job market and interviewed with startups, their offers were more along the lines of "If the company value rises to $1 billion, your stock options will be worth enough to buy a 3 bedroom family home within 30 minutes of the office" - a decent chunk of change, no doubt, but not enough to retire on, and far from guaranteed.
[1] https://www.businessinsider.com/graffiti-artist-painted-face...
Even for most engineers, somewhere between five and ten million is enough to retire on (3.5% safe withdrawal rate) at a nice standard of living, again assuming you don't want to live in the bay area, NYC, Seattle or LA.
Especially if you already own a place you're happy to live in long term, money in that range is enough to live very comfortably anywhere in the US. No, you're not flying private jets and maybe not employing multiple full-time domestic staff, but $200K-400K per year without even touching principal is absolutely enough to have a nice standard of living even in Manhattan.
Join an early stage startup as a senior engineer and you'll probably be getting less than half or a third of a % of equity. If they even deign to tell you how many outstanding shares there are. Say that company gets super lucky and sells for a $1B? You might be lucky get $1M out of it. Assuming they didn't play funny business with the paperwork during acquisition, or dilute your shares a whole bunch.
$1M is a lot of money, but it's not "goodbye workforce" or "I'm off to invest in seeding my own startup" money, not when you have a family, home, and are still youngish.
Meanwhile the founders and other major stock owners will be doing quite well for themselves.
It's not unjust necessarily. But it definitely should inform just how much passion people put into the startups that hire them. Don't be fooled. Treat it like a job.
Pretty sure that a lot of places I simply didn't hear back from was because they didn't expect to be able to even come close on comp. And I've had other Xoogler friends who applied for places, got promising initial results, and then insanely low offers (like 1/3rd or less of Google comp).
I'll do 2/3rds or even 1/2 for the right job, but not 1/4 or 1/3rd.
Now, part of that is the domestic market (Ontario Canada) has been underpaying SWEs for years and getting away with it, since most of the good talent flees south after graduation anyways.
But part of it is that FAANG compensation is just far more than anybody else could ever afford.
I don't think this is true. I've had friends getting comparable comp offers at HBO and NatGeo, for example. And of course it's been known for some time that successful startups like Uber, Lyft, Airbnb, Dropbox, etc. were doing comparable offers.
Once a company decides to really compete digitally, and that they can leverage economies of scale properly there, it suddenly makes sense to compete for the same class of engineer that can work at Google, Meta, etc.
Microsoft, Apple, Google, Amazon: MAGA
Google, Apple, Microsoft, Meta, Amazon: GAMMA
"tldr options" @ https://equity.ltse.com/calculators/tldr-stock-options is an eye-opener to see just how meh the packages offered are.
Meaning, startup or no, I’m expecting to be paid what I ask. levels.fyi shows that my pay+bonus at my startup for my role matches FAANG compensation so I’m happy with it.
I have worked for places that offered equity a few times. It did turn into something once (Google bought my employer). But the actual equity conversion was minor in quantity compared to the rest of my Google TC.
There's no way out of "serfdom" other than starting your own company. I don't think this is healthy. The incentives are not there for engineers in startups to really put themselves out there.
Took me 10 years of grinding to finally realize this. Now at this stage the only question is how do I do it?
Now I'm always sitting on an enormous pile of savings and I have short- and long-term plans to move my career and life forward. It spares me a lot of stress.
Especially since you only need to keep your expenses 10% lower (or so) than your after-tax income to hit a reasonable target.
Turns out the numbers are a lot more favorable, though. You can count on some level of compounding investment growth even after inflation, you can slash your living expenses after you retire, etc. Suddenly early retirement is a possibility even at a pretty modest income level, without compromising your quality of life too much in the mean time
I work on infrastructure, and so a few years back, when I proposed a major project, I had to demonstrate how it would save *many* times the fully loaded cost of the engineers on the team, by reducing the Storage TCO for all of Google (for example). It was not enough for the project to "break even" --- the benefits had to do more than just exceed the "nominal" SWE cost. It had to be multiple times the cost of the SWE's, to account for the opportunity cost of those SWE's --- SWE's are a constrained resource, which is why a project needs to save $$$ (or increase profits) by many multiples the fully loaded SWE cost. (That project has since been completed, successfully, and I got a promotion to Sr Staff Engineer out of it.)
The reason why SWE's are a constrained resource is becaused finding good SWE's is non-trivial. As a TL, I don't want to waste my precious approved headcount on people who just want to rest and vest, or people who believe in the crazy talk of only needing to work 30 minutes each day. I'm trying to find highly motivated, smart, and talented SWE's who can also be team players. And if they need to have domain expertise (say, be proficient kernel engineers), it's super-duper difficult.
So I don't see any indication of people getting hired just to starve statups of talented engineers. We need every single talented engineer we can get for the projects that we want to accomplish. And in the time when we may need to slow down our growth, it may mean that we will need to slow, or shut down some projects. That may suck, especially if it's a project that we had invested a lot of passion into. But it's certainly no reason to panic. Slowing down growth is not the same as layoffs, and there is no shortage of work for us to do.
And yet these people exist and get hired, even at Google. Perhaps not on your team, but they’re definitely there.
>Alphabet, Google’s parent company, said its head count rose 21% in the second quarter of 2022 to 174,014 full-time employees from 144,056 the year prior. https://www.hcamag.com/us/specialization/corporate-wellness/...
SV talks the big talk about hiring the best of the best of the best, then hires 30 thousand people in three months. There probably aren't that many 10x programmers in the world.
A 21% rise results in 17% of the new group having come from the increase.
[1] https://blog.google/products/google-cloud/dynamic-hybrid-smr...
[2] https://www.t10.org/pipermail/t10/2018-September/018566.html
On the production kernel team, colleagues of mine worked on some really cool and new shit: ghOSt, which delegates scheduling decisions to userspace in a highly efficient manner[3]. It was published in SOSP 2021/SIGOPS [4][5], so peer reviewers thought it was a pretty big deal. I wasn't involved in it, but I'm in awe this cool new work that my peers in the prodkernel team created, all of which was not only described in detail in peer-reviewed papers, but also published as Open Source.
[3] https://research.google/pubs/pub50833/
[4] https://www.youtube.com/watch?v=j4ABe4dsbIY
[5] https://dl.acm.org/doi/10.1145/3477132.3483542
We have some really top-notch engineers in our production kernel team, and I'm very proud to be part of an organization has this kind of talent.
For example:
RePD is at just wrong level at all. It should have been at CFS/chunk level and thus benefit other teams as well.
BigStore stack is beyond bizarre. For years there were no object-level SLOs (not sure if there are now), which meant that sometimes your object disappeared and BigStore SREs were "la-la-la, we are fully within SLO for your project". Or you would delete something and your quota would not get back, and they would "or, Flume job got stuck in this cell, for a week...".
Not a single cloud (or internal, for that matter) customer asked for a "block device", they all want just to store files. Which means that cloud posix/nfs/smb should have been worked on from the day 1 (of cloud), we all know how it went.
As far as "proper stack refactoring" is concerned, again, the key is to make a business case for why that work is necessary. Tech debt can be a good reason, but doing massive refactoring just because it _could_ help other teams requires much more justification than "it could be beneficial". Google has plenty of storage solutions which work across multiple datacenters / GCE zones, including Google Cloud Storage, Cloud Spanner and Cloud Bigtable. These solutions or their equivalent were available and used internally by teams long befoe they were available as public offerings for Cloud customers. So "we could have done it a different way because it mgiht benefit other teams" is an extraordinary claim which requires extraordinary evidence. Speaking as someone who has worked in storage infrastructure for over a decade, I don't see the calcification you refer to, and there are good reasons why things are done the way that are which go far beyond the current org chart. There have been a huge amount of innovative work done in the storage infrastructure teams.
I will say that the posix/nfs/smb way of doing things is not necessarily the best way to provide lowest possible storage TCO. It may be the most convenient way if you need to lift and shift enterprise workloads into the cloud, sure. But if you are writing software from scratch, or if you are internal Google product team which is using internal storage solutions such as Colossus, BigTable, Spanner, etc., it is much cheaper, especially if you are writing software that must be highly scalable, to use these technologies as opposed to posix/nfs/smb. All cloud providers, Google Cloud included, will provide multiple storage solutions to meet the customer where they are at. But would I recommend that a greenfield application start by relying on NFS or SMB today? Hell, no! There are much better 21st century technologies that are available today. Why start a new project by tying yourself to such legacy systems with all of their attendant limitations and costs?
Trust me, I intimately know what I’m talking about.
Without personal jabs, let me explain in a bit more detail:
App in VM (kinda posix) -> ext4 (repackaging of data to fit into “blocks”) -> NVMe driver -> (Google’s virtualization/block device stack, aka Vanadium/PD) -> CFS. The moment data got into ext4, it goes through legacy stack that only exists because many years ago there were hardware devices that had 512 byte sectors (as illustration, upgrade to 4K took forever). All repackaging, IO scheduling to work with 4kb block abstraction is wasted performance and cycles.
From customer perspective, all they want is VM with scalable file system. With Kubernetes, etc. they don’t want to ever think about volume size, which is major hurdle to size correctly and provision. BTW, both small and large customers run into volume sizing issues all the time.
There are also internal customers that need posix-compliant storage “on borg” because they run oss lib/software.
Anyway, optimal stack in this case is to plug in into VM on a file system level. Now, is it hard problem to solve? Yes. Would it eliminate PD? No, still required for legacy cases. Would it be enormously beneficial for modern conteinerized cloud workloads? Absolutely.
If you have an app which needs a NoSQL interface, then you can do much better by using a cloud-native NoSQL service, as opposed to using Cassandra on your VM and then hoping you can get cross-zone reliability by using something like a Regional Persistent Disk. And sure, you could use Cassandra on top of cifs/smbfs or nfs, but the results will be disappointing. These are 20th century tools, and it shows.
If customers want Posix because they don't want to update their application to use Spanner, or Big Table, or GCS, they certainly have every right to make that choice. But they will get worse price/performance/reliability as a result. You keep talking about ossification and people refusing to refactor the storage stack. Well, I'd like to submit to you that being wedded to a "posix file system" as the one true storage interface is another form of ossification. Storage stacks that feature NoSQL, relational database, and object storage WITHOUT an underlying Posix file systems might be a much more radical, and ultimately, the "proper stack refactoring". A "modern containerized cloud workload" is better off using Cloud Spanner, Cloud BigTable, or Cloud Storage, depending on the application and use case. Why stick with a 1970's posix file system with all of its limitations? (And I say this as an ext4 maintainer who knows about all of the warts and limitations of the Posix file interface.)
Of course, for customers who insist on a Posix file system, they can use GCE PD or Amazon EBS for local file systems, or they can use GCE Cloud Filestore or Amazon EFS if they want an NFS solution. But it will not be as cost effective, or performant as other cloud native alternatives.
Finally, just because you are using "oss lib/software" does not mean that you need "Posix-complaint storage". Especially inside Google, while those internal customers do exist, they are a super-tiny minority. Most internal teams use a much smarter approach, even if that means that an adaption layer is needed between some particular piece of OSS software and a more modern, scalable storage infrastructure. (And for many OSS libraries, they don't need a Posix-complaint interface at all!)
Posix-complaint means sticking with an interface invented 50 years ago, with technological assumptions which may not be true today. Sometimes you might need to fall back to Posix for legacy software --- but we're talking about "modern containerized cloud workloads", remember?
How is the world does one become a proficient kernel engineer? Without already being at a FANNG.
https://lwn.net/SubscriberLink/902854/b788a6a3d77aba7a/
If you scroll down to the Most active employers in 5.19 by commits you'll see:
1. Intel 10.9% 2. (Unknown) 7.5% 3. Linaro 5.7% 4. AMD 5.5% 5. Red Hat 5.2% 6. (None) 4.3% 7. Google 4.1% 8. Meta 3.5% 9. SUSE 3.1% 10. Huawei 2.9%
The statistics are slightly different if you count by lines of codes changed, but either way, it's not all FANNG companies, not by a long shot. There are plenty of people who get started coding via kernelnewbies.org and other resources.
Years ago my town had a bunch of lumber yards where you can get market rate lumber. Home Depot moved in and provided way lower than market rate lumber until all of the independent lumber yards went out of business in the area.
Then they raised the price above market rate and that's that. Everyone hates them for that but we all shop there.
Power is meaningless unless you're hurting somebody: otherwise, what even is the point? Money is abstracted power. For this reason, ANYWHERE you are seeing accumulations of vast amounts of money, somebody or something is being hurt.
In this case it appears to be that the whole 'make 500k as a dev' thing, which has always seemed weird to me, turns out to be 'FAANG is trying to control all access to devs worth having, after which they can starve the market and do whatever they want'. It comes down to power, always.
You'll note that the open source devs upon which their internet relies, upon which so much rests, are NOT making 500k as devs. If you're doing something worth doing and it's not harming but helping, you are not getting paid. If you're getting paid in a big way, you're doing something that enables harming somebody, whether it's people or a more abstracted thing like a market. The wealthy FAANG folks are part of a situation where an employment market is being harmed, and that's why the money is there for them.
It's a weapon. Otherwise it would not be there.
I think that's really the bottom line here. They aren't overpaying to starve out the industry, although I'm sure they're perfectly happy to benefit from that indirectly
You could count on one hand the number of startups that are serious threats to either. And when one arises, they can pretty much just buy the company, as Facebook has done repeatedly. They hardly need to be concerned with individual engineers, and even if they were, the idea that they could hire them all is pretty far-fetched
Its the inefficiencies of the first price auction.
An alternate explanation is that managers in tech crave headcount, they want to feel important and to have their projects and teams very well staffed. During the past year they may have gotten excessive with hiring and now you see the end results which are hiring freezes, rescinded offers and small layoffs across the industry.
Are you guys all bitter business owners?
https://en.wikipedia.org/wiki/High-Tech_Employee_Antitrust_L...
Remember that Google and Apple were caught trying to control the employment marketplace. Keep 'em happy, but not too happy!
Jokes on them, though, my mortgage is almost paid off, so I can work for less than I was. I'm the old guy working with all the young people at the startup. I can almost feel their eyes rolling into the back of their heads when I start going on about "back in the 90s"...
My reading is that founders and VCs got greedier (or they were always this greedy, but the reality distortion field has since failed). Back in the day, working at a startup appeared to offer a plausible path to life-altering wealth to employees with stock options - these days, the cat is out of the bag: dilution by endless funding rounds, no route to IPO, preferred stock fuckery, and a countless other ways of being screwed over by information and/or power asymmetry.
The organization is a reflection of its leadership and their vision. Or lack there off. MS and Google are almost polar opposites. Google is MS under Balmer. Just failing, tone deaf, never quite doing anything right. Enter Satya Nadella and suddenly MS is re-engaging with developers, doing all the things that were simply unmentionable before, and generally delivering great share holder value. Google needs somebody like that. The current CEO is not it. He's a caretaker. Not a leader.
I'm not a shareholder; so no stake in this game other than observing what is obvious to me.
I was never essential or important there. So my opinion is just.. whatever.
And I never really drank the kool-aid, either, really. But...
The kool-aid tasted way way way better when it was Larry, Sergey, Eric, Patrick, etc.
I can see what Ruth Porat brought to the table. Sky high stock values is one. That was nice when I still had RSUs.
But Sundar? I don't get it.
More corporate shooting-self-in-the-foot means more space for a vibrant built-in-a-garage startup ecosystem to compete - the kind the auto industry hasnt had since, like, 1955.
edit: Increased clarity.
But maybe you mean a hypothetical "you" of other people still working there. Obviously it's a mix, but I suspect on the whole most people are smart enough to see that these kinds of things are unlikely to accomplish much.
But they're also likely happy that upper management is finally acknowledging these dysfunctions.