Is a trillion dollars’ worth of programming lying on the ground?
blog.cerebralab.com
blog.cerebralab.com
If your goal is to implement some code for, say, healthcare, it is a lot easier to explain the business process to someone who is already familiar with how stuff is done on a higher level. Whenever you hire someone from a different culture / country, you need to start from the basics by explaining the very core of your society.
An American programmer might not be too concerned if a body temperature measurement returns 45 degrees Celcius. Similarly, a patient height of 95in probably wouldn't immediately raise alarm bells for a European developer.
Your offshore developer team might be perfectly capable of writing the code once they understand what to write, but getting them to understand it is a hell of a lot more difficult.
Add to that the possibility that English is the second language for both sides, and things get messy pretty quickly. Not to mention a time zone difference making communication even harder.
Silicon Valley and Scandinavian cultures are very flat, so it is possible for "the man on the floor" to tell management that it doesn't work to put diesel into a gasoline car.
In authoritarian cultures this would be a mistake as it's seen as embarrassing the management by saying they made a mistake. If you want to continue being employed then it is much better to pretend there is no problem.
Catching problems early saves a lot of money in software development, so a non-authoritarian culture can save a lot of time and thus have a lot higher wage.
The US collective mentality is sometime overly optimistic specially in public. This is sometimes a positive since it removes almost any inhibition regarding challenges, but it can lead to things blowing-up spectacularly, and then having to put in place hacks and brute force to make it kind of work relatively inefficiently and inelegantly.
But it also leads to great successes (biggest economy in the world, lots of major companies, very attractive country for talents) and innovations never seen before.
This positivism is prevalent in public statement but in private cracks start to appear. It gives me the impression that this positivism is pretty much a cultural obligation, so while not authoritarian, the group mentality is geared toward not talking to much about possible difficulties out of fear of being seen as too negative by the group.
It is an infuriating challenge to deal with, especially when you're faced withacompletely passivebusiness interface only committed to answering exactly the question you ask, not taking the question and doing the bit of contextual fill-in and translation one would expect. As a result, to cope with that, you have to ask heavily qualified, context bearing questions to Get to the bottom of anything; which after a certain point becomes unintelligible if you don't have someone actively willing to follow along, which manydon'tsee the value of, because they don't see it as doing anything of value.
It's a vicious cycle really.
I had a real heart-to-heart with the CTO where I worked since he only gets along with one of the managers and doesn't like his directors. He said the only reason he approved my promotion was because he was tired of hearing Yes Men telling him things were fine right until they blew up in his face. He's a tech guy, not a business guy so probably outside the norm.
People from cultures where sarcasm isn't common don't understand this nuance and seem to find this kind of dialogue abrasive and unconstructive. Perhaps the sentiment is that if you can't articulate it clearly, then the issue is non-existent, for better or worse, or you're just being lazy and unhelpful.
Last weekend Elon Musk put a couple of people in space at a fraction of the costs NASA originally spent.
I think and hope many industries can still be disrupted in a similar way.
You seem to be comparing the cost of doing something the first time with doing it 40 years of technology later
Perhaps it’s wording and I’m sure private companies are more cost effective compared to government
Most people are very hesitant about giving honest feedback even in US businesses because of these pressuring forces. If your manager wants to put diesel in a gasoline car, you often have to agree, then have a sidebar conversation directly with that person so their negligence, ignorant mistake, or intentional malice isn't put on a large group or public display.
Very few US workers are in positions to give blunt feedback. This was pretty common in software development due to leverage developers had but it's, IMHO, been on decline.
That should be obvious. Most people I've worked for can be challenged in private if you've actually got a point - even if you're shot down in the end. The ones who are still assholes when handled correctly are usually not worth working for. The best ones can handle a public challenge on the merits of course.
That's exactly what different cultures look like: it's absolutely not obvious in Eastern Europe where I live. We call each other out on our BS freely and directly, and almost nobody ever gets offended.
It is quite the cultural shock for many of us when we suddenly have to sugar-coat and positive-tone-first every work criticism we express. It can feel like a huge waste of time and energy while if somebody tells me "Dimi, your code in module X sucks!" and I'll just laugh and say "dude, I had only a day to finish that, sorry -- if you don't like it, talk to the boss to give me one more week on it and it's going to be swell".
So no, such things as the one you quoted aren't obvious at all for many. I'd even argue they are only prevalent in the USA but I've seen evidence in both directions so I can't claim anything absolute there.
One kind of employer only retains crappy employees.
The other gets all the good ones.
Health care industry coding seems to along the lines. Solve the immediate issue. A few years later the complexity is off the chart.
You ain't seen nothin' then.
Try challenging an elder boss in Japan (or China or South Korea, but Japan is especially bad) and watch what happens, for example.
I find that very few cultures allow challenging authority like the US does.
If you are using this as a metaphor for a decision you personally consider stupid, then maybe.
If it's actually putting diesel in a gas car, then no, I can't imagine any case in which an intern couldn't safely tell that to their director or CEO.
When something is obviously not going to work akin to diesel in a gas car, I have never met someone who wasn't willing to at least take a second look at their "solution".
Another aspect of culture: how one-sided the
communication is.
We had a lot of fulltime employees working in other countries. The company paid pretty well, and we were essentially paying US salaries to folks living in countries where that would be considered very, very good money.
This, combined with the cultural issues you mentioned, resulted in management mistakenly thinking it was doing a good job -- privately, the foreign engineers shared my concerns, but they were not willing to "rock the boat" and bring those concerns to leadership.
Years ago (not my current employer), there was a big offshore team that was a huge pain in the neck because their entire development culture was one of deception and defensiveness - they'd deliver garbage but then spend enormous amounts of energy managing up and propagandizing to their reporting chain and executives that everything was finished, high-quality, etc.
One of the security teams practically begged for someone to help get this team to fix the bugs they were filing. The other team was straight up closing pretty much anything they could as "incorrect" reports or classifying all the bugs as ultra-low severity (which is basically "will not fix") so that their metrics stayed perfect. Lots of issues like this. When they were challenged on it from the outside, by one of the guys who didn't have a sense for the political situation, the best way to describe the reaction goes way beyond "circling the wagons" - they went full nuclear, politically, on that guy. No holds barred, a total nest of vipers. For engineers who have never experienced companies with deep and vicious politics, it's hard for them to understand what it's like at companies like that.
Then one day, one of the best executives at the top of the chain decided that he was going to re-organize his directs and fold these teams together, under someone from the side of the house into which the security team reported.
Now, cultures of managing up are good at reading the waters and so you'd expect that they'd change their tune anyway, and they did, but the part that was shocking to all involved is, as a side effect of the exec shuffle, the security guy above was given an important-sounding but meaningless title, and suddenly the interactions between that organization and the security guy went from a one of subversion, political smearing, etc. to one of absolute and total deference overnight.
The offshore team wasn't any _better_, they were still marginal and you couldn't trust a single thing they said, but the instant attitude change from "defiant, we-are-power-adjacent, you'd better watch yourself" to "oh XXX, we're helpless, nothing can be done, yes we will try but.." due to what was essentially the change of a string in a database somewhere was just crazy to observe. Everything went from "that guy is terrible, that guy doesn't know, blah blah' to "yes, oh, XXX is great, we love XXX, he is helping, ..."
Scandinavians communicate more directly. At the moment of giving feedback, Americans lack initiative, communicate vaguely and avoid confrontations.
First a little background:
The military is a highly confrontational culture, which isn’t necessarily a bad thing. In that kind of culture you tell people the disagreement immediately without tripping over pleasantries. It also means people are not easily offended until you try to waste their time or bullshit them with cleverness.
In the military people have to certify to do anything (super bureaucratic) so you generally know who your actual experts are without guessing and without a bunch of lying Dunning-Kruger insanity. It also means people are generally better about knowing their limitations and staying in their lane.
That being said, in the military I have no trouble or reservations about telling a superior they are wrong and why when it comes to an area I am well qualified. Knowing that the superior can immediately reconsider before making the authoritarian decisions of an iron fisted dictator. Superiors are generally better about considering the advise of their experts because the decision maker owns all risks and consequences.
In the corporate world you have to employ tact and really be careful how you deliver the message if the topic is not something already extremely objective. There is a lot of noise in the corporate world and far less interpersonal structure leaving everything open to interpretation. I find when I am at the day job communicating anything comprising more than two steps I go far beyond normal to be precise in my wording just to be clear.
There also exists a common phrase being thrown under the bus or more commonly known as a scapegoat. This does exist in the military but there are many conventions in place to catch those kinds of failures. Once that is identified it is a major reputation destroyer, which results in a huge occupational risk. It seems to more commonly apply in the corporate world when people are pushed into a corner and risks associated with that behavior are minor.
In most of my corporate employers there generally isn’t a central authority that owns any significant business risk or job terminating consequence when things go sideways (except maybe some senior management), which is certainly different from an authoritarian culture. In the corporate world there is plenty of room to blame external factors when things fall apart.
Even though it still does not literally happen anymore, there is still the tradition of the captain going down with the ship. The captain's authority was unchallenged on board the ship. But, if the ship sank, the captain paid for it with his life. This would inherently cut a lot of the bullshit. If you are the captain, your life depends on your people telling you the facts, not just what you want to hear.
Only in the extreme short term. In the medium term, the dictator inevitably ends up severed from reality by the inability of (true) data to flow back up to the authority, which results in their decisions becoming increasingly suboptimal over time.
This is the pragmatic argument in favor of free speech, really. If for the sake of argument we ignore moral and philosophical issues, the reason why free speech is important is that in its absence information does not properly flow back up to the authorities. Free information flow does not guarantee proper functioning of the system, but impeded information flow guarantees improper functioning. If your authority dictates the use of 4-inch long bolts in a place where there's only 2 inches of clearance, and nobody in the organization has the ability to feed back to the authority that information, the only possible outcomes are bad ones. This all compounds, too; the next dictate from the authority won't be able to take into account the fact that in order to fit the 4-inch bolt in, a new hole was cut in some vital structure that shouldn't be there, and some new bad decision will be made to add load to this structure, which will cause other effects. The authority ends up living in a fantasy land.
Note how this toy example seated entirely in engineering-space doesn't have any moral or philosophical component, and yet it is already sufficient to cause a bad outcome even in pure engineering terms. Moral and philosophical issues can cause yet more problems, but they'll be arriving to a space already broken.
It also doesn't matter for my purposes why the information can't flow back up to the authority. Even a well-meaning, "enlightened" dictator can end up in this situation by sheer inability to process the incoming information. The dictator may be the nicest, most well-meaning person in human history full of compassion and love and every other virtue you care to name, and they'll still break things if they insist on authoritarianism.
These are extremely hard to find, and anyone who thinks of themselves as one is missing at least one of those properties.
"Doing the work," "managing the people doing the work," and "making the decisions about what work should be done" are not simply three ascending levels of expertise in one domain. They are three completely different domains. All good so far; the Peter principle takes this into account. Someone who's good at "doing" gets promoted to "managing", and doesn't have that skillset, and gets stuck there.
But it's also true in reverse. Just because a particular person is not good (or, more generally, less good than the best) at "doing", that doesn't mean they'd be bad at "managing". And just because someone is not good at "managing" doesn't mean they'd be bad at "deciding".
So you've got people who are good at working being promoted to manager, people who are good at working and managing being promoted to decider (which, I admit, is a bit more likely to work out than the others)—or, in many cases, people who were simply hired as managers being promoted to deciders—and then people who are good at managing or deciding, but only OK at working stuck at the worker level forever, unable to use their skills to help make things better.
However, too much focus relative to the complexity of the task at hand is never good, because focus requires ignoring some things. I would theorize that the more complex the task a team is working on, the less authoritarian it can afford its culture to be.
In an Eastern-European "authoritarian" (Bulgarian) setting, you are given tasks with not much room for interpretation and are expected to deliver a tangible "no-bullshit" result within a reasonable time frame. Not much time is spent on meetings, you don't want to ask too many questions, praise is won by delivering a technically-efficient solution which doesn't question the business intents and preconceptions of management.
In a North-Western European setting, everything is questionable and debatable, sometimes at irritating levels. Fairly incompetent people are raising their voice on insignificant matters and the decision-making debate changes direction quite sporadically. By everyone focusing too much on the business side, what you get is a technically-mediocre product which solves many problems to many people, but doesn't have much value as an integral part.
Disagree. A team can be coached to commit to a common goal.
A dysfunctional authoritarian culture is probably faster than a dysfunctional democracy, for sure. But then you are comparing which broken system is less broken, not what would be the optimal setup.
Take a sports team. They are not micromanaged. They are given a common task, a common strategy, guidance, but in the field they are alone.
They are more and more micromanaged.
* Training has become the land of measuring and controlling each detail, and has expanded into controlling your whole life as much as possible.
* Before the game, with more and more accurate and often static tactics. Don't you act another way or it means disobeying and the bench is near.
* During the game. Hardly 15 seconds go by without the coach (or the coaches) yelling orders at 1 or 2 players. Now there are even real-time data collecting and analyses, sometimes from sensors.
Perhaps the worst example of micro-management is cycling, where I am pretty sure that the only directeur sportif's dream not yet fulfilled is to be able to radio-command electrodes implanted in riders' muscles. They already get the position of almost everybody in the pack in almost real-time, they get radio-communicated real-time speed, power, heartbeat sensors measurements and two-way audio communication with each rider of the team. For now they have to do with giving imperative orders through each rider's earpiece (so the reaction can be delayed or the order not quite obeyed), otherwise managers would be computer game players. And silly (sometimes collective) punishments for those who dared disobey when they felt the orders were stupid are not uncommon: even though it means losing chances on that day or the next, the manager has one priority: showing who the boss is.
Only for relatively simple situations with a relatively known process for solving - which is the opposite of most software dev, although that seems to be changing in some ways.
Once you really understand that, you'll understand why it's difficult for those in other cultures, using other primary languages, and sometimes even remote employees to develop effective software without difficulty.
Much of software development assumes that specifications are perfectly understood, translated, and captured. This is rarely if ever the case. Developers are very frequently inferring and making microdecisions within the context of what they're developing using their personal knowledge. When they don't do this, you often get products that just don't solve the target problem.
One factor businesses really shot themselves in the foot when they butchered agile into a micromanagement time pressuring system to squeeze work out is that it made localized workers even more of a necessity because it has to be done (and interpreted) fairly quickly. This, IMHO, puts remote workers, especially those in other cultures, at a huge disadvantage as well as the systems/tools they're developing.
I don't doubt for a second talent can be distributed world wide but I do doubt that there's enough talent that can understand the context of the systems they're building to make project managers, businesses, clients, etc. happy.
Whenever you hire someone from a different culture / country, you
need to start from the basics by explaining the very core of your society.
An American programmer might not be too concerned if a body temperature
measurement returns 45 degrees Celcius. Similarly, a patient height of
95in probably wouldn't immediately raise alarm bells for a European
developer.
In addition to what you wrote, I think it may actually go beyond that?
There's a certain "team unity" that binds good teams together. It's that extra "something" that helps you get through those tough moments and really work together.
I've worked on remote, international teams that HAVE had that kind of esprit de corps. And I've worked on local teams inside the same building that haven't.
But, overall?
In my experience that kind of unity and teamwork is MUCH harder to build across cultures and/or remote work arrangements. And the difficulty is often multiplied by the "mercenary" nature of consulting work in which consultants obviously aren't really incentivized to worry about that sort of thing - in fact you kind of need to form emotional armor and avoid it to an extent.
It can be done, and is absolutely worth doing. (And, in my opinion, it's a form o institutional racism not to...)
In my experience it's often as simple as giving employees some space to banter and chitchat and get to know each other. Can be as simple as a #random or a #chitchat IRC/Slack channel sometimes and/or a company culture that allows for some chitchat and getting-to-know-eachother during daily standups. And if you are working cross-culture, at an absolute bare minimum, do some homework to at least understand each others' holidays and intangibles like differences in etiquette and so forth.
Thing is, in typical offshoring arrangements, much of this is impossible.
I am quite the outgoing and easily sociable guy but I'll agree this is impossible and will tell you why it's impossible for me:
What you propose happens on the company time. Remote contractors always have this anxiety about "am I spending my time being focused enough on the work I am paid for?"
Shortly before I was laid off from my last job I realized that I constantly postponed getting to know the team better because of that.
It's hard to let yourself be looser if don't have the status of an employee (vs. being a contractor) and/or if you aren't shown you are a valued member of the team.
But I am absolutely not interested in their political opinions, what they had for lunch or what their girlfriend told them last weekend.
In my corner of academia, programming is almost always seen as grunt work and delegates to undergrads and junior grad students. Of course students aren’t trusted to actually manage the project so usually mid-level scientists or post-grads are assigned to oversee and manage them. The managers usually have no idea how to program or what it entails, nor do they have any interest in the project. At most they’ll want to use the software after its finished. If agile/waterfall programming annoys anyone, just be thankful you have a methodology for software development at all. Since literally everyone is senior to the programmer, but nobody else knows how to program, everyone just justifies their time by suggesting new features. Since no one knows how long anything should take, all new features are accepted into spec at anytime. Eventually, the project starts to run long, and since nobody knows why, management responds by adding more layers of management. Finally, the programmer either quits or graduates and the grant for the project runs out. The final product usually either has no resemblance to the proposal or only works on sample data. If it’s a website, no one bothers maintaining the server. If it’s a package, all dependencies go out of date within a year of release. The PI, or a PI down the hall gets a new grant to do basically the same project again, and the cycle repeats itself.
The best markets for selling software are the wealthy countries with their cultures. You need people in those countries that are part of those cultures to develop that software. So your employment market is limited by people within that culture that have that skill, which is why there is still a software development skills shortage.
For purely technical things like kernel, database, network or driver development the job market is much smaller but it's also much easier to move those jobs to lower income countries, which is very common.
But most software development jobs are at the application level where having a matching language and culture with your users is essential.
Something as trivial as a signup form that asks for firstname and lastname and then refers to the user by their firstname later in the application is something you wouldn't even need to spec out for an English speaking programmer. They would just implement it like that by default because that's the culture.
But this rarely happens, if at all. So there's something that's still amiss.
It's also starting to happen with banking / finance apps. Nubank is growing very fast in Latin America, Africa is starting to have a domestic ecommerce sector, and so on. The internet is new and many countries, particular developing ones that have a sufficiently distinct culture to not just adopt Anglophone products are only now starting to have the talent and infrastructure in place.
It's also happening in messaging. Telegram is overwhelmingly more popular in Europe/Russia than it is in the US. LINE is the most popular app in Japan and Thailand, Facebook Messenger and SMS still dominate in the US.
Other countries tend to use communication applications and platforms developed at or near their country (WhatsApp seems to be one of the bigger international exceptions from my experience).
There are a few exceptions to this where massive US companies have the scale to setup native teams in markets that understand culture well. Outside of that, it tends to fall apart. Some examples come to mind: Taobao, QQ, Baidu, Yahoo Japan, Weibo, VK, Aliexpress, Line, TikTok, etc.
(Note not all of these examples may be good because I'm not intimately familiar with all of them--since I too tend to use systems developed in my country).
Facebook, Google... they seem to span cultures probably due to providing services people fundamentally want to have across cultures. Large businesses typically have the capital to invest in addressing the cultural barriers and developing solutions.
Apple has a massive share world wide but when you look at Samsung in South Korea or Huawei in China, you'll see more adoption. Some of it may be nationalistic preference but I argue much of it is due to developing products/services that properly address demands in the market (which is tightly tied to culture).
And the best US companies have developed an internal culture that might not be easy to develop.
I'm baffled by this statement... I worked in the medical device industry for years at a company of 10k people and you would be hard pressed to find a sw engineer that didn't default to Celsius. Almost all sw engineers there either had EE, CpE or Biomedical Engineering degrees; Celsius is not remotely foreign to these types of engineers. I would be incredibly surprised if it was different at any medical device company.
The average programmer at Instagram in California probably doesn't have much more in common with the average Instagram influencer or feed-scroller than the average programmer in India (both programmers tend to spend most of their time thinking about computing and abstract math/CS concepts, while the influencer spends it thinking about marketing, social relationships, travelling, etc. and the feed-scroller is all about instant gratification and pop culture).
There are a number of technology-specific hubs around the world and even just in the united states. Silicon Valley is one of the most obvious ones, but Raleigh is another one, as are Nashua, Boston and Salt Lake City (there are others).
What all of these have in common is that each of them had, at some time in the past there was a massive company with out-sized returns. In the Valley were were _several_ - Intel, HP, ... and many more. NCR (and others) in Raleigh, Westford, Petaluma (optics), Boston as a geography absolutely dominated storage until Netapp, and so on.
When you have companies like this, they produce a lasting geographic concentration of talent, especially early in a given sector's life. Think Michigan and especially Detroit for automakers or California for aerospace.
The experiment we're doing with the work-from-home thing will be interesting if it goes forward, because as a few locations in Canada demonstrated, a very active de-centralization of a given kind of expertise (in that case, the network element management software) can render a given tech hub a memory at best. If history is anything to go by, disassembling Silicon Valley, if possible, will not result in more uniform distribution, it will just move from one concentrated area to another.
I agree that domain knowledge is invaluable. But in the examples given, the difference in domain knowledge between local talent vs remote may not justify the 5x salary.
Perhaps not all the nuances, and many aspects are arbitrary choices and only the spec would reveal what the business unit chose, but there's a lot of domain knowledge required to be productive.
If we need to outsource stuff, that has to be to some outfit that has developers experienced with banking nuances, and often times they are more experienced than internal devs (because they have worked on this parictular tricky corner of banking for multiple different institutions), and if we hire devs from outside the industry, we expect the first 6 months to need a lot of handholding because they don't know the nuances of banking and they can't be productive at all until they learn at least most of them, because no realistic spec will be sufficient for a programmer without a domain knowledge, at least not in the financial domain.
It really bothers me because I generally get put on projects with little acceptance criteria and not something I've done before, yet I deliver on time and the end user is very happy because I take the time to ask good questions up-front. Why can't someone deliver something that makes me that happy.
On the topic of communications. I've consulted on projects where I've said that something I noticed should be done a specific way, even though it was not part of AC, technically worked correctly, and making the change would be feature creep. My advice was ignored. Weeks later the end users kept complaining that the software was difficult to use and could not communicate what they didn't like. Since they refused to use the software and they refused to accept it as it was, I was brought back in. I sat down, found out what work the end user was doing, and did some of that work. I literally did someone else's job. I used the software. I made a huge list of areas for improvement. The PM agreed to everything. All of the changes were made, the end users were now saying it was awesome to use.
To summarize, learn to read between the lines of the acceptance criteria. Would you want to use the software you're making? For me, small issues annoy the heck out of me. I notice these things. End users tend to just say nebulous things like "it doesn't work well".
Worked in healthcare IT in the USA. Beans counters were utterly determined to send our work overseas. Old story of training our replacements.
We could teach them programming, testing, our stack.
But we couldn't teach them our healthcare system. Stupid small stuff that challenged us were completely opaque to our colleagues. Stuff like definitions of "visit" vs "episodes of care" and how each customer had different rules. (Made worse by sharing data between orgs.)
(My billion dollar idea was to help our counterparts adapt our stack for their local markets. India, Malaysia, and Vietnam have healthcare too, right? Get in on the ground floor of those emerging markets. Alas.)
Well, that's task-dependent. For healthcare billing software, certainly.
But if you're hiring someone to implement the SVG spec in a web browser? Bézier curves are the same no matter where you are in the world.
Indeed, if knowledge of culture was the key wouldn't comparative advantage lead to low-cost-of-living countries' developers dominating technically pure, well-specified work like web browser design?
The one thing about culture that matters is the work ethic, which indeed is hard to reproduce in many countries, but other facets of culture aren't really significant.
except the vast majority of FAANgs work force are immigrants.
- with higher "programming literacy" in a country they might have more top class programmers, top class programmers where able to optimize their career early by moving to western countries.
I would be questioning professional qualities of these programmers at this point. That’s just a common sense and base knowledge about the world you live in and people you write code for.
https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...
You give the AI instruction and explanation in English, preferably as clear and as detailed as possible and you get back code.
Essentially its very similar workflow when working with offshore developer, in the future you could in theory replace the offshore dev with AI, except with much less latency.
It's called a compiler.
I specifically referring to English though, not computer language.
Those of us that have been doing it a while, know that the single biggest issue in any commercial code project is bad input to the process; otherwise known as Requirements. No one knows what they want, or, even worse, they know what they want, and what they want, sucks.
Steve McConnell wrote about how projects mess up, and how much the damage can propagate throughout a project lifecycle; with bad requirements being the nightmare scenario. This was about a quarter century ago. It is not new (https://stevemcconnell.com/articles/software-quality-at-top-...).
The best requirements for a software project is code. Even an AI will need extremely high-quality, detailed, and well-considered requirements that are likely to require their own language (so AI requirement-generation will look a heck of a lot like "coding").
What AI gives us, is (maybe) speed between requirements and executable, so, theoretically, we could have a really high-power, effective iterative process, where bad input results in bad output, and we see that quickly, then refine the input; much like we already do with code, and run the build again, to see how that results in executable.
I learned development back in the "big iron" days, where CPU time was the most expensive part of the process, and we were required to debug our code in the design phase. If we borked during our slice, we were penalized.
Nowadays, I tend to just write some junky "strawman" code, fire up the debugger, and step through until it breaks.
Working with separate contractors, whether domestic or offshore, needs really good requirements, and some kind of fast feedback and a flexible, iterative lifecycle process; which is frog-fur rare.
If we don't have a fast feedback cycle, then we can't iterate effectively. In-office staff helps to make feedback fast, but it's quite possible to have fast feedback in a distributed environment (I used to work for a Japanese corporation, so I can tell you about "feedback lag"). It's just that we then need some kind of process to afford and regulate that feedback loop.
I think that the process of software development is fundamentally broken, and that the industry is still thrashing around, trying out all sorts of "silver bullets" (See Brooks, Fred).
I also believe that overbuilt process is an enormous issue. I call it "Concrete Galoshes" (https://medium.com/chrismarshallny/concrete-galoshes-a5798a5...), and it affects new-fangled, agile shops, as much as it does old-fashioned waterfalliers.
Unfortunately, if we want to reduce "tribal knowledge," and treat our engineers like interchangeable LEGO blocks, then we need a pretty heavy process. It's still a heavy process, even if we call it "agile."
"Tribal knowledge" is a bad term in the software development industry, but it is the most effective way to have a light-touch process.
It also requires treating engineers well (not just paying them well), so they stay around. I'm not sure that foozball tables and on-site beer are doing the trick, as the average stay at a corporation seems to be less than 2 years.
I do not have the answer. If I did, I would own a private jet for every day of the week. I have my own personal process that works quite well for me, but I don't think it scales particularly well to others.
The books that are published feel like they're targeting entry developers and it's a lot of stuff you'd learn on the job. It feels like we're over-promoting way too quickly and demanding delivery over experience or quality.
Interviews: You have developers that refused to learn this skill and they just skim a few leetcode problems and expect the interviewee to solve it in 15 minutes.
Best practices: What happened to developing standards and demanding/expecting tests (and their appropriate categories unit, integeration, smoke, feature/functional, etc)
Though things could get fun if AI can enable more flexible declarative programming.
Have you read laws? That is the style of language that tries to make a "precise and clear enough description" in English, but it does not go far enough, it's still not sufficiently precise to be applied directly, you would have to use something like 'super-legalese' for that.
So even if you had a system that could accept English, nobody would want to use that because it's difficult to express precise and clear enough descriptions in English, and easier to do so in Python or some other programming language; and in any case, the difficult part of programming is thinking up the precise and clear enough descriptions.
This comes up in writing specifications for outsourced programmers. Writing a specification in English that's clear and actually precise enough takes more work than just writing the code directly - since you have to do all the hard part anyway, and then have to express it in a language that's poorly suited to do so. And a reasonable specification is very much not precise enough to be executed, it relies on the programmer putting in a lot of the details based on their domain understanding and common sense, leading to very different implementations as that domain understanding and common sense varies.
Say I was born in X (low income) and raised there, and then move to Y (high income) and work there as a programmer for several years, then it should be easy for me to hire programmers from X and be the bridge to management at Y?
This. The value a programmer brings to the table is almost entirely their cultural knowledge. It's the same reason why rarely do chart topping hip hop albums in the U.S. come from India or wherever.
Most of them are talented and a joy to work with. However, giving them work is often a form of programming... they will do exactly what you tell them to do, no more and no less.
For example, unless you describe the tests you want them to write, they may not write tests. If they see a possible refactor or optimization, they won't volunteer to do it. One of the frequent side-effects of this is that the project develops more tech debt and slows down with time.
From the perspective of an outsourced programmer, this makes sense. You're optimizing for efficiency. You're in contractor-mode by default because odds are, your career has been heavily contractor work. You haven't been hired to look at the bigger overall picture and you've never been encouraged to develop the agency to make independent decisions.
The most successful teams with outsourced engineering teams have found ways to better integrate them into the team. For example, I know a few startups with engineers domestically in the US whose main purpose is to interact with large teams of engineers in countries like Belarus. This hybrid approach comes with its own challenges but the cost benefits are huge.
Which is why I avoid contract work full stop, at either end of the table. It actively encourages mediocrity and decentivizes initiative and effort.
Yet when people try to hire me they generally tend to ask about my hour rate first not what you just mentioned. Because for my hour rate you'll get an application that you can easily hand off to lesser skilled programmers later because the code and architecture is really friendly to build upon. But tech debt that never happens is something that's hard to sell.
Yup. My anecdote: There was a middleware business application. Two database fields were last-changed-timestamp and last-changed-username. The spec document accidentally flipped the definitions, defining the value for last-changed-timestamp = current-username and vice versa. You can guess what got implemented.
"I ran this and got a type conversion error. Did you even test this?"
"Yes that is what should have happened based on the spec."
The 'Customers' table was referenced in multiple other places and tests, so the delivered system had no problems with that, the customer functionality worked properly; however, sure enough, in addition to the 'Customers' table the database schema also had a table 'Custoners' with all the fields. Because the specification required that.
As long as you start with this hypothesis whatever analysis you do will be wrong.
We do have problem coordinate between teams that share the same hall and the same boss, I cannot even imagine what would happen with complete remote team in Vietnam to save money. It would be just better to stop develop software ourselves.
Now with COVID-19 things are changing, but we must wait a full cycles of hires (read 10 years) before we can really tell. At the moment we are all remote, but the conditions are the same of when we left the office. And on-boarding people seems though now...
So perhaps proximity is not the issue?
As long as we are close to each other, very important things don't get lost.
(Personally, I'm pretty damn sick of management, of any level, insisting that these are the responsibilities, these are the tasks, if you can't get them done it's your fault, even in the face of their team telling them, repeatedly, that they're overworked. That's abusive management, period. Hire more damn people.)
The problem is, you can have 10-20 people tapping you on the shoulder all at once.
The bigger the company, the less this is true (although there are also a whole host of smaller companies where this isn't true either). My last 3 companies are not filled with people who are overworked AND have little responsibility. Remote-only has exacerbated the slack.
For example I find pinging off ideas off each other and explaining things much harder on remote due to a lack of a whiteboard or a notepad. No collaboration tool I tried was even remotely a good substitue.
I like the conversations I can have while taking a coffee. And going out for drinks with colleagues from time to time.
I hate working alone in my bubble all day long. I like having the possibility to do so from time to time, but not all the time.
As it was already mentioned in other comments time proximity is important. Managing things like pull requests or user tests over large time differences makes thing considerably slower.
The whole premise of full remote is only valid for a subset of projects and team compositions.
It reminded me of Gregory Clark's 1987 paper "Why Isn't the Whole World Developed? Lessons from the Cotton Mills" https://pdfs.semanticscholar.org/6152/0798b9dd2c691872d58db3... where he points out that despite Indian labor being incredibly cheap, Indian cotton mills were still uncompetitive with British (and later, Japanese) cotton mills for basically the same reasons. As the saying goes, "you get what you pay for"... The more something is an o-ring process https://en.wikipedia.org/wiki/O-ring_theory_of_economic_deve... , the more skimping on labor is penny-wise pound-foolish.
What makes my experience a little more representative (or so my ego says) is the fact that I am from Eastern Europe. Ever since several years now (3-4) we are no longer that attractive IT outsourcing location because a lot of people around here learned their price and value and charge normal (if still below USA's) rates. And thus many businesses started trying to outsource cheaper to Africa and parts of Asia.
The results have been catastrophic and hilarious at the same time. I've been called to "fix the mess" 6-12 months later by five businesses now, and I outright told one of the business owners that he'll have to pay me $20k a month to fix something as non-salvageable as the dumpster fire I was given. He refused which was a win for me because I still value my time and sanity.
So many businesses look at this outsourcing to cheaper staff thing very superficially, as you said. Sure you save money the first 2-3 months. Absolutely, you do. But then all the unfinished and badly done work starts peeking from the edge of the window.
And finally, you are also very correct when saying that hiring the managers that govern these people efficiently effectively destroys the cost savings.
In conclusion: I have definitely observed the same as you.
(Several edits: apparently I had a stroke and wrote some really broken English here and there.)
Eastern Europe is not a great example of productivity either.
I can totally understand why employees from poorer countries disappear all the time - why wouldn't they? They know they're getting paid peanuts in your country (so you don't expect much) and you're paying them an incredibly competitive salary in their own country (which means their lifestyle is closer to upper middle class).
Which means they can piggy-back off your seemingly efficient decisions and get away with doing less.
You should know there's a lot of world-class talent around here. Sure there's tons of slackers but I posit that's true everywhere (maybe they can't squeeze through the cracks in the USA? but they still exist).
I really hope that in the near future at least some African countries could be in a position to move up the value chain and take control of their own destiny as well.
> Garett Jones (2013) builds upon Kremer's O-ring theory [...] He then goes on to show that small international variations in average worker skill per country result in both large international and small intra-national income inequality.
This sums up the situation to me. The value of the skilled worker depends on the value (and desired quality) of what is being produced with co-workers. So in theory if you could build up a distributed team of these skilled lower-salaried workers, have them work well remotely with each other, have a high sense of quality and share other relevant cultural norms, then high-quality/value products could be produced for such a market demand.
No matter how much you try the only way to fix that is to equalize their opportunities, but you would never do that because you're inherently interested in keeping them poor.
Things do change.
sounds like specifically a work ethic problem, which is indeed an issue outside the US , Western europe, japan etc, assuming those programmers were indeed capable enough and getting paid above-decently. I don't know how you go about building work ethic in countries that don't have it
So the question really becomes, not why are salaries different, but "why is programming difficult to offshore".
If programming was easy to import and export then either salaries would even out or, more likely, would be relocated to cheaper countries like many kinds of manufacturing.
But software is difficult to outsource even from one department to another let alone between companies or countries.
If you can solve outsourcing, then you've solved the problem of knowing what you want from software.
But if you've done that, you've just become the software engineer.
Lack of professionalism and absence of a professional engineering culture in many of the places you'd seek to outsource to save money.
Doesn't mean that professionalism doesn't exist, it's just that the ones that have it are already busy with their own thing and not interested in being low-cost labor
I used to work in a relatively "low salary programming culture" and it wasn't just the salary that sucked in that location, the programmers did too. Smart people either emigrated or ventured into a profession that made $$$ (typically real estate, management or finance). Eventually I left too.
Lack of professionalism and absence of a
professional engineering culture in many
of the places you'd seek to outsource to
save money.
Contract-based engineering is nearly always a recipe for technical debt.
The team will nearly always focus on today's problems. Not tomorrow's. Because... why should they? They are judged solely on how well they're solving today's problem. And they likely won't be around in three months or three years to reap the true rewards of reduced technical debt anyway.
Contracted engineers also haven't simply been around your business long enough to truly understand it and anticipate what future needs will even exist.
All other things being roughly equal I'll take a full-time, permanent, offshore engineer over an onshore contractor any day of the week.
I had one positive consulting relationship. It was a longer-term project. I was the sole consultant and I was able to work closely with my client and explain tradeoffs between short-term deliverables and longer-term technical debt. My client was also instrumental of course and brought a very understanding approach to the table, and was able to understand the technical side of things.
But of course, that is almost never the case. Even with good intentions, that approach never seems to scale up. Not even a little bit.
Some consulting shops do try to bring a similar approach to the table in a scaled-up way. My current employer hired one of the better-known Ruby shops for a long-term engagement.
On the bright side, I would say that the consulting shop did try their best to manage technical debt and bring a good engineering approach. On the flip side, I would say that the end results were decidedly mixed.
It might be a cultural thing, but issues I've ran into- Slow/low prioritization, no movement unless you follow up often, lack of experience.
Don't get me wrong, I can be lazy too. But when things are outsourced it's really hard to keep positive relations and get stuff done on time with high quality.
Those outsourced teams are also aware of their status as "disposable cheap workforce" and are not super invested in the project. They have very little chance of career growth within the company since their whole reason to be here is to be cheap. They do the bare minimum to meet the objectives and nothing more, and in their place I'd probably do the exact same thing.
The rest is cheap in comparison, whether done in SF or Bangalore.
In other cases, it's skilled people that just don't want to apply 100% effort and are just coasting, maybe for external reasons (some folks can write decent code without 100% effort in the moment due to practice beforehand).
If your code is unmaintainable you become hostage to your own code because making changes breaks things, causes frustration, etc
I totally get the idea of job security if you’re maintaining a codebase that only you understand but that is very likely to bite you back quite badly, especially if you’re a contractor: you may not get any new work, the code you write is very visible and can be judged quite quickly by a qualified person
1. figuring out what to build
2. for non-trivial/long-lived projects, figuring out how to build it in a way that makes scaling, iteration, and maintenance something other than a total nightmare. (of course, many projects are one-off projects that don't need to be engineered for the long haul)
That’s not how software is done though.
Product managers are in a constant negotiation with engineers to decide what to do. There is a lot of back and forth and exchanges to arrive to a set of stuff that can then be built.
That’s what the agile routine is about.
As a team, we can't just go all vigilante, but we can push back and have at times gone over other people's heads to make sure the proper people understood the problem. Most of the time we got what we thought was best for the customer by going over people's heads, but there's been plenty of times that a middle ground was reached.
The main thing is to not just say "we can't". Our job is to deliver solutions, not products. If there is a problem with priorities, we find an acceptable way to make things "work". Maybe that's adjusting timelines, or features, or just asking the customer how important that timeline is to them. I can't tell you how many times I've been told about "deadlines" that the customer never actually said was a deadline.
Us: Yo cust, can we slide that project 2-4 weeks?
Them: Sure. Heck, move it back 2 months if you want, but we absolutely need to start testing on our end in 3 months.
1. Rejected from working in a gamedev in Europe because I am not EU Citizen.
2. Lots of companies on USA have to follow some crazy tax rules or general regulations that make them not allowed to hire foreigners, don't even need to be military stuff. Noticed some companies won't even hire out of state.
3. Military, healthcare and many other industries for obvious reasons don't accept foreigners.
4. Several companies have some for reason or another have all employees working on the same time, meaning they refrain from hiring outside their timezones (thus no California companies for me, and no Europeans either)
5. A bunch of times had problems because mismatch of required education and how education work in different countries.
Hiring out of state (but still in the US) means dealing with taxes for the other state, plus probably adds complexity and maybe cost to providing health insurance. The best quality per dollar spent on health insurance in the US is often for plans that have little or very poor coverage (that is, pays few or no providers, especially for non-emergency care) outside a given state or geographic area.
Yes, that's right, lots of people (and companies) in the US pay outrageous-by-OECD-standards rates for health insurance that barely even applies in most of the country.
The theoretical benefits of offshoring are obvious: get the same thing for less. The same goes for juniorization, find a young guy who can do the same thing for less (this worked out great one time in an HFT I worked for).
Your problem is there's a chance -whether you offshored or not- that not only do you not get the same thing, you get a steaming pile that your boss will not appreciate. And it's not like a 80% working software solution is worth 80% of the money, it's worth maybe 20%.
So how do we compensate? Fall back on crutches like "we only hire Ivy League or Oxbridge grads". Or make sure whoever it is had a roughly similar culture to yourself (comms style, language), is available in a similar time zone, or has done similar work in the past. It's not that these will ensure you have good programmers, it's more that you can say to your boss, "I acted conservatively but even then sometimes you don't get the right hire".
All of which point towards advantaging programmers in the West. Either you have all these qualities (culture, tz, experience), or you have to discount quite a bit.
Naturally this isn't universal. You are gonna find people in Asia who somehow make a Western salary, and good on them.
‘Oh it’s because their country sucks’
Ah ok, and you found 30 geniuses in this crappy country?
‘Yep, these particular guys are all the cream of the crop’
Got it. Okay, let’s give it a gamble. I’m not making a real point, so don’t beat up my straw man, this is just my gut feeling.
I was born in a cheap city in southern Europe, and all my life I moved to increasingly bigger and more expensive cities. Why? In my home town job opportunities are very poor. Local jobs are scarce and for full remote positions you are competing globally.
Sure, London is expensive as hell, but there are many more opportunities, so I can progress my career, and drive up my salary. A bigger talent pool attracts more companies, more job opportunities attract more people, and so the snowball goes...
Part of my team was already very affordable contractors in America. The company found a way to get even cheaper in India. I don’t think it’s as simple as cost, as in this case I think the company literally looked at the product and concluded ‘we don’t need this level of quality’. I understand I’m implying the quality will drop due to outsourcing, but it has nothing to do with a prejudice towards off shore work. If you drop prices dramatically the logic behind it is that you are prepared for a quality drop as quality is not the prime concern.
Every year, I spend more and more money on things I value. Sometimes it’s just a stupid upgrade on a chair, monitor, clothes, etc. If I fundamentally don’t find the value in a good mechanical keyboard, every year I will try to reduce my investment in it. That doesn’t mean I’m looking for a bad cheap keyboard, but that I’m okay with something passable since I see no value in a better one. I see more value in getting away with a passable one.
Is there a trillion dollars of untapped savings? Yes, for those seeking it, for those with different values.
1. Software industry is partially an oligarchy. A few companies make a huge amount of money. A large amount of companies make less but still a sizeable amount of money.
2. The amount of developers needed in the really wealthy companies is limited. The wealthy companies don't need to compete that much for talent but they also don't need to keep down the salaries that much.
3. There is a cost associated with diversifying to different countries. If you don't need to, why would you?
The market overall has about medium buying power. A small part of the market has huge buying power. The wealthy companies end up in a few places and don't have a limited interest in diversifying geographically.
[1] Using "Google" here as a proxy for companies that pay like Google.
> One could argue a huge inefficiency still exists if you assume that offices are not useful and companies should always work&hire remotely (thus removing the cost of living from the picture).
> But in that case, I would at least expect office working to go away before we see any dent in the wage gap. However, a lot of companies seem to be set on office work. Whether or not this is a good thing, I don't know, I lean strongly towards "no" but that's a post for another time.
I've been a full-time telecommuter for 6 years: Offices aren't going away anytime soon. So many people need face-to-face communication with their colleagues. Even I need to travel to the office from time to time. I've always had a technical role; when the opportunity came to be a "manager" I rejected it because I just couldn't envision myself as a manager without being physically in the office at least 1-2 times a week.
So, if you ask me, the reason for the salary difference is because we need regular face-to-face contact with our peers.
This is why developers must be paid well. It is extremely hard to focus on a problem you don't care about. Every little bit of lost focus is lost time/money. It's more bugs in the future. It's more churn.
There aren't many jobs that are this dependent on the workers will to do it.
> Again, fairly hard to find data on this
I'm surprised it's hard to find data, but yes, this consistently matches my observation over the past 30 years and four different geographic locations: very few American programmers are American at all. When I did my master's in CS 15 years ago, I was almost always the only person in the room born in the United States, and besides the (either Indian or Chinese-born) professor, the only US citizen as well.
At most schools the CS master's program is overwhelmingly international students. That's not the case for undergrad.
Although CS masters in the US are glorified paid immigration programs so it’s not a perfect comparison. But this also holds up in industry.
A master's degree from an American university is a huge benefit when trying to get a visa. That means there are plenty of international students who want to get master's degrees from American universities because it helps them get into the country.
Americans don't need visas, so most of them don't bother getting a master's degree, because there is not as much benefit relative to the opportunity cost.
Just 2-3 days ago we sent 2 chaps into space. 50 years ago with far more primitive software we sent them to the moon. And you are bitching about how your software profession is so fragile it’ll be messed up because of culture, because of time zones, because of skillset, because people “over there” don’t look like you think like you talk like you walk like you. How long can you put up with this farce ? For a long time, thats for sure. But not for ever.
This stuff aint particularly hard, people. Yes not everyone can do it. And yet, its also true that wasting the best years of your life trying to design a 10mb facebook page isn’t perhaps the best use of your brain cells. Perhaps if you are truly that smart, why not work on something more worthwhile ? Go solve cancer or robotics or self driving or....not this agile webcrap for christs sake. Is this truly what you want to stake your future on , that someone in a different part of the planet can’t write javascript as good as the L5 sitting in Menlo Park ? It doesn’t even sound good on paper. Next you’ll tell me you believe in the tooth fairy.Jesus.
TikTok is doing extremely well and rapidly gaining users. They demonstrate a good knowledge of everything that SV typically does and are based entirely out of China. These arguments about culture are laughable. TikTok is basically the engine of culture now among teens and is playing a big part in the current protests.
Zoom has its engineering based in China. This means two of the fastest growing consumer products today are built by Chinese software engineers.
The Chinese are also at the bleeding edge of Machine Learning and are taking baby steps in building cloud services. With the elimination of Huawei from Western tech ecosystem they will be forced to build an entire stack from Silicon => Pixels and its well within their reach in the next decade.
IntelliJ which everyone here swears by is built entirely out of Russia.
All FANGs are rapidly scaling up their presence outside the US in locations like Europe/Canada to hedge US visa issues which will intensify over time. The only way to employ non US software engineers along with western ones might be to have offices in non US western countries.
While they do have offices in Russia, JetBrains is a Czech company.
Overall though, I agree... there are a lot of people with their heads in the sand.
Ouch. Yeah some US/UK doctors are overpaid but the liability is not addressed. Also not sure what algorithms they are referring to. I don’t know any even in radiology/pathology that can rival a physician.
From my friends who work with outsourced programmers, this is also an issue in the coding world right? Who’s held responsible if something is not working or delivered?
A lot of people like to point to regulations and licensing as a source of inefficiency, but in reality the lack of any sort of professional licensing around programming drives all sorts of alternate inefficiency: with no objective standard to measure applicants against, employers are stuck trying to make up their own, usually broken standards.
But the flip side, using newer research, may fix incorrect/suboptimal diagnosis or treatment, but can also fail.
I'm of course sidestepping the "I'm an MD and am right, and you're a layperson and don't know medicine" mentality, but to be fair, there are fewer of these in the newer generation of MD/DO/healthcare providers.
You're saying this based on..?
This is an assumption we have about the world, and it's a true assumption for competitive markets, where new companies/busiensses can freely enter, where barriers to entry are low. But, with software busineses, barriers to entry are extremely high and this results in businesses that are far less competitive on price and thus price of labor. The result is that many many of the biggest software companies are monopolies. Monopolies don't need to be competitive on the price of their labor.
Really? I would say it's one of the lowest, where you can start a global company with less than 20 people with very low costs that then increase sub-linearly as your customer base grows.
Every effort failed miserably.
The core of the issue was getting / maintaining talent / a real lack of professionalism in the folks we ended up with / lack of dedication to the job.
If there was any reasonable amount of saving money per workload, you just couldn't get a company to outsource to who actually kept their own workers around long enough to be productive. Anyone who was productive almost immediately proved their worth and left for a better job, or as we suspect the outsourcing companies actually moved them to clients that paid more (in the case of us hiring them directly, we knew they left).
The folks you did get would close cases randomly / played a sort of hardball metrics game where they really just upset customers / state side we'd end up cleaning up the mess. Lots of technically correct solutions / statements that really don't solve the problem... And in the meantime it all ate up a TON of upper and lower management time trying to get things right / set up... meanwhile there was a US, a European, and even in one case a Japan side support team that needed little interaction from management.
Another effort tried to establish a local presence offshore (a few companies already had resources there) ... same results.
The end result was all the effort and costs resulted in it being more expensive to offshore in many cases if you looked at the real numbers.
To be clear I'm sure there are plenty of professional folks out there to be hired (we had plenty of those folks employed FROM those places who were outstanding at their job) I suspect the larger issue was that the savings weren't really that apparent, and what quality / dedication you get for X price isn't always as apparent when someone says "Hey X resource is Y cheaper over there."
Quality and outcomes are hard to calculate / something being cheaper on a spreadsheet doesn't mean it really is.
Agglomeration is the idea that there are a cluster of benfits arising from clustering people doing an activity together. Inter-firm sharing of technologies and practices, in particular. A lot of the transferred knowledge is informal and implicit, and a lot of the transfer happens outside of the formal workplace.
Transport costs: as transport costs decline (and they certainly have done so for information, voice and video communications), then previously negligible differences in costs become significant.
So, paradoxically, cheaper communications increase the relative value of clustering together, and especially the value of being physically near your biggest customers.
This is true even for whiteware manufacturers, let alone high-information services.
The most successful outsourcing group I worked with was in India. I did some math at the time and estimated that overall, they were about as productive as the US per dollar spent.
That hints that the market roughly optimized correctly. Of course, there was a huge diversity of productivity between individuals in both groups, but the management had no real idea how to evaluate that (if they had, they would have paid people quite different amounts).
I think a very important factor in a good programming job is communication. It takes communication to convey requirement and expectation, it takes communication to understand requirement and expectation, and it takes communication to spot and clarify misunderstanding and confusion. Reviewing my own work, I think I am dealing with clarifying misunderstanding and confusion most of the time. It is rare I gave a code review and the author understand my comment spot on right away. There is always some misunderstanding and cross-talking. I think the issue is so bad that even within the same department where we can freely discuss face to face with a whiteboard, we often ended up avoiding touching controversial code, avoid refactoring, instead embracing add-on orthogonal patches, and only focusing the review on trivial stuff such as white spaces and test failures. Our software sucks because of this communication factor. In fact, I think the OOP prevails because it allows us to sidestep the communication to some extent (at the cost of bad code (but functional) ).
The communication factor dictates that we hire people within the culture, within the location, and within the education background -- regardless of the correlation to skills.
"There's probably some added legal hassle and maybe even accounting-related hassle, but it doesn't seem significant (arguably even less in some situation, since people from other countries can work as contractors rather than employees)."
As programmers we'd like the simplest possible implementation of hiring: I would like to post a job offer for role x at salary y regardless of hiring location.
The reality of implementing that as a business in a compliant manner - involving international payment methods, tax reporting implications across multiple jurisdictions, legal implications, etc do not strike me as trivial -- if you're a small company they're time-consuming and require expensive consulting, and if you're a large company it may be difficult to get your internal tools and processes to align with them.
I'd love to be proven wrong, and would like these systems to be simpler and fairer. They certainly shouldn't be torn down until the positive reasons for their existence are understood and handled.
I feel like many common narratives around 'programmers can work from anywhere' come from a naive point of view and rarely include details about how to compliantly achieve those results, from either the business or individual's point of view.
There are also different cultural aspects about software that change between countries. Growing up in the US and going to college after big tech was already ascendant, probably the main reason I went into technology is that it seemed the best way for me to make a money (esp. given quality of life, eg not having to do the IB grind). Anecdotally a very large portion of the career/money oriented people I knew growing up and in college also went into the technology industry. I assume the same is true in some countries like China and India and less true in countries like Spain. You can see cultural differences regarding how well-regarded being a software engineer is in Western Europe vs. other countries in titles, how software companies are run (eg by former engineers or “business people”), pay, etc. If I had grown up in Spain and for some reason didn’t want to move to Switzerland/US I would probably not have gone into technology.
This also is evidence of/creates a feedback loop where pay->market rate->interest->talent->pay causes regional software markets to diverge.
By all means though, if you think you can get the same work done in another country for less money, do so. I think you’ll find that the $20 you’re picking up off the ground actually comes with a high interest rate. But you won’t believe me until you try it yourself
Every intervening layer or other impediment between understanding/interpretation of customer requirements and coding reduces efficiency. Language and cultural understanding barriers, whether directly exposed or mitigated by adding an intervening translating layer between the technical team and customers are real problems. And the customers with money to spend are disproportionately in richer countries. So even with equally valuable raw technical skills, programmers in richer countries are legitimately worth more. But the money offered and immigration/non-immigrant worker policies mean that the most valuable workers from poorer countries will, over time, being drawn into the richer countries, exacerbating the divide.
And, in any case, even with neoliberal global trade regimes, there are additional transaction costs to overseas outsourcing, even with programming and other information/intellectual work. So, of course the law of one price that applies in idealized integrated markets doesn't apply.
Adam Smith - An Inquiry into the Nature and Causes of the Wealth of Nations
” It is not, accordingly, in the richest countries, but in the most thriving, or in those which are growing rich the fastest, that the wages of labour are highest.”
In those places, you are willing to pay the most since it will make you the most.
Brooke’s Law
Wikipedia “ Communication overhead increases as the number of people increases. Due to combinatorial explosion, the number of different communication channels increases rapidly with the number of people.[3] Everyone working on the same task needs to keep in sync, so as more people are added they spend more time trying to find out what everyone else is doing.”
You want to cluster people together since distributing people creates communications overhead.
Adam Smith also explains why innovation tends to cluster. It is because communication lets learning and innovation occur more rapidly.
Lastly, labor moves to where there are the highest wages.
You saw all this the last 10 years. Facebook willing to pay top dollar. Companies growing their offices bigger and bigger. People moving to Silicon Valley for the money. Silicon Valley companies copying best practices in the blink of an eye.
It also explains why Zoom has such good margins. Their engineering wasn’t distributed. It was in China. Labor arbitrage but with a minimization of communications overhead.
Companies don’t attack all markets at once. They attack them one at a time (usually local market first).
It is in fact in an efficient market that you have disparities in wages.
The bigger the cluster the bigger the network effect in the cluster.
My opinion is that there are probably less than 10 viable clusters in North America. If it is too small or the culture is against it, then it fails. There have been such attempts, a few succeeded, most failed.
It shall be interesting which (new) clusters grow. My bet is on Toronto. AI talent. Large cosmopolitan city. Good universities.
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.
— Melvin E. Conway
The real power of open source is that there are enough up and coming people who haven't learned this, and so if maintainers aren't supported it doesn't matter, there will be a new library to migrate to in the future by a new college grad, and this is one reason why things tend to be so unstable and short-lived.
The most successful open source projects have large corporate backing, like Kubernetes or Linux. And even Linux is a weird case, as a lot of architectural decisions and feature development are wonky because you often can't get the e.g. IO and Network subsystem guys to agree to the same idea.
Most of the time, as long as the hire is above a certain skill level, the bottleneck is absolutely the communication bandwidth between you and them. If they don't completely understand things or you don't understand them, the number of round trips increases. Add to a general sheepishness when presented with American English and slang, as well as certain cultural tendencies, and you encounter a lot of hires who will say yes (especially from India and Vietnam) when they don't understand. This is a big problem.
I've also seen the efficient market at work, but on a level up from the programmers themselves. In a lot of cases, programmers from developing countries are hired out by agencies who will charge a lot closer to market while paying them less. This might cause a lot of the disparities in the data looked at by the author, and in any data where salaries are compared directly.
Shortly after someone from India turned off the power to an entire refinery.
In another story that I experienced myself first hand a huge customer we had had outsourced their IT department.
At some point a critical system went down in the middle of the night putting lives and hundreds of millons of dollars of equipment at increased risk not to mention violating regulations.
These things happens from time to time in large systems, but there is a huge difference between calling a guy or gal who knows it versus calling IT support, hear them tell that they will create an emergency ticket, and then get a call back 4 hours later from a smiling technician that wants to help me.
Long story short: by then we had already got someone on the site to break a handful of rules to get it working again.
I have more of these.
People in other countries are equally smart, often smarter than me, but having a bit of mechanical sympathy and a bit of knowledge of what you are supporting, knowing how to recognize an emergency when communicated from a person from another culture, all those things matter.
I assume that's a typo for "taught" but maybe it's one of those Freudian slips. I'd say being at least medium-clever and being thought of as a programmer is the prerequisite for getting paid loads of money as a "good" programmer.
Someone mentioned that contractors don't write tests unless you explicitly tell them which ones to write. Well, even if they did, are you going to be able to promote them? That dev's manager is probably hyper-focused on spec completion and billable hours. Even if you paid more for test coverage you start paying for things like assert sum(1,1) == 2.
High quality code often comes from people coding for "the job they want" not the job they currently have. In-house work usually has this carrot of a senior role that can be attained by taking initiative and thinking outside the spec for the missing pieces. This is a form of compensation and incentive missing in low end contracting work.
If you do not speak the same language at the same level of fluency of the organization, you're not a good fit for the organization.
It's also far easier to communicate when you 1) occupy the same space or at least 2) occupy roughly similar time zones.
While we'd all love the ideal "everything is written down and ultra clear", the fact is that rarely happens, and taking the time to do so takes a lot of effort that could be better spent on development.
Small differences in communication disparities can lead to large differences in outcome. You've probably noticed this in your own work - somebody missed a tiny requirement but it has a huge impact on the scope of work.
Your example of Germany and Switzerland in this case is less illustrative than say India and US - the poorest places automatically lose the best talent because the standard of living is not comparable even if you were earning the same remotely and had 1/10 living expenses. And then you factor in the quality of education available, projects available, etc. Juniors have a hard time getting remote jobs - so they are limited to local job market - and if most projects in your country are low end outsourcing jobs you get experienced in those vs. more demanding work.
Surely it's easy from Germany to Switzerland or Canada to the US, when you're moving to a different continent it's a whole different experience.
I know btw one developer personally who went from Switzerland to work in Leipzig, Germany. He said he much preferred the openness of the people there to his home country.
In the end everyone is different and people move for different reasons.. not only because of salaries or taxes..
The reason this doesn't show up in the data is that hypothesis 3 assumes that people want to do remote work from their country. What I actually observe is lots of programmers from the Middle East and Asia coming to Germany, the US and probably other high income countries. Obviously emigration is easier for more capable programmers, lowering the average skill of those left behind.
1. Even remote-first companies are likely to want to all gather together once in a while.
2. It's easy to run afowl of things like American crypto export restrictions when doing work in a foreign country
3. Startups are going to want to prefer working with local people because you can simply get more done working in a room with someone than via slack / zoom / etc. Moreover, hiring a single employee gives you more leverage than hiring an outsourcing firm.
4. Remote, foreign employees are going to be independent contractors, because actually complying with 50 different employment laws is really really hard.
5. Even if an employee is hired in a foreign country, that country's laws may lower the salary. For example, taxes vary by country. Payroll tax is higher in Europe than America, thus companies may be spending the same amount, but the employee receiving less. Similarly, in countries like India (and some in Europe too), there are much stricter requirements on when you can fire employees, whereas in many US states, like California, employment is at-will. This would also be expected to lower salaries in areas where more employees could easily become a liability.
For example, an average US employee will on average work 30% more hours per year than a German employee [1].
Working in the Netherlands, I also know that a lot of engineers will deliberately choose to work part time (e.g. 30h/week) even if it ends up with a lower annual income.
Meanwhile no bootstrapped founder in their right mind would try to hire in SF to build an MVP unless they've got VC investors with deep pockets adamant on it. The smarter move would be to hire remotely, vastly opening up the talent pool and getting significantly more bang for your buck.
If you're a U.S. citizen, you can always move to SF/NYC and make those high salaries. You just might choose not to because you don't want to move there. If you're Swiss, there are plenty of high paying jobs, not just in IT but other fields.
If you're a Ukrainian citizen, you don't have the option to make $500k+/yr at <insert big tech company> (unless you're willing and able to go through the immigration process into the U.S., which is difficult). Naturally that leads to less bargaining power.
Ultimately wages are just a negotiation.
I am not sure how that is not a simple corollary of economics behaving more or less how it usually does.
"Moving on from the right to the left of the economic axis we have a maximally corrupt market, which serves only to funnel money to the richest people at the expenses of the poorer at all levels and thus global inequality in anything (including salaries) should be correlated with how rich a country already is.
"How do we identify a "rich oppressive country"? Hard to tell.
"Is PPP per capita as good and indicator as any? Probably. Since I don't know how recent the Bloomberg data is, I will go ahead and use the GDP data from 2012, since that's when my PISA data is from."
High GDP indicates high inequality?
"Programmer salaries are very strongly correlated with how wealthy a country is.
"Programmer salaries are strongly correlated with how well-educated a country is.
"Programmer salaries are relatively well correlated with how well a certain country speaks English.
"Programmer salaries are inversely correlated with % of the population working in IT.
"But since all of those 4 things are strongly correlated to each other."
Ok, I'll buy that. But I'm still not sure what this has to do with the question asked by the title.
I worked in NZ, and I was paid in NZD, but the company sold their products in USD. This meant the profit fluctuated. Dramatically.
The exchange rate has moved from ~0.4 USD/NZD to ~0.85USD/NZD.
Right now, it's sitting about 0.65, but you can see the problem. If you price something at 40c, it goes to 85c, you've got a problem. Even a 25% swing should be expected.
A lot of these cost savings can be valid short term, but setting up a team (rather than a project) is a multi-year thing. Don't forget the currency risks.
I have specific experience with this, where they attempted to outsource the engineering team to India. However, the exchange rate shifted, and the ROI disappeared. Two years of distraction and training our replacements gone.
This can work well, it's why the Lord of the Rings movies were made. You have to know where your risks are, or you could find the developers that you thought were cheap all of a sudden aren't. Generally, it means you need to do it for some reason other than price.
In a perfectly competitive free market, wages (and capital gains, too) should correspond to subsistence wages. However, this poses a problem. If all that additional value is created by firms, to whom is it all divided, if everybody gets just subsistence?
In my view, simply, this situation cannot happen. Instead, in the real world, the markets are always full of oligopolies which were intentionally created by all market participants (including employees) to split up all the created value, somehow.
In practice it takes many forms. It is likely that company employees unconsciously form coalitions that sabotage lot of outsourcing. Some of the coalitions correspond to nation states. In fact, should they fail to do it, they will have less income, and will be less relevant, because they will make less spending decisions.
So the result is somewhat random, it is result of dynamic evolutionary pressures that create market friction, and cannot be explained away by forces of free market rationality.
Wait, why? You might want to support that claim a little better before you build a whole theory on it. Mainstream theory suggests that in the long run profit will converge to zero, but that doesn't mean that wages, which are the prices set in the labor market, will all converge on the same value. Frankly it doesn't even mean it's not possible to profit while running a business, because the cost of capital is counted against profit when it's claimed that profit converges to zero.
But if profit (value) is created in the economy as a whole (that is the reason to have it in the first place), and yet every unit in it converges to have zero profit (marginal revenue = marginal cost), what happens with all the extra money?
Or ask the other way around. Under neoclassical theory, the price of all commodities should converge to their costs. If labor is just another commodity, it should be true for labor as well. What is different for labor, from the perspective of the neoclassical theory, that it wouldn't behave like that?
I think the neoclassical theory is inconsistent, and the assumption, that because all these can converge to zero profit independently, they also can converge altogether, is simply wrong. Therefore, there will always be an area in the economy, which doesn't operate according to neoclassical theory, where the profits are close to zero, that is, there will a component of income that cannot be explained away by costs (which is what the article is trying to do).
Taxes and regulations are complicated, it is not "just hire a contractor", market is not all knowing and all optimizing because there is no global market, each country has its own market, because of taxes, regulations, visas. You still have to have trusted contractor otherwise you risk money. Good luck chasing someone from the other part of globe if something goes really bad.
For setting up branch office, that costs a lot of money and time. If you have medium business and you are a company owner you don't have time to manage an office 1k miles away. You have to spend time getting customers and selling, again you don't want to waste time on learning tax code in some other country. I bet also most people would rather earn money locally and spend time with their family. There is a huge opportunity cost.
I have a suspicion that investors in developing countries invest in a broader range of companies and it's more like a free market than in the west.
Of course if a company is remote first it will already be at advantage, but most companies I know are not known by it's management efficiency.
So a trillion dollars are lying on the ground, but its buried deep and you require the tools to dig.
1) A lot of correlation can be meaningful only if you can extract independent variables and have a big enough N.
Given the complexity of human society, we would need 100000 countries to meaningful correlations.
2) "rich oppressive country" is extremely correlated to wealth and "English skills". Silicon Valley is an excellent example.
As far as I know, some specialisations are simply not found at all (e.g. aerospace engineering).
But it's all over the economy really, Switzerland is really reliant an foreign labour. Just have a look at construction or healthcare.
Also, learning the local language helps integrating once you arrive.
https://www.linkedin.com/jobs/search/?currentJobId=189170609...
Quite a few job descriptions are in German / French. In others the description could be in English but they will ask for basic/middle level of German. And then there are the few (hard to estimate how many) that don't care for language at all. But when you consider that to begin with the Swiss job market is small, this means it's much harder to find a job there than a place like Germany / Netherlands / UK for an immigrant. Now sure, some people will happily jump through hoops because Switzerland is undeniable a high salary, low tax, high quality of life country. I guess I'm not one of those people :p)
This seems like a very ironic problem when considering the content of the "alternate" article being presented... Example of an excerpt: "The problem is that you thought you requested a core product with a couple of extra features, if they were easy enough to implement. Meanwhile, the dev team heard something else. They heard about some exciting challenges they could tackle… and a slew of boring, basic features they couldn’t be bothered to test properly or care about."
Assume for a moment that acquiring companies don't understand what organizational practices/structures make the companies they're acquiring productive. (If they did, they'd just build an internal team that does things that way.) Assume that, instead, "acquisitions" are basically a feudal transfer of power: a different knight is now collecting the taxes from the serfs, but neither the new nor the old knight understands how wheat is grown. The serfs are a black box, and a fragile one. If they want to continue to bring in taxed wheat, they have to mostly leave the black box to its business, rather than poking around inside it trying to change things they don't understand.
Under this paradigm, you can sell a co-located team to an acquirer, and it'll retain most of its productive value, because it's very easy to just "leave the serfs to their work": you just keep the same people working together in the same office. But you can't really sell a team with mostly-independent foreign subsidiaries to an acquirer, since the process of absorbing those subsidiaries into the corporate structure will naturally (for legal + political reasons) tend to force restructurings of such subsidiaries. The black box would have to be taken apart—and since the acquirer doesn't trust their ability to put it back together in working condition, they just don't want to buy it at all.
As well, a colocated team can be acquihired for the purpose of pilfering specific members of that team for ones' own org, while discarding the rest. This possible use increases the saleable-asset value of such teams. A foreign-subsidiary team cannot really be pilfered in such a way, unless the acquirer happens to also have a foreign subsidiary of their own in the same country, which they'd benefit from transferring the acquired employees into.
But also, you don’t want to hire “programmers,” you want to hire engineers—people who are capable of making large systems work together.
That's exactly the point. Also keep in mind that it's not uncommon for developers in Eastern Europe to pay only 5-10% income tax when they register as private entrepreneurs (and benefit from small business tax reliefs), far lower than the tax rate in the US.
This map near the start of the OP makes me wonder whether, similar to the markets in the Bay Area and New York, the markets in Switzerland and Norway give programmers some credible alternative (not available in for example Ukraine or Vietnam) that can be resorted to by a large fraction of programmer-employees.
In that sense, you need customers for software, and wealthy countries have it.
Silicon Valley might skew more toward global software products, but I’d take a wild guess that most software is actually more local in scope.
That banking or insurance software you wrote in Indianapolis might not work at all for non-US companies, or it is perhaps internal and company-specific.
So, in that way my unscientific guess is that the USA has a huge number of corporations that need internal software or locality-specific software.
If you outsource talent pool from Finland, the cost is not so much different on paying salary in US.
I do think, that in future it is the norm to have a remote community in all companies to control cost and to access best talent. It will be an accepted fact that people are paid differently based by location.
It might be interesting to look at the `N_programmers * AVG_salary` for various countries. Maybe no explanation is needed for disparities if most programmers aren't affected by them because they've all moved to where the high salaries are. Though I doubt this is an explanation.
Also, let's check back on this a year and two and three from now, when we've fully absorbed the impact of covid-19 shutdowns on the knowledge industry.
Sorry, but I know developers with way more natural talent than I have but without context to what they want to accomplish it means little. Can those developers learn what I know and then be 10x what I am? sure. But there are few domains where there isnt a learning curve. That creates inefficient markets. That creates disparity between salaries.
In some countries you can earn more waiting tables than a programmer in others. Much more than a waiter in others. Do you think PISA scores matter for that? Or that a waiter is three times more productive in rich countries?
The explanation (for general blue collar jobs) that I've read and was convincing enough is the lack of solid industrial jobs. If you can work in a factory for 2, you are not going to accept another same-effort job for 1, or a more challeging job for 2.
Of course in programming hot spots there's plain shortage so it's just an auction.
Sure, they tried and failed at this in the 90's and early 2000's but heck, people forget quickly.
One of the things that I like about working in an office is that the non-engineers see me there and get a better appreciation of what I do.
It goes on and on. Hard enough to manage a remote worker when that's the single issue; compounded with the rest it becomes a hard sell. So the price has to be low, to make it worth the effort.
Immigration dynamics within Europe are way too peculiar and can't be extrapolated to a global level. And immigration (visa/bureaucratic hassles mainly) is probably the #1 fact that needs to be considered when analyzing the payment gap problem.
Culture, trust and autonomy.
Having someone that identifies with a project, someone whose reputation in a city is connected with a project's execution, someone who builds relationships and creates a sense of ownership, is much more valuable than an engineer whose only relationship with a project is a monitor and Slack.
The idea that most good programmers move to rich countries is just asinine.
With all the talk about Google abusing its monopoly to force sites to reduce their load time and whatnot, I have to admit I am eagerly waiting fot the day (it will never come, but why not be hopeful) when Google starts to penalize insufficient or missing accessibility.
And obviously the money thrown in programming is at an all time high. As other markets like biotech begin to make big profits the monetary value of programming will equalize and go down everywhere.
If this guy would just talk to one single person who knew something about silicon valley then they wouldn't throw out all these useless theories into a vacuum.
You can just look at America for your answers. If you look at the history of silicon valley in the 1970s it was Chip companies in the 1980s it was PC hardware companies in the 1990s it was networking companies and the 2000s it was internet companies and in the 2010s it is transportation companies. The chip cpu companies have been ejected from silicon valley (vlsi design happens in oregon and austin). The PC hardware companies have been ejected from silicon valley (motherboard design and I/o design in Asia and oregon). Cisco is struggling to hang on but they are being ejected from silicon valley. People are packed into cubicles like sardines there.
The fact is if you have a lot of money silicon valley is the nicest place to live in the richest (overall) country in the world. in these fields you are 100% mobile and the work can be done anywhere so people choose to live with their peers in a place with more recreational and cultural opportunities than anyplace else on earth. And yes they struggle to afford it. Yes there is an underclass with lesser skills who might be well-served by moving to a lower income city or state but they are a fraction - maybe 20% of silicon valley.
People leave silicon valley for Boulder or Seattle most often. more land, less housing pressure, and almost as many recreational opportunities!
if a guy like this wants to know the answer he should talk to my friend who started a 20 million dollar software company in Chicago and then moved to Prague to try to repeat it and he knows exactly why software engineering doesn't scale in Eastern European countries! He is back in the USA running an organic foods company!
another possibility is another friend of mine who went to MIT lives in Taiwan and comes to USA gathers software design projects takes it back home writes this back and has his local programmers implement the system. I am certain he has a world of understanding of why software engineering is not uniformly spread worldwide.
End these other places like Vancouver where I worked as a professor it would have been better for me to get a job as a bus driver I would have been able to buy real estate sooner that way. The economics of deferred gratification just doesn't work in many foreign countries.
There is another answer which is probably the same as why don't other countries have professional football teams like the USA. there are feeder programs that start for talented software engineers as age as young as five years old (I interviewed someone yesterday whose mom began teaching him logo at age 5, after she herself took a class in it).
It's that easy - just freaking talk to someone who knows - don't generate ideas in a vacuum!
I'm curious, why doesn't it? What went wrong with your friend's enterprise in Prague?