Lessons from Outsourcing to India, China and the Philippines
troyhunt.com
troyhunt.com
There is a difference between offshoring and cheap-shoring (i.e what I call offshoring at very low costs). Most of the bad experiences you get from India are a result of cheap-shoring. You get what you pay for. Cheapshoring undervalues good engineers and overvalues a lot of bad engineers.
I have seen this for almost 15 years (from the Indian side), and I don't like it. While some people see Indian IT as growing with the likes of Infosys, TCS, Wipro, etc - I personally saw that as crippling good CS talent in India. However, all that is coming to an end - probably sooner than later. These biggies will either have to trim down significantly or collapse under their own weight
If you want equivalently good engineers from India (as of now), be ready to pay half(45-50%) of the pay in US. So if you are paying 120K in the US, the same should cost you 54-60K USD when the engineer is in India. This will also reduce friction. One may get lucky in the early days hiring at cheaper costs, but sooner or later the good engineers will see their worth and will most likely leave instead of asking for a raise. You have to do your due-diligence selecting a worthy engineer - that is like finding a needle in a haystack.
If somebody chooses cheapshoring, they are planning for a higher cost that is at least more than the 50% mark mentioned and in most cases if you analyze retrospectively - it would have cost lower if you did not cheap-offshore in the first place. The release may seem less costly, but the maintenance and operating costs will ultimately outgrow.
E.g. ~$65k gets you a senior developer in Berlin (=€60k) and similarly in Canada (=90k CAD), so a lot more competition in that bracket.
I think this has to do with UK culturally not valuing engineers as much compared to Germany.
I can make that in less than a week as a contractor doing the same thing in literally the same place.
Thats exactly what OP is saying you'll get in India for that price. Unless your assumption is everyone in India/China is low quality at any price.
Our rent is 490 euros net per month (in Berlin) for a 1BR apartment but we're looking for a 2BR at the moment so our kid can have his own bedroom. This will probably end up costing about 800 eur/m net (not a central or hip location, but close to transit).
Also, while Germany has high income taxes it has lots of tax relief for being married and having kids. I think we ended up paying maybe 15% income tax last year (1 kid, wife earning minimally). Plus lots of subsidies for childcare & education (in Berlin even more than in other places in Germany).
You can definitely easily support a family of 3 or 4 with a single wage in that range.
There's two sides to that coin though - these companies and other cheap-shorers, as crappy as they may be, provide better employment opportunities than most other available ones to people in smaller cities and villages. I've known many such people, and most of them see these BPO/KPO type jobs similar to how Americans see "crappy" McDonalds burger flipping jobs - a way to sustain themselves and keep employed, while looking for better more suitable opportunities. (And just like with the McDonalds burger flippers and Walmart cashiers, some get out, some choose to remain, and some get stuck in the same place "for just a few more months" forever.)
I suspect this is also the reason for the high churn rate that the author mentions.
When you are comparing to the US, are you talking about Silicon Valley or the Midwest? In Silicon Valley, $120K is probably a junior engineer that would cost $54-60K in the Midwest.
Of course, there's the tyranny of numbers to deal with when you are looking:
Population of India: 1300 million.
Population of Indiana: 7 million.
(approximately)
the only case I can think of is if you need an army of developers and for some regulatory reason cannot find them / bring them locally.
I'll take anytime mediocre dev that sits next to me to some remote superior dev, since all we do is a team effort that actual coding of lines of code is quite small part of.
Thank you for your insight. Curious, what makes you say this? Is the current system trending down that path?
The better devs know that their possible value to the company is always capped, and because the country has a huge number of engineering graduates rubber stamped by a network of substandard universities and colleges, developers are usually very easily replaceable cogs.
So Google, Amazon, Microsoft, Flipkart, Freshdesk, Hike, and a lot of smaller not known Indian companies that develop specialized tech is where folks tend to go.
That sounded very worrisome particularly considering the fact that the development was for a domain like that of Pharma, where rules & regulations shadows heavily on what and how things are done.
I have some experience in developing software products for Pharma industry and it was quite an effort in keeping track of FDA updates which would impact the features that were under development or were planned for. Examples, basic and mandatory feature sets like Audits, Logging, Security, Access management, Reports etc. are heavily influenced by compliance regulations of FDA and equivalent bodies of Europe, Canada etc. And added layer of complexity is each regulatory bodies have different approach to regulation and compliance.
Considering the fact that Pfizer were outsourcing to ‘code shops’ who would never ever get into that complexity; well thought out requirement documentation which has taken into consideration current and future landscape should have been number one priority. I would have sleepless nights if I was coordinating a project where ‘can do’ developers start working from a basic one-pager from ‘business people’.
I guess it was perhaps because the author was the go between the domain experts (he mentions them as ‘business people’) and the developers.
It would have been great if the author would have shed some light on what could client do from their end in terms of reducing ambiguity while working with teams where communication and loss in translation is one of the biggest risk.
I suspect that there is a whole missing dimension in this story.
One trivial and silly real-world example: I'm currently working on a C codebase with a few developers in India. Sometime before I started the project, the American company I now work for contracted out a security audit of the code, and one of the obvious conclusions was that "safe string functions should be used" - strcpy() had been used to copy user input to statically sized buffers such that it was vulnerable to classic buffer overflows, all over the place. Requirement set, requirement followed - now strncpy() and strncmp() etc were used everywhere, but always either uselessly (25% of the time) or wrong and unsafe (75%). Most typical was to use the length of the source, calculated with strlen(). Now, I guess it was not specified "use strncpy() correctly, with the size of the destination". Failure of specification, right?
For what it's worth, the root of the problems on this particular project are that the original American lead dev was not that great and set some bad examples, which the outsourced devs copy diligently and industriously.
(But then again, it's not enough to say "this is clearly wrong, it will overflow if (expr)" - they tend to need explicit examples to copy. Sometimes they impress me by figuring the cause of a tricky bug. Sometimes they write smart but actually useless code, like using floats and log() to figure out how big a buffer is needed to print a uint16_t in decimal - 5 bytes will suffice, no float math needed. Sometimes they write 20 lines of code to calculate how long the string result of snprintf() will be, instead of using the size of the destination, right after I've lectured on what strncpy() is for. They often seem less thoughtful than a neural net. I've even seen "buf[strlen(buf)] = '\0'" - I just have no words sometimes.)
Often when I meet a programmer, I can tell if they are genuinely interested in programming, or just happen to be a programmer because of circumstances or because it's what they decided to do to make money. The latter when writing code for you would just want to somehow meet the specifications and head off for the next lunch. They would be the first ones to complain about their job on social networks. As sad as it is for them, you really don't want to hire them if your aim is to build high quality products.
There are still a lot of good, capable developers in India, available for hire and at affordable rates. You just need a good process to find them and filter out those without a genuine passion for writing code or for building products. Hiring random people without a thorough selection process is always a bad idea, no matter which country you hire from.
He addressed that early on -- he mentioned that much of the development was "brochureware" websites. Don't think the FDA gives a wet rat about that.
It really hurts me that when someone is working with an offshore team they are looking at me as a "cheap resource".
I don't expect the valley crowd to understand this, so let me suggest an experiment. Go to your best developer, and tell her that you are there in the company cause she is "cheap labour", and if it was possible in a fraction of second you would get an engineer from Google and would hire him to do your job! Repeat this constantly for 3 weeks and see if her productivity drops or if she is still around! Its really painful to spend so much time, understand your own workflow and identify shortcomings, being self critical, overcome shortcoming with tips from HN / Stackoverflow, to concentrate on the problem at hand and implement a feature and then have someone come in and call it cheap labor. Previously I was working with a team in US and was reporting to an American. I know he was happy with my performance compared with the rest of my team members in the Valley as a lot of R&D work would be assigned to me. I have worked with angualr, react, rails, java, chrome extensions, ios, android apps. I am definetly not the best in any of this. But based on my previous work I think I am fairly competent.
In enginerring its a huge challenge to bring down cost. SpaceX is celebrated cause they have brought down the cost of launching satellites, when ISRO does it, it hursts me to see the first person who says that they were able to do it cheaper cause of the labor cost in India. I respect my work, and take pride in having built it.
If you want to offshore work don't make them feel like that the only reason you are giving them the work is because they are "cheap".
I know this is not what the article says but I wanted to share this even though this is off topic.
On my team, about half the people are not local--I'm a remotee working from a different country, and a bunch of other folks are outsourced from India and Ukraine. I don't have insight into how much the outsourcing costs, but a figure of $50 per hour has been floated in this discussion. At that point you are pretty much back to local wages, so my guess is it's not the money.
This may not be your typical outsourcing story though. Individual people are actually hired like you would hire anyone local, with interviews and all, except it's a manager traveling there to do a series of interviews and not vice versa. The team is quite stable, the churn reported in the article doesn't seem to be a problem.
Only shortage is, is a person at the price that they want to pay.
For example see this : https://www.glassdoor.co.in/Salary/Goldman-Sachs-Vice-Presid.... Average is 60K USD and max 82K USD. Technology does not includes the securities division, where some people do technical work but are paid much more. (Note that to be VP here is not a very senior position )
In my previous firm (a financial firm) we paid people fresh put of college from 16000 USD (support tech) to 45000 USD (trading tech).
I was involved in compensation at my last two firms and was looking for a new job for the past few months - hence I had exposure to this, but sure how to prove any of this though.
Keeping devs who learned something met with mixed success. On the one hand, the salaries were at least market rate for devs, I think, but management still offered the real Rs.
I doubt your bangalore credentials. Needs more "do the needful" and "sir".If you have a limited budget, realize that means your requirements will be limited. Never select the vendor with the lowest rates. Always evaluate the actual people on the team.
The idea of line items for features is unrealistic for software development. It's X hours, days or weeks for each person, which needs to be re-evaluated every week or two.
If you don't have a good developer you trust before aside from the outsource team to look at the code on a routine basis then don't bother making the software at all. If you don't have a budget that allows for automated tests to be developed (20 percent of devs will do this automatically anyway) then cancel the project. If there is no budget for refactoring then please cancel the project.
If you plan on using thinly veiled racism to blame foreign developers for your inability to specify coherent requirements or comprehend the complexity of software development, then please cancel your outsourcing project.
If you follow this advice, 8/10 outsourcing projects would never start and everyone could spend their time on better things.
a) people are indeed leaving. thats cause the main selling point (at least at that consulting company was indeed price), so there was 0 commitment to employees from the employers side, so it's normal that the employees have the same and leave immediately when they get something they consider just slightly better offer.
b) trying to "adjust" the agreemtents to their benefit. A use case example is that the chinese org puts their better candidates as named engineers on their offer. Once the project is signed they start to swap them out under whatever bullshit. (Oh, we're very sorry, we have to swap John Holmes for Ron Jemery, etc.) On that point I woudln't be surprised if some of these people only existed on CVs and were totally fabricated resources simply used to try to win projects.
c) the need for constant supervision and very precise instrumentation. If you gave these chinese engineers a brick and said "see this brick here. it's nice, but I'd like you find out how to improve it somehow. Maybe make it more durable, cheaper to manufacture etc." Two weeks later the chinese engs will get back to you and ask, "what do we need to do?" However if you give them the blueprints for a brick factory and tell them to build you the factory and 1 million such bricks, that's a task they can do.
In India your slowly starting to see agencies that aren't merely code factories but places where you can do actual product development. Its early days though.
The problem however is that there aren't a lot of jobs for other kinds of engineering. I hope that changes soon too.
A bit on India with the churn that was mentioned I've seen explained as that it is difficult to get a promotion within the company - you need to get hired by another company to move up. With one contract shop that I've worked with, over the course of 18 months (we worked with them for three years) every developer position was replaced with a new person (so twice across those three years). This resulted in a lot of loss of institutional knowledge of the product. You'd train up a person, they'ed work on it for awhile, and then move on to another company and train their replacement. That training process was always lossy.
The goal for a ((stereo)typical) Indian developer is to be where the pay is best. The way that the companies are structured (and note other comments to the effect of the pay scales) is that this is in management. One gets to that by demonstrating management potential rather than technical skill. The highly paid sr. developer path isn't as viable as the (comparably) highly paid jr. management.
Another major point that didn't appear to be touched on in the article is the intellectual property leakage and protection. Many managers here are afraid of that (rightly?) and take significant effort to make sure that overseas developers are strictly limited to what they are working on. Sometimes that goes as far as "you have access to _this library_ - not the entire application that it is part of" (or for that matter, even the integration test suite - a build breaks because of a library change... and you can't provide the full test to show how it broke) leaving the integration to the in house team. That can cause some friction in the development process that is often unaccounted for at the start of the project.
I can't empathize with the other points that the author made about India -- because I've never worked on an outsourcing job.
I think this statement misses the point completely, the difference isn't the passion about coding, it's obviously the salary!
If you are one of the juniors toiling away in the front-lines, your pay is still going to be really low. A programming role only allows to get your foot through the door, if you are to make a decent living the only way is to shoot for a promotion into management.
As opposed to "the West" where you can actually pursue a technical role throughout your career because the pay is higher and there are many more different types of roles you could be doing.
When you find a strangely perfect clone of your product on the market, do not be surprised.
I work in pharma advertising as a dev but sometimes our client has their own offshore dev team. I've dealt with Wipro many times. This right here has been a common issue with them. Our account team will ask me if something is doable and I'd say yes but Wipro would push back saying they can't. And our sites fall into that brochureware category so these were mostly front-end trendy things using history push-state or what not. We usually found that they had a very limited scope and would not venture outside of it. I guess that's the difference, you get what you pay for. Usually if something is handed to me that I've never done before I do some research first and 90% of the time I figure it out. We've had better luck with eastern European offshore team in this regard. If I were looking for offshore teams I'd suggest there or South America first
The salary was one thirteenth of what I would get in Switzerland. The efficiency was about one half, but only because the Swiss CEO himself lived in Pune and later married an Indian woman. If he stayed in Switzerland efficiency would have crashed badly. In any case, don't talk about the code quality.
I also noticed the extreme churn the article mentioned. In my half a year about 5 to 10 percent of the people left and new people started.
Here is my suggestion -- don't use outsourcing agencies (acknowledging, easier said than done.) With an agency you end up with invisible HR problems. When something weird is going on at the office, you won't know about it. If senior developers are slacking off and giving junior devs all the work, you will not know. Hire the developers directly, give them the full salary, listen when they ask for something, give them flexible time off.
If what they are working on is really important, take the time to fly and go meet them in person, or if you have a team fly everyone to the same spot.
That's probably because the developers that are strong in Microsoft stack were already out of the country most likely pirated by companies in Singapore or in other parts of the world. Majors banks in Singapore had been pirating IT people in Philippines from QA to Project Managers for years.
Well-mannered firms don't do that sort of thing, you see. It drives the wages up.
Thus the word "poached" :-)
"one of the starkest patterns I saw outsourcing there was their strong affinity to PHP. This isn't intended to be a derogatory... the LAMP stack is really appealing to folks who don't have much cash... the Microsoft stack... was simply much harder to find competence in. You'd go to a vendor and their default position was 'Yeah, we can do that in PHP and MySQL'"
I see some schools switching to Python as an introduction course. But generally, the pace at which the educational system move to update their curriculum is glacial. Heck, I even see some schools still teaching VB6!
Working with vendors is logical for big projects but an obstruction for small projects. If we eliminate the middleman, outsourcers would pay less and developers would be paid more. There'd be less bureaucracy. There'd be greater loyalty.
I hope there was an easier way for outsourcers and developers to find each other.
The net effect was that if she sent an email during the day, the earliest she could expect a response from the Indian office was about 3pm Perth time, which realistically (giving an hour turnaround time) meant one round of communication in per day unless she stayed back past 5pm.
12h time difference is the worst one, but in fact any cause some changes to your organization and you may need to learn to rely on async/writting skills.
I recommend a book: https://www.amazon.com/Working-While-Theyre-Sleeping-Separat...
That was the reason off-shoring of BPO/call-centers happened in the first place - follow-the-sun.
Why are you, the American worker, tolerating your employer's request to stay late to talk to their outsourced lackeys? Whenever I've been asked to stay late for a time zone crossing meeting with outsourced employees, I've refused. WE are THEIR customers, not the other way around. They can damn well get up at 3am and WebEx-in if we're going to have a meeting. No way am I going to stay late in the evening and miss my family so I can chat with our offshore labour.
If you're not willing to stay late to do a cross time zone meeting then I agree that you're worth a bit less to your boss than you otherwise would be.
Personally I would rather accept a bit less pay to be rid of a nuisance like a boss who thinks I'll inconvenience myself and my family for stupid shit like adjusting myself to his outsourced labour's schedule. I do stay late when shipping a release or working on an urgent bug. But stay late so some delicate flower off in India can work 9-5? Fuck that.
Um, you realize clifanatic was talking about this "delicate flower" sleeping in their office because of an 10 am CST meeting and not the other way around?
Clifanatic's boss can ask those offshore workers to stay late, and then he can damn well shell out for them to take a taxi home. Maybe he can even let them come in late, given what an amazing savings he's realizing in labour costs.
But that is _not_ what the GP is saying. You're creating a strawman argument where The Boss forces his employees to stay late in office while the 'labour' gets to work within his time.
If anything, based on the work timings mentioned in his post - the work-life balance of the offshore employees is screwed and not of clifanatic's him/herself.
BOTH employees are screwed to save a dime.
The USA is a country of some 300 million people, and if you think the 20% of the country's population that voted for him somehow represents everyone here you're completely wrong.
We don't have much of a safety net in the US...
Don't get me wrong I'd much prefer if it was E.g. Germany where your boss can't email you after work.
That said, the guys in India on our support team were fairly accommodating about working what was for them swing shift, so that their schedule pretty much matched NY Stock Exchange hours. 6 AM Pacific Time would have been pretty brutal for me, personally. 7 AM was bad enough.
In most cases bosses are hoping to externalize a cost to their local employees - if their outsourced labour has to wake up early for a call, they'll probably do it happily but then charge him for it. He's happier trying to make his salaried US employees 'stretch' a bit to help him out and make the cost savings work for him.
Outsourcing works best when you treat the people you're working with like people. Expecting them to regularly call into 3AM meetings is a recipe for disaster.
What I find works well is:
1: Find some common overlap times
2: Defend the overlap times against bullshit meetings
3: Take turns meeting at odd hours