Offshoring roulette: lessons from outsourcing (2016)
troyhunt.com
troyhunt.com
Edit: Apparently nobody gets this reference anymore, and I'm old.
And don't you even THINK of stealing my red stapler. ;)
Then they must have brought in the SEAL team six of Indian devs and just got it done with no further questions asked. I was really surprised.
I can be a difficult customer work with, especially in the beginning, but I firmly belive that letting service providers run by themselves in the beginning, after having clrealy communictaed expectations and guidelines. And to step in really hard the first moment something goes wrong. Worked out pretty well in the long run as expectations became crytal clear early on. And any deviations have been addressed when they happened, so norisk of misunderstandings.
that being said, it's a people business. And just because one approach worked in one case doesn't mean its gona work in a different situation.
I've never worked with Indians, but I've had it happen to me with people in both Thailand and Austria. The Austrian one was particularly bad because it involved the movement of people and physical goods. Everything was "yes, yes, yes" until the day of execution, when it was radio silence.
Related: https://news.ycombinator.com/item?id=19078308
For example, our org has been sued twice over ADA issues (accessibility), and my recommendations are still ignored because bosses want pretty eye-candy UI's. The org deserves to be sued again, but then there's no money for servers I need. Stupid humans follow the shiny red ball over the cliff (cue grumbling sounds). Can I try chimps instead?
High-context: don't challenge authority, except via subtle hints.
Medium-context: lightly challenge authority, and quickly back down if counter-challenged.
High-context: authority is secondary to idea merit.
I do realize a group can waste too much time debating, but even important/common issues often get flattened by social concerns in a typical US office.
#protips
To summarize, the results were literally comically bad. All kinds of hand massaged spacing using nbsp's and weird off by one pixel issues and such—things that to do wrong as they did must have taken an order of magnitude longer than it would have to do them correctly.
So, I went through several rounds of pointing out really obvious issues and asking for revisions. They would apologize profusely and say essentially that the person who had been working on it previously had been a junior developer, but they'll put someone much more senior on it now. Never really made a difference.
But here's the one that really blew my mind. The site color scheme at the time was greens and whites. For some reason though they decided that the heading of the modal box should have a thick orange underline. How did they create this ~4px underline? By embedding an enormous image of a sunset and cropping it to a 4px orange line.
No exaggeration. This was like a 2560x1440px image of a sunset, which they'd cropped to a sliver in the top corner to show an orange line.
I have to assume the developer(s) got sick of my nitpicking and decided to do something so ridiculous I'd just go away. It worked.
Having cost as the reason for going with remote devs is a very bad idea in my opinion, and it should only be seen as an added benefit. You don't want to hire cheap devs because they usually suck, and great developers will cost you quite a bit regardless of where they live.
While it may be difficult to compete with the huge corporations in places like SF, NYC, Seattle, where these competing corporations offer huge salaries, great benefits and have brand recognition, there's quite a lot of value to be had with setting up shop somewhere remote, where there's lower competition and where you can offer above average salaries to attract top talent in the area. Ultimately, every company says that they only hire the best, but unless you're paying top dollar in your area, you're most likely not hiring only the best.
I believe that most good developers will want to be part of a stable company that is actually building amazing things, instead of being part of an outsourcing company where they get passed around from project to project, and their resume will perpetually read "Software Engineer at Outsourcing Company Ltd." - most would trade that in a heartbeat for one of the FAANG companies.
If you want to have the best kind of success with remote devs, don't outsource, invest a bit more and set up an actual office and hire the people directly, people that want to have a career with your company, and not a contract for an unknown period of time, which they likely won't be able to put on their resume.
I guess lots of recruiters are only interested in hiring individual contractors, but not teams.
Basically: the 10x developer is the same developer unhindered by people that build sloppy shit or broken procedures that need to be integrated.
But you also have the power, and it's far easier to plug them into your existing work structure.
Hiring a small team is effectively outsourcing a component or subsystem to a foreign coding company, the laws change -- B2B contract vs hiring -- and there are communications and workflows to deal with.
Undoubtedly, but compared to hiring through an outsourced company, you do get more control over how these employees are treated, and can thus remove the risk of the operation being run as a sweat shop under your own nose, assuming that's something you wish to care about.
> -- are you liable in your state/province/department, or theirs?
In most parts of the world you should be able to set up the equivalent of an LLC in order to minimize your risk, though of course, this will mean some extra bureaucracy to deal with.
The biggest problem however, was that no matter how many times I asked or provided documentation/feedback, they would ignore established coding standards and established processes and just do things their way, and when I would ask about this, I would just get shrugs.
Finally, we got saddled with a "Project Manager", which I put in quotes as their contribution to the process was giving daily excuses as to not turn up to our daily progress calls, and the ones he did turn up to, he would visibly fall asleep. When I eventually pulled him up on this his response was that "he wasn't directly paid for but an included part of the cost so you can't complain about what you get."
Admittedly all of the above is rather negative, but I think this highlights the "get what you pay for" aspect of offshore outsourcing.
Definitely agree with the can-do attitude of China. Sometimes I'm pleasantly surprised, other times a vendor thinks they are doing us good and screw us over by completing work before we have agreed to it. This is mostly a temporary issue until you build a relationship with the vendor (in China, relationship is everything) and know each other's expectations.
While I've never worked with an Indian vendor directly, it is interesting to hear about how people there have very narrow knowledge bases. This is something I've noticed with people I've interviewed that are recent immigrants, they know their area, but when asked about the rest of what we would be asking them to do they have zero clue. This is also (lesser so) a case with recent Chinese immigrants. Not great when we are essentially hiring a mechanical equivalent of a full stack engineer.
But if they would pay twice that (still at most half their normal cost) using a smaller higher quality company they could get a much better job done.
Anyone who has worked with the big vendors knows that the correlation between quality and price is very loose. The lowest cost provider is rarely great, but the high cost provider can also be selling you a lot of juniors.
But that probably works in most cases/places anyway.
But then the price mysteriously increases to the point where you might as well have hired them on-shore and you're not saving any money.
So, do you want to save money by offshoring everything, or do you want to get good people to actually do the real work that needs to be done?
They got where they are by doing what they are doing. Spending years on your project is something else.
I don't need to drag the building architects, master masons or master carpenters to help frame every wall of the house; ditto for code.
Probably why parent mentioned finding a smaller company. I don't know about the Indian outsourcing market specifically, but in many sectors it seems a few mid-sized companies are the best for this sort of thing.
Which also means you need to find the right one for your problem, your purchasing department won't be set up with them, and you won't be able to just order as much capacity as you want, so getting there is hard.
The whole problem is nobody can hope to spot the real "smaller, higher-quality companies" in that chaotic, offshoring-driven environment. Race-to-the-bottom is the equilibrium outcome.
The Indian developers basically just transliterated the Windows code to Unix, and got it to compile. Then they shipped it. That's when the real problems started, and I was brought in.
Now, I've never claimed to be a developer. At the time, I was a consultant and Unix sysadmin, but the concept of "DevOps" had yet to be born.
What I could tell them was that, in 1996, basing your Unix implementation on Solaris 2.5 (not even Solaris 2.5.1) was a bad idea, because there was a whole host of things wrong with that ancient version of Solaris, and if they wanted to be on that platform then they needed to be on a much more modern version.
Secondly, threading on Unix at that time was ... problematic ... and that was true even on the most modern versions of Unix that were available at the time. Multi-process programming worked well, however. On Windows, the reverse was apparently true. So, naturally the outsourcing developers used threading, and they used threading on an ancient version of Unix that had about the worst possible implementation of threading. Oy.
Of course, learning these things at this stage didn't help Symantec. They had already outsourced their aircraft and gotten an Albatross back in return.
I did help them develop a testing regime that could relatively accurately simulate a real-world distribution of mail message sizes and types, with a real-world distribution of supposedly infected messages, and then beat the holy living crap out of the server trying to push that volume through as quickly as possible.
And I did get called back to teach a class on Unix to the lead QA team in Denver, so that they could try to support NAVIEG on Solaris 2.5.
But so far as I know, the consulting company I worked for never got called back to help them actually fix their design problems with the code, and I think that project probably died a pretty quick and vicious death.
Also, so far as I know, no one at Symantec ever again tried to develop or support any code that would run on any flavor of Unix or Unix-like OS.
edit: spelling
It requires more up front work to identify, hire, and retain the remote employees, but you need far fewer developers when you can appeal to the best developers who know what they're doing.
This doesn't work if your company has any attachment to headcount, though. It can be a tough sell to hire 3 remote employees for the price of a cheap team of 10 from an outsourcing firm if your company only looks at the numbers.
Can you expand on this? I would have thought it was cheaper to hire the remote employees directly than hire the outsourcing company...
Edit: I think I understand now: the 'remote employees' are based in say the US while the outsourcing company is based in say India
I know because I work for one.
I had written some domain models for dealing with users and I needed equivalent models for dealing with notifications. So I asked them to take the work I'd done and use it as a template for the new stuff. I got back code with the method and variable names search and replaced, pretty much exactly like what I expected. What they _hadn't_ bother to modify was the SQL that actually operated on the data (yes, I had to write it out like a cave man). So I had a bunch of notifications domain objects that operated on the USER table.
That experiment terminated early.
What can we, the highly paid locals, can do to justify our high salaries at that hypothetical future point?
(playing starcraft taught me a tremendous amount of respect for East Asian creativity and work ethic)
Well, truth be told they're already surpassed us. If you want cheaper, you might as well go to Spain or Italy or even France.
I think a lot of devs in the West would be almost shocked at the amount of money devs make in other places.
* fucking largest economy by far as your domestic market ie no import duties or import/export taxes
* truly massive investment by Government and Private sector into CS research
* high tech economy focused mostly on services which demands all tools it can get to improve business processes
* high disposable income via financial instruments for cheap debt (also less taxes than most developed countries)
* largest talent pool of Universities that keeps bringing in the best and brightest across the world
The ability to sell is far far less important than the systemic reasons listed above that give the US an incredible edge in the software industry.
Maybe these offices are just following the great standards set by the people working at the HQ.
Even then, there are a bunch of companies that have their engineering based in other countries.
I personally worked for a few companies that are major player in their fields (although not unicorns) that have offices here in the Greece.
Most of these companies have one thing in common, the leadership, everyone at the C level, work and live in the US. In fact, the companies I worked for had HQs in the SV.
I don't think SV has a significantly better talent pool. My anecdote is that my local team has had more than a couple laughs looking at code written by our SV-based colleagues.
But I think that building your business there certainly gives you a huge advantage.
Your engineering doesn't need to be there.
and I'm fair certain engineering won't be there in the future. Probably not even in the US. Just the research-y type stuff might stay here longer to be close to the universities. But assuming the market is left to its own development, ginormous US based engineering teams will likely go the way of the Dodo bird in the future.
I would expect salaries to fall but I wouldn't expect software development to move entirely. It should make sense for a lot of companies to pay a premium to keep their workforce closer to their leadership.
There are certainly ways to work around most of the problems the author outlines. They are definitely real problems, same as spending 5x for onshore resources. Ideally you should build onshore teams, if you can spend the money. But if you can not, then do not get discouraged by these problems. Employee churns are not that difficult to mitigate, just matter contracting properly with the employees and providing enough consideration in the contracts.
That said, I didn't care for the way that our company was billed $75-150/hr for a developer that makes $350-1200/mo, and that the developers made so little when their managers were billing so high.
Early in my career, I was livid when I found out that my time was billed to the client 3x what I made, so I cant imagine how demoralizing it would be knowing you are paid 20x less than you are billed out.
Last I heard they were purchased by Mercado Libre in what I'd assume was an acquihire.
I did hear, however, some discussion in the office that there was a strong suspicion that the Argentinians were working on other customers' projects during the time they were supposedly 100% on our project.
Chile and Mexico are generally more stable, and both the Chilean and Mexicans I've worked with remotely have been solid. We had a technical (data center) team in Mexico that only had 1 English speaker but still got stuff done quickly and timely.
Also heard good things about the Zona America in Uruguay. No direct experience with it, however. Part of me thinks about sneaking away down there for a few years...
1) 4 hours of time overlap rule - you need to have overlap of 4 hours between teams
2) Requirements need to be much tighter with outsourcing. Often you need a full BA whereas with some devs you can work without a formal spec but some wireframes etc
>Another pattern I found time and time again when outsourcing to India was that they'd want really detailed documentation. This is always going to be a contentious issue and there are many different views of how much should be done under what circumstances. But more so in India than the other two locations I'll talk about shortly, detail was important to them. There were many occasions where we would make assumptions that a feature requirement clearly implied certain things only to later discover they was deemed "out of scope". Now that can happen in any project in any part of the world, but it was extremely prevalent in India.
> On the plus side for China though, one of their strengths (particularly over India) was the ability to get down to work with minimal documentation.
In our case we worked with two outsourcing firms in Russia, EPAM and Distillery. Our relationship worked very well but we had to approach it correctly.
In our case we treated the outsourcing firms mostly as recruiting firms. They pre-screened candidates and then we interviewed them just as if we were interviewing and on-site employee.
Now, people are people and are imperfect, But the reality was that from both EPAM and Distillery, The candidates that they brought to us were always the highest quality. Something about how they work internally makes it a lot easier to hire through them instead of the normal American way of hiring a software developer.
One more thing: I was part of the team that chose EPAM over some of the other contracting firms. we basically interviewed one of their engineers as if he was going to be a new hire, and we did the same for other firms. The reason why our relationship was so successful was because we we're methodical and carefully chose who we used for outsourcing.
IMO, most of the complaints about outsourcing come from people who didn't screen their contractors very well.
The whole idea of sending a big pile of requirements and then getting something useful back simply doesn’t work. You need direct communication without middlemen.
Like with Avni, if she really had a baby and went to have baby, the relationships were nowhere near for anyone to know it. Which again, to the extend people stay in positions due to liking people there, that sort of "keep me there" relationships were not present.
I suspect this also because we had projects where more western company outsourced their project on us - coders here, analysts, management and decision makers there. The more skilled developer, the sooner they were leaving or were very particular about what exactly they will work on. Even those who stayed (because some perks and good relationships in our local company) dont like the projects they work on. It is more about frustrations and politics then about being proud what you do.
I can only imagine that it would be even worst in outsource only company that aims for cheap and does not have perks. And it is even harder to find good managers willing to work in such setup, they tend to avoid it too.
That's very strange - I would have assumed that the code quality of the Australian outfit would be higher
I have never seen a case where they actually add value; it would always be much easier if we could just get directly connected with the end-customer.
What is something that we can do that would say “these people are worth it, let’s go with them even though they are more expensive”?
Pretty much as described in the lessons.
One important point, setting up the cooperation is critical and you can't hope to get any positive results without considerable effort into setting up and maintaining the cooperation and close vigilance over what is happening.
There is tendency for the quality to fall the moment the team figures out you are not paying attention. Basically, anything goes unless you are constantly showing you are caring and monitoring.
When I was CTO of Andela one of my main claims to fame is pioneering the "month one" curriculum, based on principles of Applied Improv -- taking comedic improvisation education and setting it in a business context. Nigeria is definitely one of those cultures where junior people are culturally ingrained in them not to disagree with their superiors. Extremely important to break them of that practice if they're going to succeed as software consultants.
"Firefox does not trust this site because it uses a certificate that is not valid for magmalabs.io. The certificate is only valid for the following names: m.magmalabs.io, www.magmalabs.io
Error code: SSL_ERROR_BAD_CERT_DOMAIN
https://magmalabs.io/ produces the error. https://www.magmalabs.io/ does not. https://magmalabs.io/ works in Chrome. Does Chrome delay the certificate validity check until after the HTTP redirect?
Chrome has a heuristic they call SSLCommonNameMismatchHandling: if Chrome tries to connect to a site like "https://example.com" but the returned cert is for the "www." subdomain of the same domain, Chrome will silently redirect to "https://www.example.com".
The other interesting thing I got to see was a shakedown of our company after we poached talent from a connected local company. We had the option (which we took) to pay several years worth of the average per capita income to local government henchmen to be allowed to stay in business.