Hacker News Reader

Comment by fauigerzigerk | Hacker News Reader

Is there any evidence that IT outsourcing has made matters worse? Or is it only worse if the provider is located offshore?

Arguably, delegating IT operations to a proper IT company could improve security and reliability.

We use a low cost offshore IT provider and we haven't had any security or reliability issues so far (https://gsuite.google.com/)

View on HN

There's some evidence in software engineering research that offshoring isn't the problem but that the distance between team members on an organizational chart, however, is much more important. It's ok if an external party comes in as long as the resources are treated somewhat like equals to existing engineers. But when they become "that team" you will lose collaboration and things will fall apart partly because that team's manager may have different priorities than the local team.

When it comes specifically to IT, however, we're looking at almost a race to the bottom to save money immediately, sometimes to make more budget for software engineers that are revenue centers. Furthermore, a lot of IT folks want to stay current on technologies just like many developers do, but IT constraints are oftentimes even more than what their developer peers in the same company would have. This has led to a concentration of lower performers staying for a long time (not indicating skill necessarily - their environment is hostile to professional growth) and more ambitious technical folks move elsewhere.

It's been puzzling for me why IT orgs don't focus upon automation obsessively like most industries trying hard to cut costs have. The US coal industry automated a lot of labor and it's not like coal miners were making handsome salaries like most sysadmins did in the late 90s. The low performers all seem to have a strange obsession with creating as much manual processes as possible instead of, I dunno, writing some orchestration in Rundeck jobs or Ansible playbooks maybe. One decent automation engineer can replace dozens of lower skilled junior and mid level sysadmins even in bare metal environments.

The key is incentives. An IT outsourcer's goal is generally to fulfill their contractual obligations (to the letter not the spirit) for as little money as possible.

So if something like security or reliability isn't clearly stated in contracts (and it's extremely difficult to do that well) it is disregarded.

An outsourcing company is not going to spend money it doesn't have to, simple as that.

SAAS like gsuite is an entirely different prospect. you're buying a service from Google not outsourcing your IT to them. If google's service was insecure or unreliable no one would buy it.

I don't see why an IT outsourcing company would not have the greatest incentive to take security and reliability extremely seriously.

Is there anything worse for an IT service provider than being blamed for a massive IT outage at a global corporation? This is headline news.

And I don't see any difficult contractual issues at all. On the contrary. A massive outage, by definition, means that the contractual obligation is not being met.

Here is the list of things that are worse for an IT outsourcer.

1. Being more expensive than the other guy.

I doubt that anyone wants to compete solely on price. It's a margins game.

have you been involved in many IT outsourcing agreements? Cost is always a major factor in procurement, and in IT where there can be a strong market for lemons, it's often the key factor.

A company that's more expensive and can't clearly demonstrate in a bid scenario the positive impact of that increase in costs, will lose bids, a lot.

Of course cost is a major factor. I never disputed that. I'm saying that competing on price alone is undesirable for IT outsourcing companies.

But the problem is that factors like security and reliability are often invisible in outsourcing contracts, as it's very hard to specify things like security exactly in a bid contract, and very hard for customers to tell the difference between an organisation with higher security and one with poorer security practices.

Everyone will say they take security seriously, but the cost of actually doing a good job on security is much higher than paying lipservice to it, so it doesn't really make commercial sense to do it well.

In general in fact I'd say that a lot of IT outsourcing contracts tend to lead to a market for lemons. It's very hard for customer to assess the quality of a companies staff for example, so the company with the cheaper staff can afford a cheaper bid, which looks just as good as a company with more expensive staff...

Granted, security is difficult to specify. But availabilty isn't. Not on the scale we're seeing right now with BA.

You are right, but they all do it because outsourcing is a signal that corporate management has lost faith in its ability to manage IT and now has but one method of judging value - price.

If the possible negative consequences to company's image is the only incentive, they will rather try find an excuse, since they've fulfilled the requirements of the contract. Can be anything, from missing security requirements (under the assumption that their customer will take care about security himself) to incorrect use of provided software etc. And they'll probably be right, especially if they actually tried to sell the security. And they'll probably be able to defend their position in court and sue for defamation.

Security costs money, reliability costs money. If it's not in your contract, you don't pay for it.

if there's any outage that is because the customer didn't ask you do do something that's their fault not yours.

For example would you as an IT outsourcer pay for a redundant datacentre if your contract didn't call for it?

Would you patch all your systems immediately even if it caused availability issues if it wasn't explicitly outlined in the contract?

would you explain to your shareholders that your profits were lower this year because you undertook activities not specified in your contracts because they were good for the security and reliability of the services you managed...?

When outsourcing contracts are bid for there's a common experience of lower costs win. that inevitably leads to items that aren't strictly required being excluded.

>For example would you as an IT outsourcer pay for a redundant datacentre if your contract didn't call for it?

I would assume that the contract calls for particular service levels and that downing the entire fleet of a large carrier for days is in breach of that contract.

If the contract says "provide service X with 99.999% availability" the service provider cannot come back and say, oh but you forgot to specify that we should run a redundant data center to guarantee that availability.

>If the contract says "provide service X with 99.999% availability"

If you read those contracts, you have to read what the consequences are for breaking that uptime guarantee. Usually it's something silly like 10% off your next bill.

this is all correct and will not change until we reach a critical mass of devastating failures/breaches.

until then, don't hold your breath. people learn their lessons the hard way.

I've worked in outsourcing for 9 years in Romanian companies as a software developer. I had the impression that the goal was to have the customer satisfied, not to fulfill a contract (as long as the customer was also reasonable, of course) and for pragmatic reasons: a happy customer pays you longer. My longest project was for 4 years, with the same customer. And my former employer keeps working on that project, 2 years after I left the company. In these 5 years the customer's company got acquired and for some reasons they had a pretty high turnover with their own employees, but they kept working with my employer.

When I joined the company there were 60 employees, when I left it 5 years later, there were about 500. I think they could only achieve this growth based on their high work ethics and the quality of their hiring their system.

To make customer happy you don't have to build reliable software. Corporations are not machines, there's a lot of people with their own interests, who make important decisions. You have to make these people happy, first of all. They can be bribed, convinced or fooled to achieve the desired outcome for your business. And if something bad happens, you have to dodge the bullet together, because you are in the same boat. So you blame other contractors, hackers or the guy who left the customer company some time ago, and do a lot of other things to convince the customer management, that it's not your fault, you actually saved them from bigger losses somehow and you can help to build better software. And they won't be happier, but they will think that you are good guy trying to solve their problems. And that's the only thing you need to sell another contract.

Cynical much?

I wouldn't say that an outsourcing company wants to do a bad job, of course they don't, but things that are often invisible in the product (like security or reliability) will likely get less focus in a cost sensitive environment.

For example, completing a full security development lifecycle can add 10%+ to the costs of the final product. that's not a cost that a company will incur unless they have to.

In a bid for work, everyone says "we take security seriously" and the client probably can't evaluate the difference between someone who really takes all the necessary steps, and someone who pays lip service to that concept.

So the cheap company (who doesn't spend that 10% extra) looks just as good as the one that does, and they're likely cheaper.

Guess who's more likely to get the work (all other things being equal)...

Back in the 2012 I asked one of the department managers: are the customers willing to pay for automated QA tests? and he said "they usually do, because I give them the price per project and tell them 'in this price X it's included automated testing for the functionality. We can lower the price if you don't want the automated test suite".

I liked the approach and I think the same could go with security. Include an external security audit in the initial project price.

You're not wrong that delegating operations COULD improve security and reliability, but it could also suffer from the same issues that branch office suffer from -- out of sight, out of mind, distance from the corporate HQ culture and scrutiny, etc.

Furthermore, often these issues rear their ugly heads when they sack the existing staff who are keeping 1000 balls in the air and rarely dropping one. In the migration to an outsourced team, or an offshore team, or even just another equally competent team in the same office, details will get missed. It's simply not possible to do a complete knowledge transfer. Things will get missed, and sooner or later one of the things you miss will turn out to be a landmine.

Of course, the original team wasn't perfect either, and could have also suffered a large outage, but the new team is more likely to do so, at least in the short term.

In a best-case outsourcing scenario, you either hire better people, or more people, or a vendor with better processes or understanding. You do the migration very carefully and miss as few details as possible, AND the quality of the new team is such that, within 3 or 6 or 12 months, they're actually outperforming the original team. AND you manage to avoid hitting any landmines during the transition period where they're underperforming. This requires a great bit of planning, execution, and luck.

Reply on news.ycombinator.com