Breach at IT Outsourcing Giant Wipro
krebsonsecurity.com
krebsonsecurity.com
I have yet to find the first multibillion outsourcing company that was any good.
All the great people are immediately hired by the company they’re actually working for, so they’re always left with the mediocre and terrible.
Your above average talent gets trusted with high stakes, client-visible scenarios like new business development, RFP pitches, and high-visibility fire fighting. The roles tend to be a combination of subject matter expertise, solutions architect, client management, and sales.
Once you have a contract in hand, theoretically the engagement is structured at that point. It no longer needs the independent competence of members of your A Team, but rather competent project/program management, client/engagement management, and execution resources.
That's the stage where a lot of problems get introduced, and it's not always a result of cost cutting or profitability measures. A lot of it comes down to operational competency (both for the agency and the client), as well as the structure of the contract itself.
The biggest thing I've learned in my time consulting is what to do (and not do) when structuring a contract for a consulting/outsourcing engagement. Everything else cascades from there.
At enterprise scale it comes down to how well the consulting team can understand and worm their way into the business processes. They need an inside ally who can enable the consultants to navigate the silos, and not just some internal PHB barking orders.
Every enterprise org chart has a different flavor of dysfunction.
I run a marketing consultancy. I can't really offload my work to cheaper talent because my deliverables can be easily evaluated. Good writing and good design, after all, are universally accessible.
But unless you're a programmer, you can't really make out good code from bad code.
Outsourcing companies will aim for the lowest common denominator, which the contract allows and with which they can get away with.
Case in point? I can exactly tell you what the specs for the first time reporting system I worked with at MajorCorp said, without ever having seen those specs:
Must be able to enter hours
That's exactly what we could do. Enter hours. Not minutes, not ten minute intervals, not quarter - or half hours, no; hours.
When I get such a spec I hit back with questions. Send that to an outsourcer and they won't see any need to provide a yota more than what the spec says.
In other words: You entering a world of pain, which looks good on spreadsheet but really only that.
That's not always true. We recently dropped a small consulting company (~20 people, US based) that was awarded a small (but not necessarily trivial) project and really bungled things up. The end product was delayed numerous times, noticeably sluggish and riddled with bugs. It would have been an easy win for this company and would have led to more recurring work, but for whatever reason, they decided to assign the project to a single junior programmer without any oversight.
Most confusing was when things were straight horrible ... there was nothing anyone can do. I don't know if the contracts say "you give us your money and just deal with whatever we give you" but that's how people would act. The outsourced company could do all sorts of absurd things that would have serious consequences had it occurred with domestic support / new procedures and etc. But when the outsource company did it and it was as plain as day "nothing we can do about it".
Principal consultants, who actually really know their shit, are called in for the contract negotiations. Those are of course not the folks, which you get to work on the project.
The actual contractors are usually rather junior, often fresh of a plane and range in quality from quite good to absolutely abyssimal. With big name consulting companies they can still be billed at a daily rate of 2K +.
In other words, and less politely put, the consulting business is pretty much a bait and switch.
eg I used to do contract work with Fujitsu some years ago, as one of the implementation people in their projects team. We'd get things set up and working well, document the heck out of it for the ops team, and then hard it over to said ops team for daily running.
It's done this way so the more experienced / creative solution finding types ;) (eg expensive) can be engaged to solve the initial problem + implement the solution. Once that's working it just needs people to run the daily playbook and look after minor issues that crop up.
Those late-joiners had absolutely no clue on what was going on, and also didn't have the competency at all to hold the helm. Result? A much mediocre rewrite of the whole platform which never fulfilled the requirements.
The consulting agency was prospecting for enterprise clients who would not pay attention close enough to such details.
Basically any industry where manpower cost is the biggest cost and you have a fixed-ish price B2B contract, the only way to squeeze out a higher margin is to lower manpower cost.
Any company over size x is, by definition, staffed by average people. That’s the very definition of average. What, then, would lead anyone to believe that a sufficiently large company could possibly be full of geniuses? It just can’t happen.
(FWIW this is why I find the companies that do seem to break the mould – Apple, for example – so interesting.)
Average for that company, not average across profession.
> What, then, would lead anyone to believe that a sufficiently large company could possibly be full of geniuses? It just can’t happen.
It absolutely can and should (though it's another question why it rarely does). It requires expending continuous effort though - effort to bring in above-average people, to raise the level of people on-board so they stay above average, and to get rid of people that are below average.
This is actually what life itself does. The default, lowest-energy state of matter is a mixed and utterly uninteresting soup of everything. Living things continuously expend energy to maintain an organized, higher-energy state against the threat of average. They continuously pump entropy out.
If orgs outsourcing projects start optimising on quality of deliverables or security, or speed of delivery, we will see drastic change in behaviour of these IT companies.
IT companies will be better incentivised to hire smart graduates, invest in trainings etc.
Why do companies pay this premium? For one, it may be because they have a budget for a "6 month project" (wink, wink, nudge, nudge) but not for a full time hire. Sometimes it's because they think that they'll be able to pick and choose the best developers from inside a large organisation (and sometimes it works out that way... almost never, but sometimes...) But really it's just a matter of being naive, I think.
The company I used to work for (as an independent contractor... ahem...) went through a phase where the CEO was absolutely convinced that it didn't pay to hire developers -- they were a PITA (always demanding weird things, never showing up before 10:30 am, never dressing properly for work, etc, etc). They always wanted raises and you have to track progress and negotiate with them on price every year. You have to pay benefits and if a person is on disability on maternity, you've got to do something about it. You constantly need to be reviewing CVs, interviewing and hiring. If you get someone really bad, you can't actually reasonably fire them for a long time. Etc, etc, etc. So he tried a few experiments with hiring from outsourcing companies (all of which were charging considerably more than any of us were getting paid, I will add)... and it wasn't a disaster, but it wasn't really any better. But the cherry on the cake was, as soon as the project was done, they fled and no amount of pleading could get those developers back again (as you can imagine). So the CEO realised that having a crack development team was definitely worth the effort and the company really transformed as a result.
However, a lot of companies are still stuck in the mindset where it's best to have as few full time developers as possible.
Maybe I'm saying this from the perspective of a programmer and not a business owner, IDK. Cloud stuff seems promising: a nice way to deliver turnkey solutions instead of custom crap that needs an outsourced body shop to maintain.
I think from that perspective it makes total sense. Unfortunately, software is an embodiment of your business requirements and that you business requirements will probably be changing from day to day. Once you realise that, you can see that you need a team to be constantly changing the software.
In bonsai (trees in pots) there is a saying that a bonsai is only "finished" when it is dead -- because it keeps growing and changing. You can never say, "I'm done" because tomorrow the tree is different. I think this is also true of software: it's only "done" when nobody is using it any more (or, I suppose, when it's so unimportant that nobody cares that it isn't correct).
I think until we get the singularity, we will always need "programmers" -- people who translate the business requirements into something that the computer can act on. In fact, it's mostly thinking about the requirements that is important. The actual encoding of the computer instructions is (hopefully!) not that difficult.
Jesus please don't draw conclusions about things you have no idea about. Would you say that designing and developing machines that replace humans bit by bit is not difficult (hopefully!).
That's a curious choice of wording. I have considerable experience encoding requirements as computer programs. According to the latest Stack Overflow survey, I seem to be in the 99th percentile in terms of coding experience ;-) I find that getting the requirements right is dramatically more difficult than encoding the instructions for the computer (even if you happen to write a big ball of hairy code).
Consider that we have 3 types of errors that can occur in software: errors in requirements, errors in translating those requirements to code, and errors in expressing that code. In other words, we can build the wrong system because we got the requirements wrong, we can build the wrong system because our code doesn't implement the requirements that we gathered, and we can have errors where the implementation has effects that we didn't consider.
There is a reasonable chance that we can build systems that given requirements will encode them into working code. This can take care of the second 2 classes of errors. However, if you look at where most of your time as a developer goes to: it is rewriting the code where the requirements were insufficient or incorrect. Until we have an AI system that can replicate human levels of thinking we won't be able to go from "I need a sales report" to code because there is no way for the system to reason about what should be in a sales report -- and especially no way to ask relevant questions about the business to write a sales report that is good for that business.
No matter how good we get at automatically translating requirements to code, you are still going to need a person to gather the requirements and think in excruciating detail about exactly what you need. You may have noticed that normal people can't do that job. They will say, "I need a sales report" and you say "What should it contain?" -- the answer will be "The sales information". Keeping track of all the mind-boggling little details and exactly how it will affect the rest of the system is still going to be a job for a human -- until we can build a system that is capable as a human (and not just any human -- a human that is able to think very hard about requirements). This will not happen in my lifetime. It might not even happen in your lifetime (which by the very nature of statistics is likely to be longer than mine ;-) ).
The "hopefully!" part was meant to be a nod to the occasional situation where people build systems that are far more complicated than the problem they are solving. Hopefully you are not building such a system ;-)
I hope that cleared up what I was trying to say. I have to admit that I am not very clear on what you were trying to say.
> All the great people are immediately hired by the company they’re actually working for, so they’re always left with the mediocre and terrible.
By "great people" I assume you mean the best talent. I assume you also mean that corporate politics (bootlicking, backstabbing, parasitic behaviour...) has no role to play in these hirings.
So I guess Wipro will have everything under control.
> He offers insights on:
> How privileged access management will fit into a frictionless security approach; > The role of artificial intelligence, machine learning and blockchain in enabling frictionless security and faster threat detection;
I seriously considered registering justaddblockchain.com after reading that to document all the places where people feel they look smarter by just adding "blockchain!" to random technical discussions.
Well, I suppose it's the step you'd expect but a state-actor engaging in a broad fishing trip still seems like a new thing. Can we expect whatever state will be installing their official botnet in whatever country next?
What we found was that suddenly a some ports with traffic distended for the internet bounced and the black holing started. A Wipro rep proudly announced that they had caused the port bouncing. You see they found that someone (Wipro... but they didn't know it was their own people at this point) had cabled around the firewalls a year earlier because something wasn't working, and never hooked them back up.
So now we were cabled to the firewalls properly... the firewalls that as far as anyone could tell had nonsensical configs that all pointed to a null route.
So for at least a year, maybe more, there was no firewall. The end customer was a large financial institution.
The incompetence was astounding, and I worked in support for a long time, and Wipro was really astounding. Everything from security to just understanding what we were telling them was mindbogglingly bad. It wasn't a language barrier, they simply didn't have many / sometimes anyone who understood the technology on the most basic level.
Wipro would open tickets dozens at a time claiming there was some sort of technical issue, but they often couldn't explain what if anything they tried. We would find the equipment at factory defaults, last boot time was when it was in the factory.... but now it was a P1 ticket because "it didn't work and it needs to be up and running by the end of the day". Then we'd ask what how they wanted it configured and they ... wouldn't know. Then they'd escalate through sales and the executives claiming we had been "working with them for weeks and were not helping".
Then they would go silent and not respond for days or weeks only to reappear later as angry as ever that we hadn't done anything when our last questions to them might be as simple as "what isn't working?".
It was worse when they actually tried configuring things as they were masters at nonsensical configurations, looping cables back into the same equipment they came from and etc. You could look at their systems that were "working" and it was errors everywhere and you couldn't trust anything you saw.
Even internally Wipro would tell us that they "can't tell" the "other team" (another team inside Wipro working with the same customer) that they need to change their configuration. They would just repeat that they can't tell them that ... and we'd be stuck because it's obvious the "other team" is configured wrong. I'd tell them to let me be the bad guy and tell them on a call, but nope. So things would just not work.
It was a common occurrence as things got worse that we would eventually end up on a conference call with Wipro and their end customer and their customer's perception was entirely off. There was no way it was miscommunication, they were straight lying to their customer all along. Often we'd have to break the news to the customer that we haven't been working on the issue for weeks, we just heard about it today, nobody can tell us how they want the product configured on the most basic level...
The only thing worse than that situation was to look up these customer's of Wipro and see they scrapped their own IT departments in favor of outsourcing, and I'm not sure they had more than a couple people who understood what was really going on.
In the last 6 months there have been security breaches at Facebook [1] Google [2] Cisco [3] and look at those threads and some of these breaches are extremely amateurish and the general consensus is these things happen and the top voted responses mirror this attitude.
Yet on the same site on the threads about India, China and non US companies we see some kind of dissonance where these are reframed as showhow affecting these companies uniquely because of 'poor standards' and 'mediocre engineers' and the top voted responses reflect this.
Far from informed discussion this not only demonizes entire groups but creates and perpetuates prejudice that will no doubt impact everything from recruitment to general behavior. And this continues on discussions beyond security to things like corruption, surveillance and other issues.
[1] https://news.ycombinator.com/item?id=19565918
One of the things when establishing a contract with a company like Wipro is that they operate on your systems from locked down rooms where disks, thumb drives, etc. are not allowed. A secure private link is setup to your corporate environment to ensure that your customer's data is not available to the outsourced firm to do with as they please.
A breach on Wipro which allowed an attacker to gain access to 11 customer systems (i.e. maybe some Fortune 500 companies) to me means they should pretty much go bankrupt because any sane customer will stop doing business with them as soon as they can. It speaks of incompetence and complete disregard for common safeguards. How can a breach into Wipro's corporate system in any way lead an attacker to Wipro's customer's data? An obvious one could be their employees sending customer credentials via the Wipro corporate email.
- Services do transport encryption, authentication, and authorization even to internal callers; presence on the LAN does not confer any privileged access.
- Identity and permissions that are user-friendly enough not to encourage workarounds, sophisticated enough to implement principle of least privilege, centralized enough to keep up to speed with personnel changes (including instant lockout on termination).
- Applications are developed with an awareness of common vulnerabilities and associated defenses (i.e. OWASP top 10). Sensitive applications are vetted by security researchers for high-end vulnerabilities. Off the shelf applications and libraries are kept up to date.
- Current ground truth and complete change history for everything about production is known and documented. Something like Puppet manifests in Git comprehensively describe everything that has been done to a VM from its baseline image, every rule in a hardware firewall, etc.
- Activities are logged; logs are correlated and analyzed for suspicious activity; alerts are promptly and competently investigated; credentials found to be compromised can be easily rotated; systems found to be vulnerable can be quickly patched or isolated.
- A quorum of insiders is required for especially sensitive operations; this is not just policy or code, but backstopped by math and hardware. At the root of trust you will find things like Shamir's Secret Sharing, HSMs, signing ceremonies, etc. and not a dude with a private key on his workstation.
I'm by no means a security expert, but as far as I'm aware, this stuff will make you a harder target than most.
If you look at a lot of the big breaches, they have some pretty common patterns. Old operating systems (XP, 7, etc.), unpatched software, excessive vendor access. This isn't because they don't have money to manage these things, it's that other business priorities with immediately visible results have taken priority. "This business software sales needs was designed for Windows XP" takes prioritization over "It's unsafe to use anything older than Windows 10 on our network". If another department and IT have a conflict, the other department wins because it brings in revenue.
If you have people in the chain of authority above IT who support IT, and understand that securing your infrastructure prevents catastrophes on the same level as fires and the PR disasters, you will generally do much better than businesses who don't. People need to understand that IT/security personnel are not "annoying" them, but trying to help them avoid catastrophes they don't even understand.
Protecting a company's FTP server that is open to thousands of employees is not a doable task, for example
And the same is true of human knowledge, a company saying that all and everything within its walls is super secret, can't truly hold anything secret.
At one of my first jobs in Canada, an owner of the company was very clear on the point what is a commercial secret and what isn't. Whenever there was a meeting genuinely demanding it, he clearly stated at the start "this is a commercial secret covered by confidentiality agreement."
Somehow I don't think this is a phishing attack.
Well, it looks like it was used as a launching point to phish their (Wipro’s) clients. They probably had s pretty good catch. I imagine one of their clients alerted them to the issues.