Infosys leaked FullAdminAccess AWS keys on PyPI for over a year
tomforb.es
tomforb.es
Hilarious. Infosys is a known "mass recruiter" in indian colleges. WITCH (Wipro, Infosys, TCS, Cognizant, HCL) companies is where talent goes to die. No competent employee stays in those companies (from what I've witnessed). Wouldn't be surprised if this turns out to be just the tip of the iceberg, because putting people with 6-12 months of programming / computer "experience" (that they only signed up for because of the money) in charge of major production systems is a recipe for disaster.
Absolutely true from first hand experience.
Imagine being a top performer doing great work for a company whose managers insist on wasting your time putting you into needless meetings getting you to explain how you're doing everything all through badly communicated text with typos and misspellings.
They would declare everything was a P1 ticket and demand it be fixed immediately. Then we would get some output from the machine or even remotely access it and find that outside of testing at the factory this was the first time it was powered on. When we would ask them for configurations ... they were evasive.
If you got their end customer on the line you would find that they had been lying to them for months. This happened a lot ...
This could be true but you cant really generalize and it has nothing to do with the article. Infosys is not the only company leaking keys online. pretty sure tons of Amarican companies have done that
Someone hired those clowns as contractors as extra in a previous job, to loud protests from our development team. They produced what was quite possibly the most chaotic, copy-paste, typo-laden code I have ever seen in my life.
https://www.cognizantsoftvision.com/blog/pedal-metal-mozilla...
Is it overly cynical of me to wonder if this is Google's doing? Setting someone to infiltrate Mozilla's management and sabotage it, with the long-term goal of killing all serious non-chromium alternatives.
I don't know about the woke thing, I figure it's more likely to be about removing ad-block friendly API's in Manifest V3 (and presumably even more hostile changes in some future Manifest V4).
And the woke thing is non sense.
Google wants Firefox to stay around, but in a form that's much closer to Chrome. Of all the browsers that still have significant userbase remaining, Firefox is an "anomaly" in that it's one the user has more control over, and Google is slowly trying to change that.
It's interesting when you sit beside a developer who does this kind of stuff in a pair-programming context, because it immediately becomes clear that they really don't have a clue how to read and understand code in the abstract. Their process is literally copying and pasting stuff that seems similar and then running it until some arbitrary happy path test passes, not considering that it might only be passing by accident, or that they might not even be testing a real business scenario, or that there are now a bunch of unused and misleadingly-named variables floating around. And when you point that out, there isn't even a lightbulb going on that perhaps they should try to clean things up or adapt the pattern to better fit the specific use case.
I've always attributed it to a mindset that doesn't really take quality into account. And it's hard for me to argue the point when I have also been "guilty" of doing a quick hack solution or employing YAGNI to build something that might not be DRY or especially elegant but does work to solve the problem. People who just throw everything at the wall until something randomly sticks believe they're doing the same thing. Who cares if the code is unmaintainable or not performant? Who cares if there's a bug? They still get paid anyway, and the corporate machine just keeps rolling on. So - from their point of view - why make the extra effort? For me I think it's just a neatness or tidiness compulsion that makes me want to try to make code clear, robust, backward compatible and maintainable. But realistically even if I didn't do that, I'd probably still be 20 years into my career and working as a senior dev, so what's the difference?
It makes me sad.
>But realistically even if I didn't do that, I'd probably still be 20 years into my career and working as a senior dev, so what's the difference?
Similar thoughts. Looking back I think about all the time I had sacrificed to make myself better but for what? There is no value for this as I have seen totally incompetent people still standing and moving much ahead. The corporate juggernaut does not give a damn about workmanship or quality, something I learnt later in my life as I took a pause to catch my breath.
Unless you prepared well, there's often some exam questions you are clueless about, and yet there's usually no penalty for writing some bullshit in the hopes of accidentally getting a partial score. So what students are trained to do is to write whatever bullshit that seems to be relevant and hope for the best.
I realized the mindset that makes me a quality-conscious programmer is actually the anti-thesis of this. In fact during my later years I almost couldn't do that exam-bullshitting any more. It feels so bad writing something I don't understand that I almost couldn't do it.
This might be offtopic, but I guess many people who don't have a natural OCD-tendency to deeply understand their work and care about tidiness might have to actively unlearn what they trained for at least a decade in school...
I gotta say, this explains so much.
We have a FTE who came from infosys and he's very good. I have such a hard time squaring that with the team that submits an initial PR with the bin and obj directories checked in, then follows it up by adding .gitignore.txt file before FINALLY submitting a .gitignore file. And then finding them representing currency as float, or finding catch statements with a single line that rethrows it, as below (C#)
// this form throws away the stack trace from the original exception. catch(Exception ex) { throw ex; }
And when asked why this exists they add logging to it
catch(Exception ex) { _log.Debug("Unhandled exception handled.", ex); throw ex; }
----
I could go on, but the ole eyebrow just twitches whenever I think about infosys.
But then I see this other person who came from infosys. It's like trying to understand how that 6'11" basketball player came from that family who has no one over 5' tall.
Rationally I know strong technical folks can come from these companies, but damn... how? There's another poster claiming everyone makes mistakes, but no, many of the mistakes they make are not reasonable.
So it's not that everyone at companies like Infosys are bad, it's that their hiring standards are so lax and hiring rate so high that the large proportion of their engineering people are mediocre at best, and that's why most engineers at European or American companies would've been exposed to.
The typical model of a WITCH engagement is to get a new client project that requires, say 100 engineers, and immediately go to market to hire 90% of them because they don't have a bench. Screening is minimal. They're then heavily micromanaged on the project for the first few months, where it's expected that at least half of those people will fail and either their manager or the client will demand they get rotated off and then sacked. They're replaced by another cohort freshly hired and the process repeats until you have a stable-ish team of good-enough competence about 8 months in.
It works because it's still cheaper and easier for big corps and big projects and the delivered quality is fairly shit, but still acceptable. And the margins are so good that in the rare event there are late delivery penalties they're fairly easily absorbed.
The other reason is that programming as a hobby during college isn't a thing in India. (This might have changed in recent years). So you only get a chance to really mature as a programmer in the first few years out of college. So when he was ready to move on from Infosys, he had matured, but still had the Infosys 'stigma'.
And then it's really a numbers game. Infosys has hired millions in the past decade or two.
But those talented students take up any job offer they get (I.e. WITCH). when they get experience, they switch to higher tier companies.
There are many talented folks at WITCH companies, they just don't stay there.
This is not true. We had a good culture of self assembled PC enthusiasts in our district. This was in early 2000s. In fact it has reduced now maybe due to lack of interest or something even though the prices of PCs have dropped significantly. I see many people use their PCs as locked up phones with no curiosity of hardware.
>Tinkering is not encouraged in colleges in the country.
This sad state is still present to this day and it has become worse as the hardware gets more and more locked down. I don't see much interest in Linux in the younger generations.
It is absolutely true. Just because you don't consider yourself rich doesn't mean that these devices aren't out of the reach of a vast majority of the population.
Almost everyone has access to a smartphone, but most of those are poorly made, overheating pieces of plastic - barely suitable for use as a phone, let alone as a computing device on which you can learn something.
In my second statement, I'm also trying to say that though you have a huge number of people in the workforce who haven't had easy access to computing devices, there are also many who do (in absolute numbers but not in percentage). But the numbers are such that given an average Indian programmer, they are more likely to be someone who got educated in a substandard setting among unmotivated peers who have an aversion or even a fear of tinkering and putting a big investment (like a PC) at risk.
People who have money to get a PC as a hobbyist do it for unworthy reasons - like the chance to play garbage tier games like pubg or whatever, or becoming an influencer. This is a generalisation but it will affect the probabilities.
>that given an average Indian programmer, they are more likely to be someone who got educated in a substandard setting among unmotivated peers
This too has not changed much from our time to now. The changes I have noticed in students is the rise of memorizing leetcode type problems due to the plentiful jobs now which need this skill for interviewing.
I'd probably just quit.
A unix engineer could only work with Aix Tar and would not touch GNU Tar on Linux, because his manager had not approved it.
Onshore engineers flying home to India due to a stomach ache, instead of seeing a doctor for free in the host country due to being afraid. Of course messing up the flow of our projects.
10/10 will leave jobs to avoid such projects and situations again.
This feels like an exaggeration to me, although I'm open to hearing specifics to the contrary. I know of (non resident) immigrants who delay medical visits and treatments until they get back to their home country, but flying home (spending a good deal of money on air fare) for just a stomach ache sounds pound foolish, which immigrants generally aren't.
I've worked at one of these companies but left over a decade ago. I know how we're looked at when we do client work (part of why I left). Some of my colleagues were less competent, true. But, some will wipe the floor with the client employees we did the work for.
To WITCH employees: If you are an employee at one of these companies, remember you are not the worst. Many of you come from humble backgrounds and are just learning the ropes. The world is cruel. It is a tough place, and you will be discriminated against. This is your fuel. You've already made great strides; keep going. You have to.
Lower -> middle
Middle -> upper middle
Some even got rich.
In a caste discriminating society, they leveled the playing field.
Their business partners continue to do business with them. I remember an internal story, during the GFC, we worked on credit for a client who couldn’t pay their invoice($ millions). These companies are not angels, nor they only hire the best. But they’ve been the launchpad for millions of IT careers that wouldn’t have happened otherwise.
WITCH salaries are a joke, no wonder they deliver substandard results.
Getting a job in India is not a joke. And that's saying something.
Most poor people in India, are not the same poor you see in the USA. Many people who make it to WITCH companies are likely succeeding despite all odds, and are starting their career at such companies, while they can get trained, and work on projects and later use the experience to do some thing good on the longer run. Several lucky also get overseas travel opportunities many even settle outside India.
Sure things are way less than perfect. And if you come from a rich family do not join a WITCH. You can either wait out for a better job, or may be go overseas for studies or just try to immigrate to some western lands.
Most people complaining about WITCH companies are typically from a background which is already better off. And they generally find such companies to be downgrade from their current social class. The remainder do just fine.
They made lot of their shareholders very rich
Most companies the size of WITCH do not utilize access keys nor add them to source control. While a developer may make a mistake, you would expect there would be guardrails around the development process, either by way of an automated scanner or a more experienced software engineer catching it as part of a code review. The fact that none of this happened is quite concerning, IMO.
You could also perhaps say this is a management problem than an employee problem; and while that is true, such distinctions are rarely made. As an example, I'm sure you've had bad experiences with customer support which you simply summarized as "The support rep at Corp X sucks" when talking to other people; whereas the truth might be somewhere closer to "The support rep was out of luck because they didn't have a process to do A, B and C because management didn't think of it."
Most companies the size of WITCH do not use barely out of college engineers for rock bottom prices, driving them to deliver, features, features, features at all costs.
Literally all costs. It's a lot simpler to work with AWS if you can just plonk your full access key down everywhere, and even someone just out college can understand it.
Conversely, dealing with AWS Roles/Profiles and permission is a whole separate profession by this point.
InfoSys is not a company I worked with, so I can't and won't comment on them. TCS is a company I have had the misfortune to encounter. The problems with TCS is numerous, a few examples: they oversell, you're denied access to consultants that can actually help and they will always prefer to prolong an issue, rather than escalating to senior consultants. There's no incentive for one of their consultants to be pro-active or take responsibility. There are so many departments/team and layers in their organisation that there's always some one else to point the finger at.
The consultants are TCS aren't stupid or incompetent, but they also aren't being helped, pushed or motivated by seniors or their management. I do got the feeling that they would be reprimanded if they where to escalate an issue. In a meeting with TCS I suggested added 8GB of memory to a VM, as either a temporary fix, or a sort of "let's see what that does for the client". That suggestion was rejected because: It wouldn't fix the underlying issue (which was true, but they also didn't want to upgrade Java or the operating system, which was part of the problem. The OS being an old unsupported version of CentOS), and also wasn't something you could "just do". That would require involvement from 5 or 6 other departments. A month later, someone finally caved in an escalated to a higher up TCS consultant, which just added the memory as a fix until the service could be migrated to a new OS and JRE.
Anyway my point is: No, it's not the staff, not as such. They skills are for the most part perfectly fine. The company did have true experts available, if required. It's just that the culture is a really bad fit for western style companies, if you're in Northern Europe it's an even worse, because we don't share many of their values and fears. This could be solved if the Indian companies better understood the market they're selling into, because they do have the technical skills. As it stands, people like me get annoyed that we have to tell the clients that we can't fix their issues, because someone in Mumbai is afraid of looking bad to their boss or ask a colleague for help. If it has to be like that, then at least have the balls to tell the client yourself why you don't care that their systems haven't been running right for a month.
For an executive, it's easier to justify outsourcing to a large consulting firm simply because of the security afforded by the choice and the ease of justification; rather than any technical abilities they may or may not possess, and certainly it does not imply its correctness.
The anecdotes you hear are from a engineering perspective, which is where the consulting firm has to walk the walk, exposing their true abilities. It is incorrect to dismiss that as being "salty" or "pretentious", and tint them with an angle of "discrimination". The lack of processes and guardrails in these consulting companies is an objective fact.
People understand the world through dimensionality reduction and lossy compression of information in domains that they are not involved on a daily basis. This causes an inherent issue when you stack these phenomena across multiple organizational levels; this is how you end up with Intel's or Ballmer-era Microsoft's management failures, to use some non-WITCH examples, or the issues that we're talking about in this article.
Calling out incompetency which exposed privileged patient data is not discrimination.
Rough analogy: you don’t want a pilot who flunked basic aviation class to fly your plane and it’s not discrimination to keep him or her out of the cockpit.
I always used to wonder how can someone be so stupid repeatedly but then I learnt along the way that engineer's opinions hold very little value in the way of making money at the lowest cost and quality possible that they can get by.
In my experience the further you get from the money, the less of a shit people give. At a 5 person start up the result of any effort you put in is considerably more noticable, you don't have to share the credit of a innovation with a thicket of business analysts, scrum masters, executive vice presidents, etc. In that type of environment people tend to put more effort in as generally a sizeable portion of the rewards for that effort will find it's way to them. (Side note: this has changed with the innovation of Hollywood accounting[0] for start ups, and the number of truly innovative start ups has also seemingly declined)
Now think of a large company. The rewards tend to be nearly entirely rank based. You are a Software Engineer III, that pays between $x and $y, if you want a promotion you'll need to change fields into management. Perhaps a really bright idea or large effort will result in a small bonus, so you still have some reason to put effort in but probably won't go crazy.
Now go one step further, you are a employee of a 3rd party firm working for a large corporation. A big part of the firm's value prop is that they are cheap, as in they demand less of the reward for effort, they share a small portion of that with you but also have their own thicket of business analysts, scrum masters... you get the point. At that point honestly why bother? You have so many middle men between you and the results of your efforts that it's very unlikely that you'll ever see any meaningful reward. Just do what it takes to not get fired.
I’ve noticed that for awhile I had carried an innate aversion to offshore outsourcing, but only when it’s predominately non-white. It’s difficult to rid yourself of these intentional or unintentional exposure based thought patterns.
I had the privilege and good luck of ending up in a position where I ran an educational, science focused nonprofit. Then I started a business that had needed skills far more expensive in the US, before we could quite reach that level of expenditure. You learn quickly in those kinds of situations that if you carry those innate perspectives you can end up locking yourself away from some excellent talent; capable people who can work magic if you set them up for success.
This comment is only in reply to the topic of race. I’m not making any judgements or assertions about Infosys or any company in particular. Some companies and some people are bad at what they do, and that’s a global truth that is blind to race, culture, creed, politics, and anything else. I’m in full agreement that this type of security failing can, will, and has affected any company no matter what their employees look like or where they are based/operate.
Why is that surprising? Are you making reference to judging people on the color of the skin versus where they CHOOSE to work? I'm don't know anything about WITCH companies, but this is a serious false-equivalence.
I think people are unnecessarily being much more considerate and respectful than this company and its people (including the British PM’s father-in-law) deserve.
Infosys and anything or anybody related to it are worst of the worst.
That has been in the case in most investment banks as well.
To be fair, in many countries (probably most developed ones) there are regulated mandatory min and max notice periods. E.g. in France the standard is 1 to 3 months, negotiable of course.
But it was also absolutely the right thing to do. A god mode key floating around for over a year unrevoked, with real human beings’s medical data on the other side… I am glad the post author revoked the key. It is probably too little too late but they did close that door and maybe saved someone some pain: not the negligent development team, but a real patient and human being, perhaps many of them.
If this was a production key or something that seemed like it would cause financial harm/downtime, I would have never deleted it.
I do remember reading about that too though, maybe it missed it because it was JSON data not a variable definition or something?
https://docs.github.com/en/code-security/secret-scanning/sec...
I can't find anywhere that specifies the actual pattern though.
there's a json file on GitHub referencing the download of the source archive, stored on pypi infra.
in the tgz you can download from pypi you can find python code containing the secret.
https://github.com/orf/pypi-data/blob/main/release_data/i/h/...
You might be horrified by how many shitty developers want all the good guardrails GHE provides switched off, and how many managers will support them because they're a "superstar who gets things done".
They make more than $3B in free cash flow, they are worth more than $80B in market-cap and they gainfully employ more than 100k people. Folks commenting here about the "competency" of a company should realize this. Most of their clients are based in US and UK. These companies have been using Infosys' services for decades and also have locked in deals for the coming decade. If a company was really that incompetent, it really wouldn't be on the scale they are today.
You might call them a "boring services company" but they matter a great deal to a lot of people. Less pretension, more focus on "value", please? :)
The engineers who complain here don't have any influence in the decision making or otherwise they wouldn't be crying and complaining here.
It's not because they're so competent, it's because they're a convenient scapegoat when things inevitably go wrong.
Things inevitably go wrong for them because people hiring a company like Infosys do not want to be told how to do tech by competent engineers (and are probably not able to distinguish competent from incompetent engineers in the first place).
And what’s more, if one of them realizes their mistake, do you think any of them want to admit that to themselves, much less their boss, after sinking billions into it?
For the same reason the Canadian government spent billions on IBM, and Hertz on Accenture, with a complete dumpster fire for a result, and other organisations still trust Accenture and IBM (Kyndryl now) with their money. It has never been about quality with these types of contracts.
The power of computing is such that every organization on the planet is forced to lower the bar to get people who are marginally competent, even if they lack attention detail and cannot be relied on to solve problems of this sort. This kind of leak is the result.
It’s truly impossible for a single human to actually understand the physics of electronics, the world of CPU micro-architecture, packet shuffling network equipment, the nuance of CSS, and the never ending complexity of UI/UX design.
The only way this statement could be accurate is if you arbitrarily start cutting parts of the “stack” out.
Personal example: I have an electronics engineering degree that was 1 semester short of a physics degree, so I learned quantum mechanics, electromagnetic field theory, transistors, and how to create a CPU (I even created a CPU out of simple gates and way too much wire wrapping). I love computer software, so I learned assembly, how to write compilers and operating systems, and libraries. I have configured network hardware and written network software at various levels. I've also used CSS and implemented UI/UX. I've written code in many programming languages, including JavaScript, Python, C, C++, Java, Ruby, Rust, Common Lisp, and Scheme. I eventually got a Computer Science degree as well.
None of these things are magic, and the info is relatively easy to get. You simply have to keep learning and be willing to try new things. It can be fun, too.
Yes, today it can be helpful to specialize at any particular time in your life. But I think it's best to use that as a launching pad to branch out.
I guess my key point is that you always need to keep learning, and don't box yourself in too much. Ideas from elsewhere will show up... being aware of them makes it easier to use them and be ready for them.
Even if I pretend I don't know anything about AWS, if somebody handed me credentials with access called "FullAdminAccess" and told me to use them for my little script that only needs read-only access to S3 I would be extremely skeptical.
The reality is that the culture at Infosys seems to place zero value on security of customer data.
If you ask for an access key for your little script and get one, you usually only check if it works for your case and not always check if it has any other access, so I can easily see it happening without proper access controls.
At my job the alarm bells would be ringing and they would bring this up, but Infosys doesn't seem to have a culture that promotes that kind of security awareness.
And puppy mills explain the aptitude with some company cultures :p
Personally, I would have stopped right before "The Cleanup", and made a report.
Which option would I choose from https://www.cisa.gov/report? None of them stand out, so I would have chosen the last option, to send an email.
I wrote more about it here: https://tech.bluesmoon.info/2017/04/a-tale-of-datacenter-sec...
"...I glanced over at the other boxes, and they all had stickers on them saying "Administrator/password"...The three of us from TSPL looked at each other, and our president told me to decide. I asked the datacenter guy why he needed that. He said that sometimes they need to shutdown the boxes so they can move them to a different power strip. I asked him if it would be sufficient to give him an account that only had local access and could only reboot the box. He thought about it for a bit and said yes... So I created a new account that required a physically attached keyboard for login, and all it had was the ability to reboot the box. Our app was set up to start up automatically on boot, so we weren't worried about someone having to start it. DC guy physically locked the box to a rack, showed us that he was keeping they key, and we headed back to the office...
...We now needed to test our setup, so we asked everyone in the office to let us use the internet connection. We tried accessing our app, and it worked!...
...Since I had Admin access to our box, I was also able to open the "Network Neighbourhood" of our box in the datacenter. On that network, I saw all the other hosts that were in the datacenter. They had names identifying them from India's largest IT companies. These were companies I'd initially though of interning at...I looked at our president and grinned, and he looked back and said, "Send me a safe summary report when you're done" and walked off to his office.
I double clicked on one of the other big boxes and was prompted for a username and password to connect to it...
You can probably guess what happened next ;)..."
InfoSys-> US$16 billion Revenue in 2022
Wipro -> US$10 billion Revenue in 2022
I want to get out of this Universe and get into one that makes sense...
Glad to see nothing has changed.
It’s like the completely backwards on the wrong foot.
They will be fired and instead of retrospectively improving the security Infosys will ban all OSS contributions from their developers.
Sounds... good to me?
EDIT: That GitHub user is gone for good.
https://securitytxt.org/security.txt 404's
As does https://securitytxt.org/well-known/security.txt
nvm, I missed a '.'
It's funny and annoying to read every week or so about another epic fail of a multi-billion "multinational information technology company". Good luck with outsourcing your critical services and medical data to neurodivergents.
Thanks again for making my day.
PS Good old usenet. :)
Still some american banks store user passwords in plain text, Allow sim awapping without a proper check and so
I understand that it’s useless to seek answers to such questions. Let’s leave it to philosophers. :)
Can you please explain this?
Thanks for reviewing my comments, though.
A lot depends on seniors and guidance that team members provided and off course personal zeal to learn and learn every day.
How they go about stuff with that felt so weird, cause I would never get the same recruiter, makes sense they would do something like this.
They have case studies on it I suppose: https://www.infosys.com/industries/financial-services/case-s...
Should I file the HIPAA complaint, or has someone else already done that?
(the stupid government website for filing complaints is, of course, not loading for me now)
The breach notification to HHS typically comes from the covered entity. They often have the information on exactly what PHI was out there, how many individuals were impacted, and can provide the right info to HHS.
And with my experience in healthcare IT, I can say privacy and compliance officers take reports like this incredibly seriously. Those might not be the right people but getting an email to compliance folks inside the covered entity and saying “here’s a likely breach” will absolutely get the ball rolling.
Any ideas why?
Here[1] are the prefixes used for all AWS IAM access keys. Here[2] is the API definition for an access key. If you're going to search all of PyPy for keys, here's some more keys you can look for: [3] [4]
[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i... [2] https://docs.aws.amazon.com/IAM/latest/APIReference/API_Acce... [3] https://github.com/Josue87/GiveMeSecrets/blob/master/rules.p... [4] https://github.com/BitTheByte/Eagle/blob/master/plugins/spid...