Engineer admits he wiped 456 Cisco WebEx VMs from AWS after leaving
theregister.com
theregister.com
When you leave a job, it's in your own best interest to make sure that all of your access is removed. It's a lot harder for them to blame unexpected happenings on you if you can't even log into the thing. (Not that this happened here. I just wanted to point out a gotcha you might not have thought about.)
If you find out that they missed something, report it to them immediately and keep that paper trail demonstrating your good intentions toward them. Then hound them about it until they get around to fixing the situation. And for the love of God, don't ever, EVER log in "just to look around". Absolutely no good can come of that.
Room-elephant number two: motive. The reported facts naively summarize as "oops, ex-employee blew up some stuff in prod, caused problems". <meme>But whyyyyy??</meme> There's no indication of specifics, and seeming denials of some obvious guesses: attempts at hacking (e.g. data exfiltration for profit, which are denied), ransomware, revenge, or anything else that would explain this behavior.
Further confounding everything is the bit where the new employer's response to these revelations is apparently "shrug".
Timing aside, I myself would have to have Malicious Hate in my heart, or some ethical//moral equivalent in my brain, to do active big-cost "fire in the hole" damage on to a former employer.
Violating numerous compliance regulations by leaving the accounts of a terminated employee active for months doesn’t put Cisco “legally in the clear.” Depending on the regulator they could be in for a good sized fine.
Was it a script on a personal machine he had that was connecting to an old account he didn't thing would work? They say "deployed code", and that can be frightening easy to do in a cloud centric workflow (and if it's old code, who knows what would happen).
Something like that would also explain is current employers reticence to fire him. A mistake where you run something you don't imagine will even work, much less cause major problems that then does so because your prior employer forgot to remove credentials is something that might be looked on with a bit more understanding (and a lot of schadenfreude about he other company's lax controls causing them major problems).
A common piece of auditor evidence across many compliance frameworks is whether employees have access proportionate to their role (which is naturally highly subjective), but also proving that access is revoked when employees leave the company. This seems like an outright failure on Cisco’s part.
Hopefully they’ve learned from this and put effort into enhancing their identity governance situation.
I say this without rosy glasses about Google.
But the biggest, as others have already said is - why wern't his credentials revoked after leaving?
I can understand this at smaller companies, but Cisco has no excuse - they have enough people around that there's surely multiple people who's job it is to ensure that credentials are tied to a person, and that after a person leaves they're all revoked on anything approaching a production/customer-facing environment.
Cisco should wear this too though, this is shockingly negligent. The only reason I can think of suggests a lot more problems and likely noncompliance with regulations and standards I’m sure they claim to comply with.
One day an AWS API had an outage causing it to return an empty array for the tags list; the script deleted over 500 in-use instances.
I kept asking the sys admins to create a limited access account for my testing so that it flat couldn't delete existing customers VM's. I would walk into the office of the lead for that team once every few months and make the request again.
Until 1 day a bug in the VPS automation accidentally deleted a customer VPS thinking it was a failed deploy. They finally got around to giving me a limited account for dev/testing work.
It's scary how often this stuff falls through the cracks, even when employees KNOW it can happen.
My money is that this guy was just negligent an did not remove his old credentials + used whatever thing he was using before without doublechecking. It's extremely plausible he had this setup for a test account and ran it with the wrong credentials.
On Cisco side this bad all around. Revoke the credentials. Audit the credentials periodically. Don't allow direct production access even for engineers that work there unless it's a live production issue or a deployment that's going on (and even then you allow them access to the tooling that does the deployment not to run whatever they want with : permissions)
Edit: And https://news.ycombinator.com/item?id=24320495 provides a terraform command that is claimed to be potentially very destructive.
Terraform has a nasty habit of instead of changing what you have into what you want, destroying what you have and then building what you want from scratch. It needs to be used very, very carefully.
>According to a court document, Ramesh is in the US on an H-1B visa and has a green card application pending. "Although he and his employer recognize that his guilty plea in this case may have immigration consequences, up to and including deportation, his employer … is willing to work with him regarding the possibility of his remaining in the country and continuing to work for the company," the document [PDF] says.
Why would you re-hire someone who quit and wiped your servers?
cloud providers hate him
All work visas have a no criminal charges rule, so if this is a criminal case I believe being found guilty puts him in the area of instant visa revocation
> During his unauthorized access, Ramesh admitted that he deployed a code from his Google Cloud Project account that resulted in the deletion of 456 virtual machines for Cisco’s WebEx Teams application
It sounds like this may have been more accidental than malicious.
Not properly setting up and configuring auth could result in long duration of auth tokens, which could be sitting around unknowingly.
> Sudhish Kasaba Ramesh, who worked at Cisco from July 2016 to April 2018, admitted in a plea agreement with prosecutors that he had deliberately connected to Cisco's AWS-hosted systems without authorization in September 2018
How does that sound accidental to you?
That said, the lack of details on this do leave a lot to be imagined - it’s just as easy to read this as revenge, and that large companies don’t bother to publicly shame people most of the time.
I think he is pleading guilty to unauthorized access which was intentional - but not to the deletion which was unintended.