I deleted data from production
blef.fr
blef.fr
Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless!
"But this kind of thing could happen to anyone" - sure, anyone who is irresponsible.
Am I ranting? Let me continue. I work on a team now with someone who seems to consistently forget to save router configurations and another person who yawns through meetings because he doesn't sleep. These guys make careless mistakes and we "learn" from them. Except we don't learn the most important lesson: Irresponsible people need to be off the team. They make more work for everyone else and make the team look bad.
But I guess the tech horror stories that amuse everyone on the forum or at the pub are better than saying, "I am responsible and do good work."
Be humble because the complexity of modern systems is insane, there is no way you have all cases covered, if you had you would had a formal proof.
We all are just doing our best to cover most of the edge cases. That is why we need to keep learning from other peoples mistakes.
>It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason
hdfs dfs -rm /data
We had nightly back-ups though.
Copy pasting is not an excuse, before you run anything destructive, you double check what you're running.
Anyone who is responsible will double check before running this kind of command, and if I can't get that person off the team, I'd severely restrict his access (in general, I think most people in a team should not have access to production data).
And we do have disaster recovery plans so there's very little that could be done that would be catastrophic. But still, a lot of the disaster recovery plans call for downtime because it's not worth the cost benefit to engineer the system to be completely resilient to idiocy.
And regarding your team members: if you don't trust them, don't rant about it on public forum, man, move them away from important systems, implement personal improvement plan, and when that fails, fire them.
Companies cut corners by not hiring architects because they are not cheap, and then they cut corners on salary for roles on mission critical projects because they are also cheap, and they often pay the price anyway just later-on because failure is expensive.
Hire experienced people for the right money if it's a mission-critical system. The simple facts are all there.
Nobody successful designs a good building without a good architect, and the project never turns out well when pay isn't right.
Also important to note: The architect should not also be the one to physically build the "house", in order to maintain proper objectivity and in order to avoid conflict, even if the company insists on being cheap.
OP was using HDFS which is a distributed/clustered filesystem, their data was replicated across multiple machines. But "rm" still deletes it from everywhere.
I’ve worked with similar people and had similar experiences. They are negligent, never question their own abilities nor actions, and to compound this they don’t learn from the experiences.
Whilst no-one is perfect, it’s always possible to test before proceeding to cause uncontrolled havoc. Just remind oneself, check, check, check.
Assume that there will be problems before proceeding; forewarned is forearmed.
I don't know on which kind of teams you work in, but I personally worked a lot in teams where I was the only person responsible to do data related tech stuff for other people. I never said that I wasn't responsible but I was alone, so setuping backup was often the last of my priorities. I was and I did all my best to fix things as fast as possible and to remove any issues it could lead in the future.
When I was setuping Hadoop cluster in 2014, almost no-one knows what it was in my local market. So let us try, then fail and learn.
If you also think from 3 stories I'm writing on the internet that I'm careless so be it. But I also recommend you to fix your "router configuration" problem rather than ranting on posts on internet.
I've been on both sides of this - the person that wants everyone to think twice before they type... but also the one that messed up.
I'm really here just to say that... sometimes things aren't so clean-cut.
For example: 'Business needs' often push myself and others to make calls that we'd normally never make. Coming from the top, down - executives rarely care about 'fact'.
Just what they promised, and everyone suffers for it.
In the end, the whole experience is a wash. We don't hit the target, and everyone is more stressed out. Repeat this enough, the diligence of most will falter.
Friendly advice: you don't sound like you rant, you sound like something else.
It's kind of "welcome to the club" right of passage
Yes, everyone should experience making catastrophic mistakes but please spare some effort to only do it in non-prod...
Easy to mess things up when you routinely execute a bunch of commands everyday, many of which look similar in form but differ greatly in the pains they can create
This is false dichotomy.
If people can screw something up, sooner or later they will. It doesn't matter how responsible or irresponsible they are - the processes in place should prevent mistakes from being made regardless of that, or to mitigate their consequences if they're unavoidable. Anything less and you're not addressing the root cause of the issue. These same processes should ensure that no one can create changes to the state of the system before them first going through another set of eyes and being validated.
Most sane OSes at least prompt you before deleting a file - the very kind of safeguard that makes you double check whether what you're doing makes sense. Similarly, you'll notice that cars have seat belts and air bags, even if you're not going to crash daily. Get the irresponsible people off the team if you'd like, but if you can't put the appropriate processes in place to prevent mistakes, you probably need to rethink your priorities and provide the adequate pushback against "the business", when they expect you to SSH into prod and do anything.
What that looks like in my current environment:
- all server configuration is managed through Ansible and Git
- developers have read only access to the servers when needed, no one can change the state of the system there
- all changes are versioned and use merge requests and automated CI processes, need to be reviewed first and synced with the change management system, to know who is attempting to change something, why and also how (with the appropriate permissions)
- furthermore, the Ansible processes are scheduled to run every day as well, just to make sure that the server status is as expected
- of course, the changes to servers are also first done on development environments, then on accept testing environments, then on clients' testing environments (any number of them that's necessary) and then finally to prod
- there is Zabbix for monitoring the infrastructure with alerting in place, as well as Skywalking for application APM, as well as some lower level tools
- the apps are also shipped as containers, which have very similar processes in place, e.g. full CI run on every merge request, before it gets merged
- those containers run in clusters, so if any of them fail for whatever reason, the load will be balanced as necessary and/or restarts will take place, each container also having health checks
- lots of manual QA in there as well, to catch the things that aren't easily automated, as well as regression testing
- automated integration and load tests done every now and then, to check that there are no regressions in a new release
- testing the reproducibility of this can also be done by wiping any server and letting the CI processes restore a new one, complete with the appropriate access roles, firewall configuration, cluster membership, observation tools for developers to use etc.
- everything also has backups and rollback strategies as needed
- despite all of the procedures in place, delivering a new version can be as easy as clicking a button on a CI pipeline, the container images being delivered and becoming available on the clients' side in a few minutes for further processing
Disclaimer: of course, the amount of effort that goes into designing a system and the workflows around it depends on a variety of factors, so not all systems need that sort of focus on quality. For most CRUD apps out there, you can probably just wing it, even though then you must be ready for the eventual outcomes and downtime that it will cause.
I was supposed to just be a PM, not even supposed to be a Dev... Served me right for acting like I might have been able to fix the issue. No more volunteering outside of my role... Possibly also why job descriptions include so much out of normal scope "responsibility" now.
The company had a bad habit of underpaying roles, and regularly hired inexperienced admins to do key PROD work for clients.
I was also blacklisted for rehire from the company and took the fall I guess when the issue was explained to the clients.
Oh well.. Spilled milk... The company kept failing after I left and lost all it's major contracts weeks after I got cut anyway. I got a decent severance payout and still won a 25% raise on my next gig... Moved forward without scratches.
Lessons -
Back everything up well, no matter how much time it takes, before touching PROD or even before others touch PROD.
Don't work for places that are regularly dedicated to underpaying talent to do mission critical work.
When a door closes, another opens, learn from the past, but don't carry it forward, and don't fight ignorant and tech-blind arrogance in leadership, it only hinders your career growth if you crusade against bad leadership. Karma does the real work for you eventually if you remain calm & composed.
Quickly identify if tech talent and problem solving stops at your level, and if so, protect yourself from being a scapegoat for leadership failures.
Don't be the one that screws things up, be the one that both anticipates and quickly and fixes all varieties of screw-ups from everyone else.
But... the database wasn't gone? If you delete a symlink, you can just put it back.
This sounds really wrong - installation of all of your dependencies should be run as a part of building the container for a particular version of your application, say, as a directive within the Dockerfile.
That should also be done before any unit tests and integration tests are run against that container, well before it is deployed to any environment. The end result of doing that properly is a container that will work correctly until the end of time (or at least any vulnerabilities become apparent and a new version needs to be built, or for other reasons, such as bug fixes or added functionality).
Furthermore, all of your IAC should be treated just like regular code, which means full descriptions and comments for any non-trivial parts, or even explaining why the trivial parts are there to begin with (the greater circumstances, links to change requests etc. that the actual code doesn't otherwise contain). If you can't look at any single piece of your code and understand both what it does as well as why it's attempting to do it, then clearly there is still more work to be done. Whether that's refactoring or documentation, however, depends on the circumstances.
IAC doesn't absolve you from the need of having documentation. It simply allows you to lower the degree to which PEBKAC is a problem in manual deployments and allows you to document the actual code that does things, rather than creating .docx descriptions that no one will read.
It's amazing how much impact a small console application can have on preventing mistakes. All you really need to do is wrap the danger with "are you sure?" and "enter the exact commit hash..." kinds of prompts. For the super dangerous stuff you can bake in some centralization and approval loops.
But... some of this just feels like carelessness. I've certainly made my share of mistakes, but these just feel egregiously bad. This takeaway in particular is just all wrong:
> Measure the risk when you give all the permissions to one developer, one data engineer or one SRE — it means similar stories could happen
There is no need to measure that risk, because no one should have permissions to take down your production infrastructure so easily. This just seems like table stakes for running or working at a company that operates this kind of infrastructure.
I think it's three things: first, abide by the principle of least privilege, and make company-destroying permissions hard to come by; second, put safe (web, command-line, whatever) interfaces in front of common tasks that need to be done that could turn into accidental downtime; and third, when it's required that you do things outside of the safe interfaces, drill it into people that you never do them without a copilot who can check over your work, in real-time, before you do anything (and if anyone develops a reputation for being a cowboy... seriously, fire them).
More recently, had a coworker want to run terraform plan, couldn't, because the state file is access controlled, and so he just reset the configuration locally to a blank local state, plan (which wasn't read, again), and then subsequently created a lot of duplicate resources. (Since TF was essentially starting from a blank slate.)
We had an admin interface that used a generic UI; every database table was just a generic CRUD "it's a table, here's the columns & values" to it. The mobile app was an OAuth app, just like any third-party app was. One day, someone on the QA team (why did QA have access to prod, you ask? Good question…) deleted the mobile app's OAuth client from the database. "Why did you do this?" :shrug: "Why do we even have this lever, Kronk?" I suggested at the time we should have kept the delete button, but special cased it to remove the admin privs of anyone who clicked it, since clearly they failed the test…
I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no? (I'm not sure I'd want to try that on a live OS … and … if this is the cloud, usually growing the underlying EBS or cloud disk or whatever is pretty easy to do? But I also usually try to keep any "it could grow" data on a separate partition to spare / the pain…) Though I think I'd bind-mount it back in place, not symlink it.
The way it reads, they didn't know how POSIX permissions work, and maybe still don't. The article says that sudo needs to be owned by uid 0, but does not mention the critical suid bit. Maybe they just recursively copied /usr without preserving permissions?
Also, the mention of sudo makes it pretty clear that they did not do this in an interactive root shell (obtainable with sudo -s/-i or su if there is a root pw). If you do this kind of shenanigans, you better have multiple open root shells in front of you, and some statically linked binaries (busybox?) to recover from the inevitable chaos of missing binaries, shared libraries, and dynamic loaders that you'll transiently have.[1] It's not impossible to perform, but requires careful planning and a good understanding.
[1] Though if /usr was indeed copied without preserving permissions, recovering would have been very tedious.
You're totally right in your message. I did not had access to root password.
I was junior and while I was trying to fix stuff I was adding more problem to the initial one that was a disk space issue.
Although certainly people still do bad things, dumb things, and careless things -- this (the middle part -- team/company controls and culture) is a wise management approach, although best applied as a principle ahead-of-time.
This is a personal post about what I learnt and some takeaways.
Did you ever feel the same way?
I never said I was an expert in the post, I never said I consider myself better than everyone, and moreover I'm 100% sure I can bet I've not impacted the life of one person that commented my post.
Is it my fault when I just graduated that I had been given the responsibility of building an Hadoop cluster for a +$100m revenue company? No, I just tried to do my best. I fucked up things and I fixed it. Period.
I'd love to work with all people telling me I was incompetent, but unfortunately you weren't here to bring me the light in 2014.
7 years after I can sleep at night and live with my mistakes. So, yeah, sorry for that.
I'm closing this, love.
If you mix production with dev something bad is going to happen.
If you expect no mistakes to happen, well, something bad is going to happen. Authorization and prompts are there for a reason.
@comments discussing about dismissing OP -- doesn't matter: you don't have the data anymore. Production data should be indestructible practically or it doesn't matter.
Sure your client/user is going to be OK if you fired your junior dev Bob... Just comment how many times that has worked.
For some reason the move into the repository folder was instantaneous but the opposite wasn't. All data import attempts errored during the 14 hours required for the data to be moved back to its original place. I emailed and voiced my apologies to the team present both in India and in France.
While I was not fired on the spot, it might have indirectly led to the disappearance of our devops (devoops?) team in Paris. Fortunately enough I guess I was the last team member to be sent away.
For my defense, our dev team had a intervention scope so big that they were essentially super-men once they have been granted access. One the rights granted to us was full write-access to all incoming data folders.
Quoi? Do *not* hide stuff from colleagues.
That's probably what ended up causing `terraform destroy`.
Do not hide stuff from your coworkers. It's toxic in a workplace. Hopefully it's a typo.
On my teams, I tend to be the person put in the role of doing DevOps-ish work because it's something I'm familiar with. I honestly find it incredibly stress inducing. It seems to me there is an asymmetry in the risk of work between team members that never gets captured by management.
There are degrees of difference with security and data concerns in regards to frontend versus backend versus devops tasks. There is some work that can carry the risk of permanent and irreversible data loss, yet everywhere I've worked, it is apportioned as a regular tasks. Failures like Ops are first and foremost an organizational issue.
(emphasis mine) Was this a typo? If it is, it's kind of funny that an article about making mistakes in production also made this tiny but critical mistake.
Basically a tool that allow someone to run some SQL on production but, before doing it, request his credentials, then run the sql command in some "--dry-run" mode and if you modify more than 10 rows, then it stops everything. And if it goes on, record the SQL that was run somewhere so we can audit...
I've seen my share of honest mistakes (and committed some of them), but... damn.
Not _these_ mistakes, though. The author describes incompetence, nothing else.
That way it's almost impossible to have anything catastrophic happen unless the whole planet goes boom!
Currently the main font:
font-size: 0.6em;
Example screenshot: https://imgur.com/IsPvHqE