I Accidentally Deleted 7TB of Videos Before Going to Production

This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it. Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue, for the first few files/bits of data. Only after that was ok'd too did it run the remainder.

In other things I've worked on, I've taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.

- make sure the sum of their lengths == number of total current items

- make sure items_to_be_kept.length != 0

- make sure no two items appear in both lists

- check some items chosen at random to see if they were sorted in the correct list

At this point the only possible mistake left is to confuse the lists and send the "to_be_kept" one to the delete script; a dry run of the delete list can be in order.

It's like the difference between "do what I say" and "do what I mean"...

This pattern has the added benefit that it makes it really easy to write unit tests, which is something often sorely lacking in these sorts of batch scripts. It also makes full automation down the line a breeze, since you have nice shearing layers between your components.

http://www.laputan.org/mud/mud.html#ShearingLayers

https://news.ycombinator.com/item?id=29083367

It defaults to not doing anything so you can gradually and selectively have it do something.

Learned about when I posted my command line checklist tool on HN: https://github.com/givemefoxes/sneklist

(https://news.ycombinator.com/item?id=25811276)

You could use it to summon up a checklist of to-dos like "make sure the collection in the dictionary has the expected number of values" before a "do you want to proceed? Y/n"

    (step1) Download video from URL.  Include the Id in the filename.
    (step2) Grab the list of files that have been downloaded and parse to get the Id.  Using the Id, delete the original file.

As they say, measure twice, cut once.

Don't feel bad, I think every professional in IT goes through something similar at one time or another.

    Decide, then act.  

Another of my favorites is egregious use of caching, because traversing a DAG can result in the same decision being made four or five times, and the 'obvious' solution is to just add caches and/or promises to fix the problem.

As near as I can tell, this dates back to a time when accumulating two copies of data into memory was considered a faux pas, and so we try to stream the data and work with it at the same time. We don't live there anymore, and because we don't live there anymore we are expected to handle bigger problems, like DAGs instead of lists or trees. These incremental solutions only work with streams and sometimes trees. They don't work with graphs.

Critically, if the reason you're creating duplicate work is because you're subconsciously trying to conserve memory by acting while traversing, then adding caches completely sabotages that goal (and a number of others). If you build the plan first, then executing it is effectively dynamic programming. Or as you've pointed out, you can just not execute it at all.

Plus the testing burden is so drastically reduced that I get super-frustrated having to have this conversation with people over and over again.

Automated tests are awesome :)

    During buildup of the our_id list: assert (vimeoId not in our_ids). 
    After creating the list:  assert len(set(our_ids)) > 10000 and assert len(set(our_ids)) == len(our_ids)
    Before each final deletion: assert id not in hardcoded_list_of_golden_samples. 
    Depending on the speed required you could hit the api again here as an extra check. 

Yes, that can be a simple but powerful live on screen log. I developed a library to use an API from a SaaS vendor, in much the same way as the author. It was my first such project & I learned the hard way (wasted time, luckily no data loss or corruption) that print() was an excellent way to keep tabs on progress. On more than one occasion it saved me when the results started scrolling by and I did an oh sh*t! as I rushed to kill the job.

If both paths are on the same disk moving files is a fast operation - and if you discover a screw up, you can easily undo it. On the other hand if everything still looks fine after a few days, you just `rm -rf` that folder and purge the files.

Instead of performing the dangerous action outright, just log a message to screen (or elsewhere) and watch what is happening.

Alternatively, or subsequently, chroot and try that stuff on some dummy data to see if it actually works.

Make sure you got everything out and off before you pull up your pants, or else you better be prepared to deal with all the shit that might follow!

   SELECT COUNT(1) FROM table 
   -- UPDATE table SET col='val'
   WHERE 1=1

Good decisions come from experience. Experience comes from making bad decisions.

(Your description is so, so, spot on.)

While I didn't know what I was doing, I did manage to get the beeping to stop, and had to come in at 5 a.m. the next day to restripe the drive I'd yanked out.

Did I mention there were no backups? When I was a little bit more seasoned on the job, I raised a polite but persistent issue with management of the need for durable backups. Although I kept at it for months, they thought about it, talked about it, and ultimately did nothing. A few months after I left, the entire array failed. Since the group's work relied on the irreplaceable data, all work ground to a halt for the several months it took for an off-site company to recover the data.

That backups comment sounds very familiar.

I accidentally deleted a clients products table from the production database in my early years as a solo dev. There was only a production database. Luckily I had written a feature to export the products to an excel sheet a while before and happened to have an excel copy from the prior day. I managed to build an export to ingest the excel and repopulate the table in record speed while waiting for my phone to ring and the client to be furious. Luckily they never found out.

Thank God my automatic backups were so close to the mistake I made and I didn't lose 24 hours.

Haven't made a mistake like that since and I don't destroy DB records like that anymore.

I know how these things happen. Support ticket queues and all. And while I don’t fully know the difference in cost, I would assume a customer upgrading to an Enterprise plan would get a better support experience.

Whoever within authors company negotiated the upgrade to Enterprise (or didn’t) and failed to embed some agreement around OTT to Enterprise transition assistance was the one who made the first mistake.

Yes and No. At the end of the day, you as a business have to insulate yourself from your infrastructure provider.

> These steps are decoupled so that the list can be validated. Each step can be tested independently. And the scripts are idempotent and can be reused.

This is the most underrated comment.

I'm saying it as someone who had the ultimate oversight of deleting hundreds of TBs per day spread of billions of files on different clouds and local storage.

If you need to do it once, it’s probably 2-3 hours of work? That is identifying a duplicate video and then clicking the button(s) to delete it once every 20 seconds.

Reminds me of https://xkcd.com/1205/

The point is not to prevent these mistakes, but to keep the consequences low.

Have backups, have version control, etc.

Learn to learn and learn to work carefully. It starts in school and should be part of a proper college/university education or vocational training.

There's several ways of learning the specifics: by experience on-the-job, which can be hard if mistakes can get you fired; or by putting in the work in your free time.

If your job is to work with certain web frameworks and you're not very experienced, either ask senior devs to assist/review before going live with critical changes. Alternatively, practice at home. Unpopular, but you need to get experience from somewhere. OSS projects are a great way to do that - be that by creating your own or by contributing to an existing one.

You will do it at least once in your career. If you're old enough you will do it twice. If you're really old, you get the joy of doing it a third time.

The subtlety increases each time because you do learn.

If someone delivers code that looks like that, especially if intended for a production system, I'm firing immediately.

It's a miracle nothing has happened sooner.

    List<Foo> getFoosToUpdate(List<Foo> foos, List<Bar> bars) 

Say Foo and Bar have fields in common, such that you can say a Foo object "equals" or "matches to" a Bar object, like if they have name and dateOfBirth fields or something else that are the same (nothing like a common ID between the two). Now say there are some other fields too, like amountSpentThisYearOnDogFood that you know is always accurate for Bars, but might be out of date for Foos. How do you get the list of all the Foos to update?

Initially I did the nested for loop solution that's like

   List<Foo> getFoosToUpdate(List<Foo> foos, List<Bar> bars)
   {
    List<Foo> returnList = new List<Foo>();
    foreach (var foo in foos)
    {
     foreach (var bar in bars)
     {
      // check if "equal" or "matching" based on some criteria
      // if equal, update foo dog food expenditure with bar dog food expenditure, add to returnList, and break
     }
    }
    return returnList;
   }

The solution with a Dictionary is obviously better. All you need to ensure is that you have a method for both the Foo and Bar classes that will produce the equivalent hash for both, if they would be considered equal or matching by whatever criteria you are using.

So you could have something like

    int GetHashOfFoo(Foo foo)
    {
     string firstName = foo.FirstName;
     string lastName = foo.LastName;
     DateTime dob = foo.Dob;

     return (firstName, lastName, dob).GetHashCode(); // convenient c# method
    }

    int GetHashOfBar(Bar bar)
    {
     string firstName = bar.FirstName;
     string lastName = bar.LastName;
     DateTime dob = bar.Dob;

     return (firstName, lastName, dob).GetHashCode();
    }

   List<Foo> getFoosToUpdate(List<Foo> foos, List<Bar> bars)
   {
    List<Foo> returnList = new List<Foo>();
    Dictionary<int, Bar> barsByHash = new Dictionary<int, Bar>(bars.Count);

    foreach (var bar in bars)
    {
     int barHash = GetHashOfBar(bar);
     barsByHash[barHash] = bar;
    }

    foreach (var foo in foos)
    {
     int fooHash = GetHashOfFoo(foo);
     if (barsByHash.ContainsKey(fooHash) 
     {
      returnList.Add(foo.CopyWith(dogFoodExpenditure: barsByHash[fooHash].DogFoodExpenditure))
     }
    }
    
    return returnList;
   }

I actually messed up something like OP with this, but with doing undesired additions instead of undesired deletions.

You can think of it as having two endpoints, both expecting a .csv with rows being the things you were updating/changing/deleting.

The problem was, there was a column to indicate (with a character) whether the row was for an edit, or addition, or deletion, but this was only with one of these endpoints. For the other, there was only addition functionality, but I thought changes and deletions were also options for the other kind of .csv due to some unwise assumptions on my part (thinking that the other .csv would have the same options as the other). That's how we accidentally put in over 100 additions that should have been changes that had to be manually deleted. Luckily I had a list of all the mistaken additions.

That part's crazy! If you think it was a firing offence wouldn't they've been fired? (I don't think it is, but obviously requires system changes/explanation.)

Things are complicated, people are human and forget things, there are pressures to "get it done" and override the guardrails. Everybody has horror stories. Some worse than others. Welcome to the OP's day of horror. I would think "Silicon Valley" dev-ops horror stories make this one seem like a triviality.

In my defence, we had to get 2 PR approvals before anything was merged! But I definitely learned a thing or two from that experience

What happened is that no one knew how to react and I was probably the best suited for it, we don't really have seniority in office.

That said when I deleted the videos I immediately told my boss. He was kind of scared but his reaction was mostly "Well, now we have to re-upload them immediately, find a way. The people that uploaded them once won't be doing it twice". I was basically left on my own to find a solution (which I luckily did).

Please note that I'm in no way blaming my company or accusing it of something, this is the standard knowledge base and way of dealing with things in many places, contrary to what working in big tech or reading HN might make you believe!

It does the same thing as `https://api.ourservice.com/media?page${page}&step=100` [sic] in Javascript, or "https://api.ourservice.com/media?page$page&step=100" in Bash, PHP, Perl or Groovy (and other languages). It outs you into variable substitution / interpolation in the string literal.

In Python these string literals are called f-strings if you want to look it up. They are defined in PEP 498 - Literal String Interpolation [1] and available since Python 3.6.

[1] https://peps.python.org/pep-0498/

[sic] there probably would be a missing '=' in this url after "?page"

[0] https://docs.python.org/3/tutorial/inputoutput.html#tut-f-st...

Even as a Sr Dev I'd share stuff like that, it's code that'd appear on a stack overflow post anyway.

At least store in large TB hard disks connected with a SATA adapter when needed, and put them in a case in a safe place (better: two copies, stored in two places). What is the HD + copy time price relatively to production work ?

It's not even necessary to the story.

Thanks for the comment :)

And yes - I've literally done this exact same error (with TB of video data!). Spending the following week remediating all of that data loss was a great lesson in patience and attention to detail. :-)

OP: If you're ever looking for a job be sure to send me a message. Contact info in profile.

I do think Vimeo was irresponsible in the whine affair though.

I have made a couple of huge ones - luckily I kept my job

But nope, they went right back to scripting and got it done.

I have multiple years of experience than this man and still I could *very* *too* *easily* make a 7Tb mistake (or likely more :P )

I envy those who claim to do no mistakes at all.

After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have the infrastructure in place, if you are in a hurry. I doubt that's the case for a 10 heads shop.

‘Judgment comes from experience, and experience comes from poor judgment.’

:-)

The article hardly comes across as 'blaming' them for the core issue but they were definitely not helpful.

A million times better than your comment.

Also, would you rather everyone only ever posted about all the times they were successful?

(We do this sort of thing to protect users, usually as the result of an emailed request, and you can tell when we've done it because of the word 'redacted' in square brackets.)